[2025-08-14 09:01:23.714344] INFO New syslog file info: [address: "172.77.0.7:2882", observer version: OceanBase_CE 4.3.5.2, revision: 102020032025070315-563ee271f78ff8521c8e0e09ae682065b065a046, sysname: Linux, os release: 3.10.0-1160.92.1.el7.x86_64, machine: x86_64, tz GMT offset: 00:00] [657]OB_MALLOC COST TOO MUCH TIME, cost_time=961078, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=32896, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=961071], seq:[0] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=1970708, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=17946, WASH_BLOCK_END=1952754, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=3], seq:[0, 1, 2, 3, 4] [2025-08-14 09:01:21.745038] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1970762, owner: ObLog, click_count: 5, time dist:[FORMAT_END=10, ObMalloc#start=2, ObMalloc#end=1970748, ALLOC_END=2, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [383]OB_MALLOC COST TOO MUCH TIME, cost_time=1965477, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=12548, WASH_BLOCK_END=1952906], seq:[0, 1, 2] [2025-08-14 09:01:21.750526] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5438][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1965524, owner: ObLog, click_count: 5, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=1965487, ALLOC_END=25, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [399]OB_MALLOC COST TOO MUCH TIME, cost_time=1968931, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=15849, WASH_BLOCK_END=1953080, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1, 2, 3, 4] [2025-08-14 09:01:21.747162] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6119][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1968952, owner: ObLog, click_count: 5, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=1968941, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:21.257220] EDIAG [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1519022, owner: ObLog, click_count: 9, time dist:[FORMAT_END=26, ObMalloc#start=0, ObMalloc#end=480074, ObMalloc#start=18, ObMalloc#end=1032003, ALLOC_END=6900, APPEND_END=1, ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=2], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x18257d0c 0x1823b885 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:18.774717] EDIAG [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3983204, owner: ObLog, click_count: 5, time dist:[FORMAT_END=3959580, ObMalloc#start=1, ObMalloc#end=23619, ALLOC_END=4, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x989c577 0x989bc96 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:21.748878] EDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1968615, owner: ObLog, click_count: 5, time dist:[FORMAT_END=234, ObMalloc#start=3, ObMalloc#end=1021023, ALLOC_END=947354, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0xfc0e73d 0x993f560 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:20.780858] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:20.780864] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:20.780871] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:20.780876] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:20.780888] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:20.782524] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:20.788578] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:20.822583] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:20.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:20.822778] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:01:20.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:20.823770] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:20.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:20.830153] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=26551] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:20.840417] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=35] PNIO [ratelimit] time: 1755162080840415, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007612, add_bytes: 0 [2025-08-14 09:01:20.855128] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:20.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:20.866375] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:01:20.866400] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:01:20.874792] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=23] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:20.922644] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:20.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:20.923853] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:20.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:20.930181] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=11] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:20.955189] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:20.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:20.974852] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=14] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:20.991582] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:20.989, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.011674] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:21.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.022667] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:20.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.023917] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:20.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.122729] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:21.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.222757] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:21.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.322789] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:21.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.422866] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:21.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.522929] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:21.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.622990] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:21.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:20.780896] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:01:21.730473] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=948140] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:01:20.790044] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:21.730585] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=939119][errcode=0] epoll handler too much time :940561 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:20.855552] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:21.730609] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=873731][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:20.975907] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=1869137][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162080.790022 cost: 0.185882 loop:26429, slowcnt: 1 [2025-08-14 09:01:20.996000] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:20.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.730638] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=733523][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:20.995|207=734.63ms|210=0us|222=2us|total=734.66ms) [2025-08-14 09:01:21.730660] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4389] timer task use too much time(end=1755162081730659, start=1755162080995997, delta=734662) [2025-08-14 09:01:21.730663] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [804][T1002_TimerWK0_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=1997830] get_disk_usage(ret=0, capacity(MB):=24883, used(MB):=8642) [2025-08-14 09:01:21.730675] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=486041, expected_run_ts_us_=1755162081244633, ObClockGenerator::getRealClock()=1755162081730674, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:21.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.730702] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:21.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.030230] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:21.730821] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=699510] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:21.055245] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:20.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.730842] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=674573][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:21.552|207=675.60ms|210=1us|222=1us|total=675.61ms) [2025-08-14 09:01:21.730854] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task use too much time(end=1755162081730854, start=1755162081055243, delta=675611) [2025-08-14 09:01:21.720333] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:21.074912] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=12] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:21.730877] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10526][errcode=0] epoll handler too much time :10554 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:21.730885] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=654972] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:21.092168] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:21.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.730900] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=637765][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:21.921|207=638.73ms|210=0us|222=1us|total=638.75ms) [2025-08-14 09:01:21.123959] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:21.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.257120] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:01:21.667479] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:01:21.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.731017] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=976272][errcode=0] epoll handler too much time :977758 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:21.730911] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=980676][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:20.748|207=982.16ms|210=0us|222=2us|total=982.30ms) [2025-08-14 09:01:21.731051] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=139][errcode=-4389] timer task use too much time(end=1755162081731051, start=1755162080748744, delta=982307) [2025-08-14 09:01:21.723062] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:21.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.731080] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] task scheduled too delay(scheduled_delay=999642, expected_run_ts_us_=1755162080731436, ObClockGenerator::getRealClock()=1755162081731079, ret=0, ret="OB_SUCCESS", *this={this:0x7f98918114b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f73a0d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:21.737349] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=6257][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918114b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f73a0d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:21.737496] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=143][errcode=0] task scheduled too delay(scheduled_delay=1003043, expected_run_ts_us_=1755162080734452, ObClockGenerator::getRealClock()=1755162081737495, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f728aa90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:21.737529] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f728aa90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:21.737754] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:21.730937] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=606058][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:21.123|207=606.98ms|210=0us|222=1us|total=607.15ms) [2025-08-14 09:01:21.731119] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=473124][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:01:21.737919] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6780][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:01:21.737932] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:00:41.841, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=08:59:41.841], idx=471, next_allow_print_ts="09:01:11.841") [2025-08-14 09:01:21.737943] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:01:21.737968] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:01:21.737984] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:01:21.737996] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:01:21.737632] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:21.730918] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task use too much time(end=1755162081730911, start=1755162081092165, delta=638746) [2025-08-14 09:01:21.737820] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=6872][errcode=-4389] timer task use too much time(end=1755162081737819, start=1755162081123958, delta=613861) [2025-08-14 09:01:21.738011] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:01:21.738644] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=609][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=08:59:57.942, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:27.942], idx=868, next_allow_print_ts="09:00:27.942") [2025-08-14 09:01:21.738837] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:01:21.738848] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:01:21.738859] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:01:21.737548] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] task scheduled too delay(scheduled_delay=1006730, expected_run_ts_us_=1755162080730817, ObClockGenerator::getRealClock()=1755162081737548, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7f0490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:21.739004] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1453][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7f0490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:21.739021] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] task scheduled too delay(scheduled_delay=1006619, expected_run_ts_us_=1755162080732401, ObClockGenerator::getRealClock()=1755162081739021, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7f0610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:21.739032] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7f0610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:21.716022] WDIAG [STORAGE.TRANS] unlock_ctx (ob_trans_ctx_lock.cpp:154) [865][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4389] ctx lock too much time(arg.trans_id_={txid:8095929}, arg.ls_id_={id:1001}, lock_ts=1995266, lbt()="0x26a1806d 0x1b543a8a 0x96d0e30 0x96d031d 0x971cc91 0x971c9f1 0x971c80a 0x971c5dd 0x986bcba 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3") [2025-08-14 09:01:21.739169] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [14780][T1001_TimerWK4_SSTableGC][T1001][YB42AC4D0007-00063C4974099C4E-0-0] [lt=18][errcode=0] cost too much time:mds_table_base.cpp:int oceanbase::storage::mds::MdsTableBase::advance_state_to(oceanbase::storage::mds::MdsTableBase::State) const, (this=|threshold=1.00ms|start at 09:01:21.734|total=4.20ms) [2025-08-14 09:01:21.739065] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] task scheduled too delay(scheduled_delay=1005627, expected_run_ts_us_=1755162080733437, ObClockGenerator::getRealClock()=1755162081739064, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7281290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:21.731142] WARN [LIB] alloc (ob_malloc_allocator.cpp:924) [657][T1001_Occam][T1][Y0-0000000000000000-0-0] [lt=971190][errcode=-4013] [OOPS]: alloc failed reason is that server memory has reached the upper limit(server_hold: 4293320704, server_limit: 4294967296, alloc_size: 2097152) [2025-08-14 09:01:21.739347] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=279][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7281290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:21.739136] WDIAG [STORAGE.TRANS] reset (ob_trans_ctx_lock.cpp:220) [865][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=23070][errcode=0] [slow ctx lock](request_used=0, hold_used=1992690, lbt()="0x26a1806d 0x1b543ef5 0x96d03e7 0x971cc91 0x971c9f1 0x971c80a 0x971c5dd 0x986bcba 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3") [2025-08-14 09:01:21.739379] INFO [STORAGE.TRANS] finish_iter_single (ob_trans_functor.h:92) [865][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=242] single tx cost too much time(functor_name="IteratePartCtxAskSchedulerStatusFunctor", tx_id={txid:8095929}, ls_id={id:1001}, cost_time=2017420, single_begin_time_=1755162079721958) [2025-08-14 09:01:21.739375] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] task scheduled too delay(scheduled_delay=997747, expected_run_ts_us_=1755162080741627, ObClockGenerator::getRealClock()=1755162081739374, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891825810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:21.739396] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891825810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:21.739406] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=995853, expected_run_ts_us_=1755162080743552, ObClockGenerator::getRealClock()=1755162081739406, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891971290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:21.740848] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:21.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.742514] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:21.742592] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:21.750896] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:21.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.755347] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:21.755412] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:21.755432] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=5744] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:21.755484] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2776] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:21.757408] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=8] gc stale ls task succ [2025-08-14 09:01:21.023331] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=19] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:01:21.762615] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=38102] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:21.738266] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=0kB/s, net_out=0kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=1280000kB/s [2025-08-14 09:01:21.739415] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891971290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:21.763410] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23953][errcode=0] task scheduled too delay(scheduled_delay=1020252, expected_run_ts_us_=1755162080743155, ObClockGenerator::getRealClock()=1755162081763408, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f73a0c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:21.763440] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f73a0c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:21.763486] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=43][errcode=0] task scheduled too delay(scheduled_delay=1006586, expected_run_ts_us_=1755162080756899, ObClockGenerator::getRealClock()=1755162081763486, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fc830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891825210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:21.763497] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fc830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891825210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:21.764658] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=22] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:01:21.762699] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=25] current monitor number(seq_=-1) [2025-08-14 09:01:21.767086] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:21.767106] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] temporary file module use no blocks [2025-08-14 09:01:21.767112] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=5] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:21.767121] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:21.767124] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=42] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:21.767137] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] temporary file module use no blocks [2025-08-14 09:01:21.767142] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=4] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:21.767149] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=7] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:21.767131] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43622983205596, flush_io_finished_ret_:0, flush_io_finished_round_:26257, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:21.767157] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=7] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43622983206676, flush_io_finished_ret_:0, flush_io_finished_round_:26231, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:21.731004] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=979191][errcode=0] epoll handler too much time :980689 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:21.767947] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=36873] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:21.763659] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:21.755180] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2269] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:21.770015] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14812][errcode=0] epoll handler too much time :14846 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:21.738438] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=7502][errcode=0] task scheduled too delay(scheduled_delay=534139, expected_run_ts_us_=1755162081204297, ObClockGenerator::getRealClock()=1755162081738437, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:21.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.770356] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31868][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:21.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.770371] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=531225, expected_run_ts_us_=1755162081239145, ObClockGenerator::getRealClock()=1755162081770371, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:21.239, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.770391] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:21.239, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.770403] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=514916, expected_run_ts_us_=1755162081255487, ObClockGenerator::getRealClock()=1755162081770403, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:21.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.770412] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:21.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.730706] WDIAG alarm_if_necessary (ob_timer_service.cpp:143) [804][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4389] timer task's elapsed_time is greater than period(token={this:0x7f991b7a5298, timer:0x7f9922ffd5d0, task:0x7f9922ffd618, task_type:"N9oceanbase7storage10checkpoint19ObCheckPointService24ObCheckClogDiskUsageTaskE", scheduled_time:1755162000783360, pushed_time:1755162000785000, delay:2000000}, elapsed_time=80945681, period=2000000) [2025-08-14 09:01:21.763508] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1007319, expected_run_ts_us_=1755162080756188, ObClockGenerator::getRealClock()=1755162081763508, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7282910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:21.772047] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8524][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7282910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:21.772069] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] task scheduled too delay(scheduled_delay=1015750, expected_run_ts_us_=1755162080756319, ObClockGenerator::getRealClock()=1755162081772069, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a54b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7282a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:21.775031] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:01:21.781244] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [409][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] [sql nio session](*s={this:0x7f98f7334030, session_id:3221557375, trace_id:YB42AC4D0007-00063C4962CA3040-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:262}, err:0, last_decode_time:1755161121788142, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:2389, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:01:21.781270] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [409][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=26] [sql nio session](*s={this:0x7f98f7333030, session_id:3221589989, trace_id:YB42AC4D0007-00063C4962CA3068-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:273}, err:0, last_decode_time:1755161766744050, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1903, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:01:21.781283] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [409][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] [sql nio session](*s={this:0x7f98f7339030, session_id:3221590023, trace_id:YB42AC4D0007-00063C4962CA3069-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:254}, err:0, last_decode_time:1755161784744265, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1683, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:01:21.781303] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [409][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] [sql nio session](*s={this:0x7f98f733d030, session_id:3221590940, trace_id:YB42AC4D0007-00063C4962CA306D-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:247}, err:0, last_decode_time:1755162007739086, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1265, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:01:21.783557] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:21.788667] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:21.788684] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:21.823127] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=8065][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:21.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.823223] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:01:21.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.830673] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:21.848032] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=17] PNIO [ratelimit] time: 1755162081848030, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007615, add_bytes: 0 [2025-08-14 09:01:21.867228] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=7] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:01:21.867261] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=27] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:01:21.876340] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] [sql nio session](*s={this:0x7f98f72d0030, session_id:3221589415, trace_id:YB42AC4D0007-00063C4965FA43DC-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:245}, err:0, last_decode_time:1755161751714366, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1662, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:01:21.923165] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:21.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.976770] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=26][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:01:22.062868] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=738177] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:01:22.464045] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fa190, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa470, time_wheel:0x7f99341cdbf0, thread_pool:0x7f99341cd3d0, is_running:1, time_interval:2.00s, expected_run_time:09:01:22.334, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:22.583461] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fe410, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe6f0, time_wheel:0x7f9922df9d40, thread_pool:0x7f9922df9520, is_running:1, time_interval:2.00s, expected_run_time:09:01:22.432, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.830941] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=6] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:21.992080] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:21.989, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:22.719617] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=726433][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:21.992|207=727.53ms|210=0us|222=2us|total=727.57ms) [2025-08-14 09:01:22.719648] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4389] timer task use too much time(end=1755162082719648, start=1755162081992078, delta=727570) [2025-08-14 09:01:22.719659] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=714170, expected_run_ts_us_=1755162082005487, ObClockGenerator::getRealClock()=1755162082719658, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:22.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.994072] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=74][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:21.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.772081] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a54b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7282a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:22.720087] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [14780][T1001_TimerWK4_SSTableGC][T1001][YB42AC4D0007-00063C4974099C4E-0-0] [lt=19][errcode=0] cost too much time:mds_unit.ipp:int oceanbase::storage::mds::MdsUnit::get_latest(OP &&) const [K = oceanbase::storage::mds::DummyKey, V = oceanbase::storage::ObTa..., (this=|threshold=10.00ms|start at 09:01:21.763|890=1us|891=1us|total=956.81ms) [2025-08-14 09:01:22.720114] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] tx gc loop thread is running(MTL_ID()=1) [2025-08-14 09:01:22.720124] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] try gc retain ctx [2025-08-14 09:01:21.838665] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=763][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:21.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.876365] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=24] [sql nio session](*s={this:0x7f98f72d5030, session_id:3221590200, trace_id:YB42AC4D0007-00063C4965FA43E0-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:248}, err:0, last_decode_time:1755161776777167, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1593, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:01:21.930732] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:22.720204] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=788275][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:21.956635] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=40][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a6178, timer:0x7f9966fddee0, task:0x7f994f1b2fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161720715668, pushed_time:0, delay:900000000}, running_task_token={this:0x7f994e1a65d8, timer:0x7f9966fddee0, task:0x7f994f1bbcc0, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161694824965, pushed_time:1755161925772608, delay:5000000}, total_delay=361240965, threashold=1000000, tenant_id=1) [2025-08-14 09:01:21.830927] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:21.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:22.721447] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=889170][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:21.830|207=890.52ms|210=0us|222=1us|total=890.54ms) [2025-08-14 09:01:22.721469] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4389] timer task use too much time(end=1755162082721468, start=1755162081830925, delta=890543) [2025-08-14 09:01:21.882824] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb3df8, timer:0x7f99333cb180, task:0x7f99333cb1f0, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755161959782853, pushed_time:0, delay:1000000}, running_task_token={this:0x7f9931bb3ed8, timer:0x7f99333cb180, task:0x7f99333cb208, task_type:"N9oceanbase8observer18ObTableLoadService25ObClientTaskAutoAbortTaskE", scheduled_time:1755161959782824, pushed_time:1755162045729074, delay:1000000}, total_delay=122099970, threashold=1000000, tenant_id=1001) [2025-08-14 09:01:22.023225] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:21.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:22.721655] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=697370][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:22.232|207=698.43ms|210=0us|222=1us|total=698.44ms) [2025-08-14 09:01:22.721668] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task use too much time(end=1755162082721668, start=1755162082023223, delta=698445) [2025-08-14 09:01:22.062131] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [571][T1_ArcSender][T1][YB42AC4D0007-00063C495BA99C16-0-0] [lt=25][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:01:22.239034] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:01:22.721698] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=481929][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:01:22.721719] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:01:22.668050] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=63395][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571634a0, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957163780, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:5.00s, expected_run_time:09:01:22.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:21.766863] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=1012384] finish to resize log external storage handler(current_ts=1755162081766855, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-08-14 09:01:22.719675] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:22.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:22.719492] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:22.721751] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=53617][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:01:22.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:22.729082] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [14780][T1001_TimerWK4_SSTableGC][T1001][YB42AC4D0007-00063C4974099C4E-0-0] [lt=12][errcode=0] cost too much time:mds_table_impl.ipp:virtual int oceanbase::storage::mds::MdsTableImpl, oceanbase::st..., (this=|threshold=5.00ms|start at 09:01:20.763|559=0us|total=1.97s) [2025-08-14 09:01:22.720047] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=946528][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:21.772|207=947.97ms|210=1us|222=1us|total=948.00ms) [2025-08-14 09:01:22.729222] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9159][errcode=-4389] timer task use too much time(end=1755162082729221, start=1755162081772068, delta=957153) [2025-08-14 09:01:22.729247] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] task scheduled too delay(scheduled_delay=1968699, expected_run_ts_us_=1755162080760547, ObClockGenerator::getRealClock()=1755162082729247, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f72edc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:22.729274] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f72edc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:22.720146] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=880147][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:21.838|207=881.48ms|210=0us|222=2us|total=881.52ms) [2025-08-14 09:01:22.721729] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:00:41.841, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=08:59:41.841], idx=471, next_allow_print_ts="09:01:11.841") [2025-08-14 09:01:22.729341] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7591][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:01:22.729369] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:01:22.729381] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:01:22.729395] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:01:22.729413] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=08:59:57.942, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:27.942], idx=868, next_allow_print_ts="09:00:27.942") [2025-08-14 09:01:22.729850] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:01:21.999211] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] EASY SLOW: thread: 7f99547f7300, lastrun: 1755162081.750513 cost: 0.248696 loop:26454, slowcnt: 2 [2025-08-14 09:01:22.722858] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=887314] tx gc loop thread is running(MTL_ID()=1001) [2025-08-14 09:01:22.722833] WDIAG [SHARE] operator() (ob_event_history_table_operator.cpp:402) [338][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4012] sync execute sql failed(sql="INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1755161098776838), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1002, 'LS_ID', 1001, 'PREVIOUS', 'LEADER RECONFIRM', 'CURRENT', 'LEADER ACTIVE', '', 0, '', 0, 'reconfirm_stage_cost:1008305, reconfirm_to_active_cost:1021537, config_version:{proposal_id:34, config_seq:34}', '172.77.0.7', 2882)", ret=-4012) [2025-08-14 09:01:22.729295] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] task scheduled too delay(scheduled_delay=1964076, expected_run_ts_us_=1755162080765218, ObClockGenerator::getRealClock()=1755162082729295, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f72edb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:22.729310] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=9150][errcode=-4389] timer task use too much time(end=1755162082729310, start=1755162081838663, delta=890647) [2025-08-14 09:01:22.729866] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:01:22.735020] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=5139][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:01:22.734991] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=5685][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f72edb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:22.735058] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=62][errcode=0] task scheduled too delay(scheduled_delay=1968935, expected_run_ts_us_=1755162080766122, ObClockGenerator::getRealClock()=1755162082735057, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7398490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:22.735105] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7398490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:22.735116] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1966483, expected_run_ts_us_=1755162080768633, ObClockGenerator::getRealClock()=1755162082735116, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7398310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:22.736752] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:22.730529] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=7657] try gc retain ctx [2025-08-14 09:01:22.740322] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=9777] try to retry start_working [2025-08-14 09:01:22.743519] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:22.680306] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EE8F-0-0] [lt=40843] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:01:22.750177] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EE8F-0-0] [lt=69755][errcode=-4016] task executed too delay(delay_time=70326, *p_task_={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:0, time_interval:3.00s, expected_run_time:09:01:25.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:22.750468] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:22.735203] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=86][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7398310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:22.752390] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17155][errcode=0] task scheduled too delay(scheduled_delay=1991390, expected_run_ts_us_=1755162080760998, ObClockGenerator::getRealClock()=1755162082752389, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7289110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:22.752413] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7289110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:22.752425] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1981590, expected_run_ts_us_=1755162080770834, ObClockGenerator::getRealClock()=1755162082752424, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919ac910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:22.752434] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919ac910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:22.752613] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:21.739653] INFO [STORAGE.TRANS] print_stat_ (ob_trans_functor.h:106) [865][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] ls trans functor stat(functor_name="IteratePartCtxAskSchedulerStatusFunctor", ls_id={id:1001}, this={iter_cnt_:4, begin_time_:1755162076778925, finish_time_:1755162081739653, total_cost_time:4960728, single_expired_limit_:100000, single_expired_cnt_:2, total_expired_limit_:3000000}) [2025-08-14 09:01:22.754300] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:22.754330] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:22.754346] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=9] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:22.755909] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:22.745999] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [564][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=1004924] XA scheduler heartbeat task statistics(avg_time=0) [2025-08-14 09:01:22.750148] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=954546] snapshot_traffic_info once [2025-08-14 09:01:22.752451] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1973112, expected_run_ts_us_=1755162080779339, ObClockGenerator::getRealClock()=1755162082752451, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f728ac10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:22.756270] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=3811][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f728ac10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:22.756296] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] task scheduled too delay(scheduled_delay=1953390, expected_run_ts_us_=1755162080802905, ObClockGenerator::getRealClock()=1755162082756295, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919b5690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.802, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:22.756309] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919b5690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.802, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:22.756576] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EE91-0-0] [lt=33] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:01:22.757490] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=23] gc stale ls task succ [2025-08-14 09:01:22.757555] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [14780][T1001_TimerWK4_SSTableGC][T1001][YB42AC4D0007-00063C4974099C4E-0-0] [lt=22][errcode=0] cost too much time:ob_i_tablet_mds_interface.ipp:int oceanbase::storage::ObITabletMdsInterface::get_latest(OP &&, mds::MdsWriter &, mds::TwoPhaseCommitState &, share::SCN &, const int64_t) const [T = oceanbase::storage::ObTabletCreateDeleteMdsUserDat..., (this=|threshold=10.00ms|start at 09:01:20.758|449=1us|458=2us|460=0us|467=2.00s|479=0us|490=3us|total=2.00s) [2025-08-14 09:01:22.756168] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [804][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=41237][errcode=-4389] timer task too much delay in priority queue(token={this:0x7f991b7a41f8, timer:0x7f9922df93b0, task:0x7f9922ffab60, task_type:"N9oceanbase3sql27ObSqlPlanMonitorRecycleTaskE", scheduled_time:1755162050755123, pushed_time:1755162051779970, delay:1000000}, delay_in_priority_queue=1024847, threashold=1000000) [2025-08-14 09:01:22.743571] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:22.756320] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1954846, expected_run_ts_us_=1755162080801474, ObClockGenerator::getRealClock()=1755162082756320, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891970990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:22.760948] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=4618][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891970990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:22.760984] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] task scheduled too delay(scheduled_delay=1956673, expected_run_ts_us_=1755162080804310, ObClockGenerator::getRealClock()=1755162082760983, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728f9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7292010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:22.761002] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728f9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7292010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:22.761017] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] task scheduled too delay(scheduled_delay=1942477, expected_run_ts_us_=1755162080818540, ObClockGenerator::getRealClock()=1755162082761017, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f72eae10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:22.733779] WDIAG [SHARE] load_pure_tenant_info_ (ob_tenant_info_proxy.cpp:430) [870][T1002_DBMSSched][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4012] failed to read(ret=-4012, ret="OB_TIMEOUT", exec_tenant_id=1001, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-08-14 09:01:20.762702] WDIAG [SHARE] calc_db_time (ob_active_session_guard.cpp:190) [14881][TimerWK4_ActiveSessHist][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] calc_db_time catch an unexpected exception and needs attention, possibly due to an irregularity in the precision of rdtsc(stat.session_id_=4611686018429608692, cur_touch_ts=130867998163199340, last_touch_ts=130867875185151412, delta_time=40992682, cur_wait_begin_ts=130867896228058920, cur_event_no=110, stat.retry_wait_event_no_=110, stat.event_no_=0, stat.total_non_idle_wait_time_=50033087, stat.prev_non_idle_wait_time_=10010095, delta_non_idle_wait_time=40022992, stat.total_idle_wait_time_=0, stat.prev_idle_wait_time_=0, delta_idle_wait_time=0, tm_idle_time=14044187) [2025-08-14 09:01:22.762732] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:22.730943] WDIAG [SHARE] exec_insert_update (ob_dml_sql_splicer.cpp:1297) [342][LSMetaTblUp0][T0][YB42AC4D0007-00063C495B39A91A-0-0] [lt=6][errcode=-4012] execute sql failed(ret=-4012, sql=INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.77.0.7', 2882, 2881, 2, '172.77.0.7:2882:1', 0, 0, 'NORMAL', 0, 100, 1, 'zone1', 1, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.77.0.7:2882:1', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1, zone = 'zone1', paxos_replica_number = 1, data_size = 0, required_size = 0, rebuild = 0, learner_list = '') [2025-08-14 09:01:18.774717] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] [thread pool stat](pool_stat={pool:0x564d6e8922c0, last_check_time:1755162078774715, last_idle_time:102477764954, last_thread_count:12}, *pool={name:TimerWK, this:0x564d6e8922c0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:10, threads_idle_time:102477764954, tenant_id:500}, qsize=7, current_idle_time=102477764954) [2025-08-14 09:01:22.760819] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [804][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=4644][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a41f8, timer:0x7f9922df93b0, task:0x7f9922ffab60, task_type:"N9oceanbase3sql27ObSqlPlanMonitorRecycleTaskE", scheduled_time:1755162050755123, pushed_time:1755162051779970, delay:1000000}, delay_in_thread_pool=29992046, threashold=500000) [2025-08-14 09:01:22.766263] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=14] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:01:22.761032] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f72eae10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:22.769086] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8037][errcode=0] task scheduled too delay(scheduled_delay=1946177, expected_run_ts_us_=1755162080822907, ObClockGenerator::getRealClock()=1755162082769085, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f739f290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:22.769112] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f739f290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:22.769429] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=3560] current monitor number(seq_=-1) [2025-08-14 09:01:22.762411] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4471] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:21.719120] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=29] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:22.771454] WDIAG ev_invoke_pending (ev.c:2136) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1050722][errcode=0] epoll handler too much time :1052355 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:21.257220] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=12] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:22.770847] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [14780][T1001_TimerWK4_SSTableGC][T1001][YB42AC4D0007-00063C4974099C4E-0-0] [lt=23][errcode=0] cost too much time:ob_i_tablet_mds_interface.cpp:int oceanbase::storage::ObITabletMdsInterface::get_latest_tablet_status(oceanbase::storage::ObTabletCreateDeleteMdsUserData &, mds::MdsWriter &, mds::TwoPhaseCommitState &, share::SCN &, const int64_t)..., (this=|threshold=10.00ms|start at 09:01:20.758|97=1us|total=2.01s) [2025-08-14 09:01:22.771309] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [804][T1002_TimerWK0][T1002][YB42AC4D0007-00063C495F2A1959-0-0] [lt=3159][errcode=-4389] timer task too much delay in priority queue(token={this:0x7f991b7a5c38, timer:0x7f9935f93f20, task:0x7f99200fe2d0, task_type:"N9oceanbase8datadict17ObDataDictServiceE", scheduled_time:1755162050755132, pushed_time:1755162051779973, delay:1000000}, delay_in_priority_queue=1024841, threashold=1000000) [2025-08-14 09:01:22.779509] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [804][T1002_TimerWK0][T1002][YB42AC4D0007-00063C495F2A1959-0-0] [lt=8186][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a5c38, timer:0x7f9935f93f20, task:0x7f99200fe2d0, task_type:"N9oceanbase8datadict17ObDataDictServiceE", scheduled_time:1755162050755132, pushed_time:1755162051779973, delay:1000000}, delay_in_thread_pool=30984021, threashold=500000) [2025-08-14 09:01:22.771733] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=1043964] gc stale ls task succ [2025-08-14 09:01:22.771477] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:22.769124] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1903766, expected_run_ts_us_=1755162080865358, ObClockGenerator::getRealClock()=1755162082769124, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7289e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:22.779689] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10547][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7289e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:22.779713] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] task scheduled too delay(scheduled_delay=1912939, expected_run_ts_us_=1755162080866773, ObClockGenerator::getRealClock()=1755162082779712, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7288f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.866, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:20.779435] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=32] [sql nio session](*s={this:0x7f98f72ab030, session_id:3221589435, trace_id:YB42AC4D0007-00063C4965DA1F35-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:246}, err:0, last_decode_time:1755161762721798, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:2456, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:01:22.779925] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=1997429] [sql nio session](*s={this:0x7f98f72a9030, session_id:3221590528, trace_id:YB42AC4D0007-00063C4965DA1F34-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:250}, err:0, last_decode_time:1755161762715415, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:205, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:01:22.783259] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:22.820251] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:22.821556] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:22.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:22.828251] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:01:22.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:22.855646] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=14] PNIO [ratelimit] time: 1755162082855644, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007614, add_bytes: 0 [2025-08-14 09:01:22.856695] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c0b90, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571c0e70, time_wheel:0x7f9959fef990, thread_pool:0x7f9959fef170, is_running:1, time_interval:3.00s, expected_run_time:09:01:22.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:23.669412] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=7293][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:01:23.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:22.771784] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=1510759][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162081258718], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:22.771756] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1042744][errcode=0] epoll handler too much time :1044347 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:23.718424] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=945203][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162081258718], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:23.718451] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162081258718]}) [2025-08-14 09:01:22.777362] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=14] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:01:22.788785] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=17] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:22.789558] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:22.821741] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:22.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:23.718551] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=927587][errcode=0] epoll handler too much time :929002 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:22.870584] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=0] get_fetch_log_dst failed or invalid(ret=0, palf_id=1, self="172.77.0.7:2882", fetch_log_dst="172.77.0.7:2882") [2025-08-14 09:01:23.718585] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=846715] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:01:23.718567] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=895456][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:22.821|207=896.81ms|210=0us|222=12us|total=896.88ms) [2025-08-14 09:01:23.718622] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] get_fetch_log_dst failed or invalid(ret=0, palf_id=1001, self="172.77.0.7:2882", fetch_log_dst="172.77.0.7:2882") [2025-08-14 09:01:23.718634] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:01:23.718640] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=59][errcode=-4389] timer task use too much time(end=1755162083718626, start=1755162082821740, delta=896886) [2025-08-14 09:01:22.921619] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:22.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:23.718686] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=795852][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:22.921|207=797.06ms|210=1us|222=1us|total=797.08ms) [2025-08-14 09:01:23.718698] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task use too much time(end=1755162083718698, start=1755162082921618, delta=797080) [2025-08-14 09:01:21.745038] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:23.718849] WDIAG ev_invoke_pending (ev.c:2136) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1970762][errcode=0] epoll handler too much time :1973821 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:23.718875] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:21.750526] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5438] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:23.719039] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1965524][errcode=0] epoll handler too much time :1968524 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:23.719053] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:21.747162] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6119] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:23.719111] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1968952][errcode=0] epoll handler too much time :1971958 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:23.719124] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:23.028451] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=794471][errcode=0] EASY SLOW: thread: 7f99543f7300, lastrun: 1755162081.745026 cost: 1.283423 loop:26434, slowcnt: 12 [2025-08-14 09:01:22.778395] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=980874] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:01:23.063559] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=13] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:01:22.777984] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=990897] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:01:23.235222] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:01:23.719546] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=483581][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:01:23.719569] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:01:23.719579] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:00:41.841, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=08:59:41.841], idx=471, next_allow_print_ts="09:01:11.841") [2025-08-14 09:01:22.723170] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=3485][errcode=0] task scheduled too delay(scheduled_delay=633853, expected_run_ts_us_=1755162082089316, ObClockGenerator::getRealClock()=1755162082723170, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:22.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:23.719594] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:01:23.719603] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=994925][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:22.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:23.719613] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:01:23.719616] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:22.723|207=996.44ms|210=0us|222=2us|total=996.46ms) [2025-08-14 09:01:23.719623] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:01:23.719627] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task use too much time(end=1755162083719627, start=1755162082723169, delta=996458) [2025-08-14 09:01:23.719636] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:01:23.719636] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=0] task scheduled too delay(scheduled_delay=1515338, expected_run_ts_us_=1755162082204297, ObClockGenerator::getRealClock()=1755162083719635, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:22.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:23.719646] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:22.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:23.719647] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:00:52.998, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:22.998], idx=771, next_allow_print_ts="09:01:22.998") [2025-08-14 09:01:23.719670] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=08:59:57.942, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:27.942], idx=868, next_allow_print_ts="09:00:27.942") [2025-08-14 09:01:23.719670] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] task scheduled too delay(scheduled_delay=980525, expected_run_ts_us_=1755162082739145, ObClockGenerator::getRealClock()=1755162083719670, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:22.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:23.719681] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:22.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:23.719690] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=964202, expected_run_ts_us_=1755162082755487, ObClockGenerator::getRealClock()=1755162083719690, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:22.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:23.719698] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:22.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:23.719801] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=1519022] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3688) [2025-08-14 09:01:23.719811] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:01:23.719832] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:01:23.719833] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:23.719843] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3689) [2025-08-14 09:01:23.719847] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:01:23.719844] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:01:23.719855] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:01:23.719953] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3691] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:23.718443] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=945237] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:23.720033] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:23.720042] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3690) [2025-08-14 09:01:23.720048] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:01:23.720305] WDIAG [SHARE] load_tenant_info (ob_tenant_info_proxy.cpp:385) [870][T1002_DBMSSched][T1002][Y0-0000000000000000-0-0] [lt=27266][errcode=-4012] failed to load purge tenant info(ret=-4012, ret="OB_TIMEOUT", tenant_id=1002, for_update=false) [2025-08-14 09:01:23.720353] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8079] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:21.748878] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=10] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:12632581, max_lsn:8461098908, max_scn:{val:1755161719775179002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:8214679957}, reuse_lsn:{lsn:8461098908}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:8419155868}}, last_submit_lsn:{lsn:8461098786}, last_submit_end_lsn:{lsn:8461098908}, last_submit_log_id:12632581, last_submit_log_pid:45, max_flushed_lsn:{lsn:8461098786}, max_flushed_end_lsn:{lsn:8461098908}, max_flushed_log_pid:45, committed_end_lsn:{lsn:8461098908}, last_slide_log_id:12632581, last_slide_scn:{val:1755161719775179002, v:0}, last_slide_lsn:{lsn:8461098786}, last_slide_end_lsn:{lsn:8461098908}, last_slide_log_pid:45, last_slide_log_accum_checksum:1091335677, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f992979c530}, [StateMgr]={this:0x7f99297a09b0, self:"172.77.0.7:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:46}, leader:"0.0.0.0:0", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:8461098908}, scan_disk_log_finished:true, last_check_start_id:12632582, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161899748977, palf_role_change_cb:0x7f99297a7470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:45, prev:{version:1, config:{config_version:{proposal_id:43, config_seq:41}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:45, config_seq:42}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:41, prev_lsn:{lsn:8461098218}, prev_mode_pid:45}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:41, prev_lsn:{lsn:8461098218}, prev_end_lsn:{lsn:8461098377}, prev_mode_pid:45}, reconfig_barrier:{prev_log_proposal_id:41, prev_lsn:{lsn:8461098218}, prev_end_lsn:{lsn:8461098377}, prev_mode_pid:45}, persistent_config_version:{proposal_id:45, config_seq:42}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755162077751212, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755162077751212, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f992979dc70}, [ModeMgr]={palf_id:1, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:45, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:46, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:46, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:1753719139394335001, v:0}, min_block_id:93, min_block_min_scn:{val:1753712284262753001, v:0}, base_lsn_for_block_gc:{lsn:8186781696}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:46}, log_config_meta:{version:3, proposal_id:45, prev:{version:1, config:{config_version:{proposal_id:43, config_seq:41}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:45, config_seq:42}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:41, prev_lsn:{lsn:8461098218}, prev_mode_pid:45}, log_snapshot_meta:{version:2, base_lsn:{lsn:8186781696}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:46, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:655360}, readable_log_tail:{lsn:655360}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8522305536}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/meta", dir_fd:214, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66449408, block_header_serialize_buf_:0x7f99297a2bb8, flashback_version:0}, log_storage:{log_tail:{lsn:8461098908}, readable_log_tail:{lsn:8461098908}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8455200768}, min_scn:{val:1755160384851871002, v:0}, curr_block_id:126, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:215, min_block_id:93, max_block_id:127, curr_writable_block_id:126}, logical_block_size_:67104768, curr_block_writable_size_:61206628, block_header_serialize_buf_:0x7f99297a48b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1755161900739661, this:0x7f99297a10f0}, [Reconfirm]={palf_id:1, self:"172.77.0.7:2882", state:"FETCH_MAX_LOG_LSN", new_proposal_id:46, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"172.77.0.7:2882", majority_max_accept_pid:45, majority_max_lsn:{lsn:8461098908}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:1755161954782466, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f99297a0b30}) [2025-08-14 09:01:23.720445] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=3983204] [thread pool stat](pool_stat={pool:0x7f99667cbcf0, last_check_time:1755162083720444, last_idle_time:34692939658, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x7f99667cbcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:34692939658, tenant_id:500}, qsize=-1, current_idle_time=34692939658) [2025-08-14 09:01:23.720473] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=27] [thread pool stat](pool_stat={pool:0x7f99667ccaf0, last_check_time:1755162083720473, last_idle_time:26467359812, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x7f99667ccaf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:26467359812, tenant_id:500}, qsize=-1, current_idle_time=26467359812) [2025-08-14 09:01:23.720484] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] [thread pool stat](pool_stat={pool:0x7f9969de56b0, last_check_time:1755162083720483, last_idle_time:102938681541, last_thread_count:17}, *pool={name:TimerWK, this:0x7f9969de56b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:16, threads_idle_time:102938681541, tenant_id:1}, qsize=8, current_idle_time=102938681541) [2025-08-14 09:01:23.720493] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] [thread pool stat](pool_stat={pool:0x7f99356b0e70, last_check_time:1755162083720493, last_idle_time:102366343170, last_thread_count:15}, *pool={name:TimerWK, this:0x7f99356b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:13, threads_idle_time:102366343170, tenant_id:1001}, qsize=10, current_idle_time=102366343170) [2025-08-14 09:01:23.720508] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=1968615] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161956777957, last_flushed_end_lsn:{lsn:8461098908}}]) [2025-08-14 09:01:23.720524] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=15] current log_task status(palf_id=1, self="172.77.0.7:2882", log_id=12632582, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161956777957, last_flushed_end_lsn:{lsn:8461098908}}]) [2025-08-14 09:01:22.779726] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7288f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:20.866, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.721545] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=940391][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:22.779|207=941.83ms|210=1us|222=1us|total=941.86ms) [2025-08-14 09:01:23.029448] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1770][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:23.151, task_priority:1, with_handle_protected:1}) [651]OB_MALLOC COST TOO MUCH TIME, cost_time=919320, tenant_id=500, label=CommonArray, ctx_id=0, prio=0, size=15872, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=3], seq:[0, 1] [2025-08-14 09:01:22.719695] EDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=724523][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1015349, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=41100, FORMAT_END=974246, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ed35c 0x97aead0 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:23.721898] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=691400][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:23.294|207=692.45ms|210=0us|222=1us|total=692.46ms) [2025-08-14 09:01:23.721916] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] timer task use too much time(end=1755162083721916, start=1755162083029446, delta=692470) [2025-08-14 09:01:23.721936] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=429585, expected_run_ts_us_=1755162083292349, ObClockGenerator::getRealClock()=1755162083721935, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:23.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:23.721950] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:23.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:23.718464] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162081258718]}) [2025-08-14 09:01:23.722132] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3659][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:01:23.722162] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=26] LogLoopThread round_cost_time(us)(round_cost_time=1971700) [2025-08-14 09:01:23.736265] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:23.720501] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] [thread pool stat](pool_stat={pool:0x7f99230b0e70, last_check_time:1755162083720501, last_idle_time:102534969155, last_thread_count:17}, *pool={name:TimerWK, this:0x7f99230b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:16, threads_idle_time:102534969155, tenant_id:1002}, qsize=11, current_idle_time=102534969155) [2025-08-14 09:01:22.719695] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=724523][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:21.994|207=725.62ms|210=0us|222=1us|total=1.74s) [2025-08-14 09:01:23.736611] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1015349][errcode=-4389] timer task use too much time(end=1755162083736609, start=1755162081994070, delta=1742539) [2025-08-14 09:01:23.736628] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1491992, expected_run_ts_us_=1755162082244633, ObClockGenerator::getRealClock()=1755162083736627, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:22.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:23.736647] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:22.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:23.736665] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1003134, expected_run_ts_us_=1755162082733531, ObClockGenerator::getRealClock()=1755162083736665, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:22.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:23.736675] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:22.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:22.760916] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17316][errcode=0] epoll handler too much time :17351 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:23.736750] INFO [SQL.PC] need_check_schema_version (ob_plan_cache_value.cpp:2334) [740][T1001_Occam][T1001][YB42AC4D0007-00063C4966399E1E-0-0] [lt=20] need check schema(new_schema_version=1753074886385104, cached_tenant_schema_version=1753074886385104, is_contain_synonym()=false, contain_sys_name_table_=true, is_contain_tmp_tbl()=false, is_contain_sys_pl_object()=false, need_check=true, constructed_sql_=select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = ?) [2025-08-14 09:01:23.738498] WDIAG [COORDINATOR] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [657][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8214][errcode=0] cost too much time:ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_(), (this=|threshold=1.00s|start at 09:01:23.738|total=3.97s) [2025-08-14 09:01:23.721573] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4389] timer task use too much time(end=1755162083721573, start=1755162082779711, delta=941862) [2025-08-14 09:01:23.728617] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1539] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:23.741279] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12642][errcode=0] epoll handler too much time :12670 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:22.760775] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:980) [699][T1001_DupTbLeas][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4012] failed to read(ret=-4012, ret="OB_TIMEOUT", exec_tenant_id=1, sql=select * from __all_ls_status where tenant_id = 1001 and flag like "%DUPLICATE%" order by ls_id limit 1) [2025-08-14 09:01:23.741718] WDIAG [COORDINATOR] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [657][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=0] cost too much time:ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_failure(), (this=|threshold=1.00s|start at 09:01:23.741|total=120.02s) [2025-08-14 09:01:23.741736] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [657][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:22.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:23.741751] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [657][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:01:23.741|total=120.02s) [2025-08-14 09:01:23.740470] WDIAG [SHARE] load_tenant_info (ob_tenant_info_proxy.cpp:356) [870][T1002_DBMSSched][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4012] failed to get tenant info(ret=-4012, ret="OB_TIMEOUT", tenant_id=1002, proxy=0x564d6cd536e0, for_update=false) [2025-08-14 09:01:23.742061] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=974348] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:23.742340] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:23.742918] WDIAG [SHARE] get_tenant_role (ob_tenant_info_proxy.cpp:259) [870][T1002_DBMSSched][T1002][Y0-0000000000000000-0-0] [lt=1348][errcode=-4012] fail to load_tenant_info(ret=-4012, ret="OB_TIMEOUT", tenant_id=1002) [2025-08-14 09:01:23.742956] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EE81-0-0] [lt=29] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-08-14 09:01:22.737214] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=623][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1015610, owner: ObLog, click_count: 5, time dist:[FORMAT_END=999078, ObMalloc#start=2, ObMalloc#end=16528, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:22.761692] EDIAG [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1003017, owner: ObLog, click_count: 5, time dist:[FORMAT_END=975363, ObMalloc#start=1, ObMalloc#end=27649, ALLOC_END=3, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0xfebfd41 0x94edd90 0x94edb97 0x97a8fbd 0x986bc46 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:23.744469] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:23.746457] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:23.750265] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:23.753255] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:22.737214] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=623] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:23.754399] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1015610][errcode=0] epoll handler too much time :1017192 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:23.754417] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:23.754477] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:23.754503] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:23.754519] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:23.754528] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:23.754583] INFO [PALF] check_parent_health (log_config_mgr.cpp:2875) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=7] re_register_parent reason(palf_id=1, self="172.77.0.7:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1755162077751212, last_first_register_time_us=1755158728414267, register_time_us=1755162077751212, curr_time_us=1755162083722182) [2025-08-14 09:01:23.756671] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:22.777440] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:23.759130] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=980195] temporary file module use no blocks [2025-08-14 09:01:23.759141] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:23.759153] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:23.759165] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43622984217025, flush_io_finished_ret_:0, flush_io_finished_round_:26265, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:23.740501] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18888][errcode=0] task scheduled too delay(scheduled_delay=2411401, expected_run_ts_us_=1755162081329099, ObClockGenerator::getRealClock()=1755162083740500, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7281b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.751101] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24995] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:01:23.759293] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:23.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:23.759395] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=10] gc stale ls task succ [2025-08-14 09:01:23.759151] WDIAG [SHARE] is_standby_tenant (ob_tenant_info_proxy.cpp:275) [870][T1002_DBMSSched][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4012] fail to get tenant_role(ret=-4012, ret="OB_TIMEOUT", tenant_id=1002) [2025-08-14 09:01:23.759208] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18677][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7281b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.761665] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=2443][errcode=0] task scheduled too delay(scheduled_delay=2035824, expected_run_ts_us_=1755162081725840, ObClockGenerator::getRealClock()=1755162083761664, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72756b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919b5510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.761687] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72756b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919b5510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.761699] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=2037566, expected_run_ts_us_=1755162081724133, ObClockGenerator::getRealClock()=1755162083761699, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7283b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.724, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.761710] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7283b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.724, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.742795] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [804][T1002_TimerWK0][T1002][YB42AC4D0007-00063C495F2A195B-0-0] [lt=17][errcode=-4389] timer task too much delay in priority queue(token={this:0x7f991b7a7918, timer:0x7f99329e19d0, task:0x7f98f55f1250, task_type:"N9oceanbase5share24ObVectorIndexHistoryTaskE", scheduled_time:1755162050762295, pushed_time:1755162051779984, delay:10000000}, delay_in_priority_queue=1017689, threashold=1000000) [2025-08-14 09:01:23.763177] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [804][T1002_TimerWK0][T1002][YB42AC4D0007-00063C495F2A195B-0-0] [lt=20350][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a7918, timer:0x7f99329e19d0, task:0x7f98f55f1250, task_type:"N9oceanbase5share24ObVectorIndexHistoryTaskE", scheduled_time:1755162050762295, pushed_time:1755162051779984, delay:10000000}, delay_in_thread_pool=30999551, threashold=500000) [2025-08-14 09:01:23.763205] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8137] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:01:23.763262] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:01:23.763304] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:01:23.763354] INFO [COMMON] print_io_status (ob_io_manager.cpp:2639) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:0, norm_iops:0, bw:455, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:8907, total:8917) [2025-08-14 09:01:23.763684] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] task executed too delay(delay_time=955599, *p_task_={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:0, time_interval:500.00ms, expected_run_time:09:01:23.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:22.835066] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=5677][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:22.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:23.763902] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=927430][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:22.835|207=928.83ms|210=0us|222=1us|total=928.85ms) [2025-08-14 09:01:23.763916] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] timer task use too much time(end=1755162083763916, start=1755162082835065, delta=928851) [2025-08-14 09:01:22.798210] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=952857] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:01:23.743116] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:23.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:23.763995] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=20844][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:23.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:23.764258] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7124] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:23.764697] WDIAG [SERVER] check_tenant (ob_dbms_sched_job_master.cpp:497) [870][T1002_DBMSSched][T1002][Y0-0000000000000000-0-0] [lt=2493][errcode=-4012] check is standby tenant failed(ret=-4012, tenant_id_=1002) [2025-08-14 09:01:23.764711] INFO [SERVER] check_tenant (ob_dbms_sched_job_master.cpp:506) [870][T1002_DBMSSched][T1002][Y0-0000000000000000-0-0] [lt=11] check all tenants(ret=-4012) [2025-08-14 09:01:23.762835] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=12] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:23.761721] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=2032348, expected_run_ts_us_=1755162081729372, ObClockGenerator::getRealClock()=1755162083761721, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a36b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7283c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.764856] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=3129][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a36b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7283c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.764878] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] task scheduled too delay(scheduled_delay=2025972, expected_run_ts_us_=1755162081738906, ObClockGenerator::getRealClock()=1755162083764878, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919911b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f989196cc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.764889] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919911b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f989196cc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.764899] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=2026607, expected_run_ts_us_=1755162081738292, ObClockGenerator::getRealClock()=1755162083764899, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d40b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f72ec190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.764911] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d40b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f72ec190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.764920] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=2030511, expected_run_ts_us_=1755162081734408, ObClockGenerator::getRealClock()=1755162083764920, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f728b390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.764929] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f728b390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.764939] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=2026808, expected_run_ts_us_=1755162081738130, ObClockGenerator::getRealClock()=1755162083764938, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f728b510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.764957] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f728b510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.764976] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] task scheduled too delay(scheduled_delay=2026689, expected_run_ts_us_=1755162081738286, ObClockGenerator::getRealClock()=1755162083764975, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f728ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.764985] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f728ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.764995] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=2026475, expected_run_ts_us_=1755162081738519, ObClockGenerator::getRealClock()=1755162083764995, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f728af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765004] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f728af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765014] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=2020489, expected_run_ts_us_=1755162081744524, ObClockGenerator::getRealClock()=1755162083765014, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919b9210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.744, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765024] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919b9210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.744, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765034] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=2022945, expected_run_ts_us_=1755162081742088, ObClockGenerator::getRealClock()=1755162083765034, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919aaf90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765043] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919aaf90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765052] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=2024751, expected_run_ts_us_=1755162081740301, ObClockGenerator::getRealClock()=1755162083765052, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919aba10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765065] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919aba10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765076] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=2019616, expected_run_ts_us_=1755162081745459, ObClockGenerator::getRealClock()=1755162083765075, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919ad390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765085] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919ad390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765094] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=2017052, expected_run_ts_us_=1755162081748042, ObClockGenerator::getRealClock()=1755162083765094, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f989196cb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765106] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f989196cb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765116] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=2021944, expected_run_ts_us_=1755162081743171, ObClockGenerator::getRealClock()=1755162083765115, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f989196b810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765127] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f989196b810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765140] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=2013192, expected_run_ts_us_=1755162081751947, ObClockGenerator::getRealClock()=1755162083765139, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919ad210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.751, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765149] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919ad210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.751, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765158] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=2001616, expected_run_ts_us_=1755162081763541, ObClockGenerator::getRealClock()=1755162083765158, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891825b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765178] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891825b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765191] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=1986075, expected_run_ts_us_=1755162081779116, ObClockGenerator::getRealClock()=1755162083765191, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919acf10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765200] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919acf10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765209] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1992278, expected_run_ts_us_=1755162081772930, ObClockGenerator::getRealClock()=1755162083765209, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7f0790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765218] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7f0790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765228] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1981706, expected_run_ts_us_=1755162081783521, ObClockGenerator::getRealClock()=1755162083765227, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919abe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765247] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919abe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765257] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1980391, expected_run_ts_us_=1755162081784865, ObClockGenerator::getRealClock()=1755162083765256, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919b5090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765264] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919b5090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765272] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1979509, expected_run_ts_us_=1755162081785763, ObClockGenerator::getRealClock()=1755162083765272, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199fe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919acd90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765280] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199fe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919acd90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765292] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1982539, expected_run_ts_us_=1755162081782752, ObClockGenerator::getRealClock()=1755162083765291, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e80b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891973b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765307] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e80b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891973b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765317] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1975786, expected_run_ts_us_=1755162081789530, ObClockGenerator::getRealClock()=1755162083765317, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a31b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f72ede10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765325] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a31b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f72ede10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765334] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1959134, expected_run_ts_us_=1755162081806200, ObClockGenerator::getRealClock()=1755162083765334, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919ab890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.806, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765344] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919ab890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.806, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765353] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1957908, expected_run_ts_us_=1755162081807444, ObClockGenerator::getRealClock()=1755162083765353, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f989196a610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765362] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f989196a610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765370] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1960990, expected_run_ts_us_=1755162081804380, ObClockGenerator::getRealClock()=1755162083765370, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7280210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765381] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7280210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765389] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1964950, expected_run_ts_us_=1755162081800439, ObClockGenerator::getRealClock()=1755162083765389, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7288510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.800, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765399] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7288510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.800, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765407] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1960063, expected_run_ts_us_=1755162081805344, ObClockGenerator::getRealClock()=1755162083765407, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728e330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7290690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.805, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765416] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728e330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7290690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.805, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765426] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1961700, expected_run_ts_us_=1755162081803726, ObClockGenerator::getRealClock()=1755162083765426, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7f1b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.803, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765435] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7f1b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.803, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765444] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1940827, expected_run_ts_us_=1755162081824616, ObClockGenerator::getRealClock()=1755162083765443, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f739e990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.824, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765452] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f739e990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.824, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765461] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1933127, expected_run_ts_us_=1755162081832333, ObClockGenerator::getRealClock()=1755162083765461, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891969b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.832, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765471] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891969b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.832, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765481] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1900551, expected_run_ts_us_=1755162081864930, ObClockGenerator::getRealClock()=1755162083765481, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919ade10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.864, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765490] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919ade10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.864, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765502] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1900413, expected_run_ts_us_=1755162081865089, ObClockGenerator::getRealClock()=1755162083765502, ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891921990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765511] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891921990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765522] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1900100, expected_run_ts_us_=1755162081865422, ObClockGenerator::getRealClock()=1755162083765522, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919adc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765531] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919adc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765539] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1899998, expected_run_ts_us_=1755162081865541, ObClockGenerator::getRealClock()=1755162083765539, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919adb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765547] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919adb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765556] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1887445, expected_run_ts_us_=1755162081878111, ObClockGenerator::getRealClock()=1755162083765556, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72865b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7288990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.878, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765567] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72865b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7288990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.878, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765577] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1865737, expected_run_ts_us_=1755162081899839, ObClockGenerator::getRealClock()=1755162083765576, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728de30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7290b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.899, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765585] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728de30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7290b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.899, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765595] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1862739, expected_run_ts_us_=1755162081902855, ObClockGenerator::getRealClock()=1755162083765595, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7ee210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.902, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765603] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7ee210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:21.902, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765616] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1653960, expected_run_ts_us_=1755162082111656, ObClockGenerator::getRealClock()=1755162083765616, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7289d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.111, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765624] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7289d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.111, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765643] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1034920, expected_run_ts_us_=1755162082730722, ObClockGenerator::getRealClock()=1755162083765643, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891968390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765653] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891968390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765662] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1034452, expected_run_ts_us_=1755162082731210, ObClockGenerator::getRealClock()=1755162083765662, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919ac790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765673] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919ac790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765682] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1033435, expected_run_ts_us_=1755162082732246, ObClockGenerator::getRealClock()=1755162083765682, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891991430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f989196ce10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765691] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891991430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f989196ce10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765699] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1023293, expected_run_ts_us_=1755162082742406, ObClockGenerator::getRealClock()=1755162083765699, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919aac90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765707] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919aac90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765718] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1018012, expected_run_ts_us_=1755162082747705, ObClockGenerator::getRealClock()=1755162083765718, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919b9510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765727] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919b9510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765735] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1023199, expected_run_ts_us_=1755162082742535, ObClockGenerator::getRealClock()=1755162083765734, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919ad090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765743] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919ad090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765752] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1018446, expected_run_ts_us_=1755162082747306, ObClockGenerator::getRealClock()=1755162083765752, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e62b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891972910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765762] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e62b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891972910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765771] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1020070, expected_run_ts_us_=1755162082745700, ObClockGenerator::getRealClock()=1755162083765770, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73940b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7ee690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765779] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73940b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7ee690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765796] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1008297, expected_run_ts_us_=1755162082757499, ObClockGenerator::getRealClock()=1755162083765796, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7eeb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765805] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7eeb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765816] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1008171, expected_run_ts_us_=1755162082757645, ObClockGenerator::getRealClock()=1755162083765816, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73936b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7eec90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765825] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73936b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7eec90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765834] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1006019, expected_run_ts_us_=1755162082759815, ObClockGenerator::getRealClock()=1755162083765834, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7f0c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765843] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7f0c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765860] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] task scheduled too delay(scheduled_delay=999310, expected_run_ts_us_=1755162082766550, ObClockGenerator::getRealClock()=1755162083765860, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919ac010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765873] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919ac010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765882] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1005342, expected_run_ts_us_=1755162082760539, ObClockGenerator::getRealClock()=1755162083765881, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72765b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919b5e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765890] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72765b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919b5e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765899] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=997201, expected_run_ts_us_=1755162082768698, ObClockGenerator::getRealClock()=1755162083765899, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919ac610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765908] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919ac610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765919] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=999551, expected_run_ts_us_=1755162082766368, ObClockGenerator::getRealClock()=1755162083765919, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891969410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765928] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891969410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765937] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1004244, expected_run_ts_us_=1755162082761692, ObClockGenerator::getRealClock()=1755162083765937, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7f0d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765946] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7f0d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765954] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=991402, expected_run_ts_us_=1755162082774552, ObClockGenerator::getRealClock()=1755162083765954, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919aab10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:22.774, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:23.765973] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:01:22.761692] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=11] get_max_applied_scn(scn={val:1755161719775179002, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:45, palf_committed_end_lsn_:{lsn:8461098908}, palf_committed_end_scn_:{val:1755161719775179002, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161719775179002, v:0}}) [2025-08-14 09:01:23.766344] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=16] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:01:23.767278] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=35] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:23.767294] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] temporary file module use no blocks [2025-08-14 09:01:23.767300] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:23.767311] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=9] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:23.767324] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43622985206996, flush_io_finished_ret_:0, flush_io_finished_round_:26259, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:23.722143] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:01:23.769867] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=47631] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:23.769878] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162083769862], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:23.769904] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162083769862], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:23.769919] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162083769862]}) [2025-08-14 09:01:23.769928] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162083769862]}) [2025-08-14 09:01:23.770107] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:01:23.770117] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:01:23.770130] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162083770101], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:23.770140] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162083770101]}) [2025-08-14 09:01:23.770147] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162083770101]}) [2025-08-14 09:01:23.771296] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=15] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:23.771315] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16] temporary file module use no blocks [2025-08-14 09:01:23.771321] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:23.771332] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:23.771343] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43622985209250, flush_io_finished_ret_:0, flush_io_finished_round_:26233, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:23.771797] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=7815] gc stale ls task succ [2025-08-14 09:01:23.784169] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:23.788804] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:23.804539] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:01:23.818702] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:01:23.818772] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:01:23.829183] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:01:23.863275] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=14] PNIO [ratelimit] time: 1755162083863273, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007629, add_bytes: 0 [2025-08-14 09:01:23.926113] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [741][T1001_MultiVers][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:01:23.964060] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:01:23.992827] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:01:23.995758] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:01:24.119674] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=654808] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:01:24.220030] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:01:24.220065] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:01:24.220083] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:01:24.220102] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:00:41.841, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=08:59:41.841], idx=471, next_allow_print_ts="09:01:11.841") [2025-08-14 09:01:24.220118] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:01:24.220142] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:01:24.220140] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:24.220170] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:01:24.220191] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:01:24.220208] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:00:52.998, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:22.998], idx=771, next_allow_print_ts="09:01:22.998") [2025-08-14 09:01:24.220230] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=08:59:57.942, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:27.942], idx=868, next_allow_print_ts="09:00:27.942") [2025-08-14 09:01:24.270398] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=6] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:24.270427] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162084270395], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:24.270444] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162084270395]}) [2025-08-14 09:01:24.473969] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:01:24.584589] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:01:24.670366] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:01:24.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:24.718630] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=928342] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:24.718656] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=939736] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:01:24.719292] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:24.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:24.719411] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:24.719628] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:24.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:24.719960] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:24.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:24.719995] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:24.720273] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7f1e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:24.718, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:22.730951] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [338][Occam][T0][Y0-0000000000000000-0-0] [lt=8104][errcode=-4389] run this task use too much time(*p_task_={this:0x7f98f7295430, caller:ob_event_history_table_operator.cpp:add_event_to_timer_:419, function_type:ZN9oceanbase5share27ObEventHistoryTableOperator19add_event_to_timer_ERKNS_6common11ObSqlStringEE6$_2335, timer_running_flag:1, total_running_count:215, func_shared_ptr_.ptr:0x7f98f7299a10, time_wheel:0x7f99667c9d00, thread_pool:0x7f99667c94e0, is_running:0, time_interval:15.00s, expected_run_time:09:00:58.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:24.727265] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:24.730022] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd110, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd3f0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:10.00s, expected_run_time:09:01:24.726, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:24.730346] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f739fd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:24.726, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:24.730408] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=60][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a67b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7397290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:24.728, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:24.220406] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:01:24.734274] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=513085][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:01:24.734309] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:01:23.738368] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=816785] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:01:24.734416] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=994541] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:01:24.734430] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:24.734442] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:23.766425] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:663) [699][T1001_DupTbLeas][T1001][Y0-0000000000000000-0-0] [lt=979439][errcode=-4012] fail to inner get ls status info(ret=-4012, ret="OB_TIMEOUT", sql=select * from __all_ls_status where tenant_id = 1001 and flag like "%DUPLICATE%" order by ls_id limit 1, tenant_id=1001, exec_tenant_id=1, need_member_list=false) [2025-08-14 09:01:24.734567] INFO [PALF] register_parent_ (log_config_mgr.cpp:2707) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=21] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.77.0.7:2882", reason=1, leader="172.77.0.7:2882", register_time_us=1755162084734527, last_submit_register_req_time_us=1755162084734527, register_parent_reason=1) [2025-08-14 09:01:24.734602] INFO [PALF] check_parent_health (log_config_mgr.cpp:2884) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=32] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.77.0.7:2882") [2025-08-14 09:01:24.734326] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2964371] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161101770923, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755161572801635) [2025-08-14 09:01:24.735037] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=687] get_max_applied_scn(scn={val:1755161547738516001, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:50, palf_committed_end_lsn_:{lsn:5282737655}, palf_committed_end_scn_:{val:1755161547738516001, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161547738516001, v:0}}) [2025-08-14 09:01:24.735067] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] get_min_unreplayed_log_info(lsn={lsn:5282737655}, scn={val:1755100947825033001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161624848121, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:13063726, ret_code_:0}}, next_to_submit_lsn_:{lsn:5282737655}, next_to_submit_scn_:{val:1755100947825033001, v:0}, base_lsn_:{lsn:4831543296}, base_scn_:{val:1755063445733337001, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7605000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:5282737655}, end_lsn:{lsn:5282737655}, read_buf:{buf_len_:3747840, buf_:0x7f98f7605000}, block_size:67104768, log_storage_:0x7f993e529bf0, log_storage_:{log_tail:{lsn:5282737655}, readable_log_tail:{lsn:5282737655}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:5234171904}, min_scn:{val:1755152244802535004, v:0}, curr_block_id:78, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:206, min_block_id:42, max_block_id:79, curr_writable_block_id:78}, logical_block_size_:67104768, curr_block_writable_size_:18539017, block_header_serialize_buf_:0x7f993e52a8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:85, read_io_size:301367296, read_disk_cost_ts:2364604, start_lsn:{lsn:4831543296}}}}}}) [2025-08-14 09:01:23.741670] WDIAG [SHARE.PT] update_replica_ (ob_persistent_ls_table.cpp:548) [342][LSMetaTblUp0][T0][YB42AC4D0007-00063C495B39A91A-0-0] [lt=32928][errcode=-4012] execute update failed(ret=-4012, ret="OB_TIMEOUT", replica={modify_time_us:1755161954778364, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.77.0.7:2882", sql_port:2881, role:2, member_list:[{server:"172.77.0.7:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-08-14 09:01:24.270463] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=24, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:01:24.737037] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=465855] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:24.737052] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162084737032], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:24.737071] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162084737032]}) [2025-08-14 09:01:24.737083] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts statistics(tenant_id=1001, gts_rpc_cnt=0, get_gts_cache_cnt=16, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:01:24.737264] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:01:24.737279] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:01:24.737308] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162084737258], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:24.737328] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162084737258]}) [2025-08-14 09:01:23.766279] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=1003017] finish update min start scn(max_decided_scn={val:1755161719775179002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1755161287778691002, v:0}, keep_alive_scn_:{val:1755161287778691002, v:0}, update_ts_:1755162083719728}) [2025-08-14 09:01:23.761494] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4015] clock out of order(cur_ts=1755162083742138, cur_ts_=1755162083719728, delta=22410) [2025-08-14 09:01:24.737960] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:24.737998] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:24.736513] WDIAG [SHARE.PT] update (ob_persistent_ls_table.cpp:405) [342][LSMetaTblUp0][T0][YB42AC4D0007-00063C495B39A91A-0-0] [lt=993330][errcode=-4012] update follower replica failed(ret=-4012, ret="OB_TIMEOUT", new_replica={modify_time_us:1755161954778364, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.77.0.7:2882", sql_port:2881, role:2, member_list:[{server:"172.77.0.7:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-08-14 09:01:24.738048] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [342][LSMetaTblUp0][T0][YB42AC4D0007-00063C495B39A91A-0-0] [lt=1528][errcode=-4012] update replica failed(ret=-4012, ret="OB_TIMEOUT", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.77.0.7:2882", sql_port:2881, role:2, member_list:[{server:"172.77.0.7:2882", timestamp:1}], replica_type:0, proposal_id:50, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=true) [2025-08-14 09:01:24.738074] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [342][LSMetaTblUp0][T0][YB42AC4D0007-00063C495B39A91A-0-0] [lt=25] update ls replica(ret=-4012, ret="OB_TIMEOUT", time_used=129979085, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.77.0.7:2882", sql_port:2881, role:2, member_list:[{server:"172.77.0.7:2882", timestamp:1}], replica_type:0, proposal_id:50, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=true) [2025-08-14 09:01:24.740057] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:24.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:22.779740] EDIAG [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1958707, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1958695, ObMalloc#start=1, ObMalloc#end=4, ALLOC_END=1, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1bdd73e2 0x1bdd6e36 0x986bc6f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:22.779740] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=5] finish update min start scn(max_decided_scn={val:1755161547738516001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1755161493768447002, v:0}, keep_alive_scn_:{val:1755161493768447002, v:0}, update_ts_:1755162082760282}) [2025-08-14 09:01:24.742589] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:24.734455] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, param_config_:{memory_limit:536870912, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=865568, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:01:24.746057] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:24.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:24.750681] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=49][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7277c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f72eac90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:24.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:24.750714] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919ad990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:24.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:24.750728] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919ad810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:24.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:24.750780] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:24.750818] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:24.753265] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:23.773544] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] [COMPACTION DAG STATUS] (start_time=1755162073770594, end_time=1755162083773541, dag_status={}) [2025-08-14 09:01:24.754713] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:24.754752] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:24.754778] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:24.755054] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:24.220171] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:227) [1031][PxTargetMgr0][T1][Y0-0000000000000000-0-0] [lt=27] refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3691) [2025-08-14 09:01:24.755470] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:24.755494] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:24.755518] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:24.755378] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8] LogLoopThread round_cost_time(us)(round_cost_time=22) [2025-08-14 09:01:24.755616] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=234] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:01:24.755633] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:8607757, max_lsn:1604090331, max_scn:{val:1755161756730847001, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:1556737504}, reuse_lsn:{lsn:1604090331}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:1562147291}}, last_submit_lsn:{lsn:1604090203}, last_submit_end_lsn:{lsn:1604090331}, last_submit_log_id:8607757, last_submit_log_pid:50, max_flushed_lsn:{lsn:1604090203}, max_flushed_end_lsn:{lsn:1604090331}, max_flushed_log_pid:50, committed_end_lsn:{lsn:1604090090}, last_slide_log_id:8607755, last_slide_scn:{val:1755161749741193000, v:0}, last_slide_lsn:{lsn:1604089968}, last_slide_end_lsn:{lsn:1604090090}, last_slide_log_pid:50, last_slide_log_accum_checksum:1446576759, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f990bfe4530}, [StateMgr]={this:0x7f990bfe89b0, self:"172.77.0.7:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"PENDING", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:51}, leader:"0.0.0.0:0", leader_epoch:-1, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:1604090331}, scan_disk_log_finished:true, last_check_start_id:8607756, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161901828039, palf_role_change_cb:0x7f990bfef470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:50, prev:{version:1, config:{config_version:{proposal_id:49, config_seq:45}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:50, config_seq:46}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_mode_pid:50}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_end_lsn:{lsn:1604088821}, prev_mode_pid:50}, reconfig_barrier:{prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_end_lsn:{lsn:1604088821}, prev_mode_pid:50}, persistent_config_version:{proposal_id:50, config_seq:46}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755161738854289, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755161738854289, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f990bfe5c70}, [ModeMgr]={palf_id:1, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:50, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:1543409664}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:51}, log_config_meta:{version:3, proposal_id:50, prev:{version:1, config:{config_version:{proposal_id:49, config_seq:45}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:50, config_seq:46}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_mode_pid:50}, log_snapshot_meta:{version:2, base_lsn:{lsn:1543409664}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:712704}, readable_log_tail:{lsn:712704}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:1610514432}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/meta", dir_fd:225, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66392064, block_header_serialize_buf_:0x7f990bfeabb8, flashback_version:0}, log_storage:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1755161901828095, this:0x7f990bfe90f0}, [Reconfirm]={palf_id:1, self:"172.77.0.7:2882", state:"RECONFIRM_MODE_META", new_proposal_id:51, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"172.77.0.7:2882", majority_max_accept_pid:50, majority_max_lsn:{lsn:1604090331}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:1755161901828373, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f990bfe8b30}) [2025-08-14 09:01:24.755849] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=208] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161974728707, last_flushed_end_lsn:{lsn:1604090331}}]) [2025-08-14 09:01:24.755866] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14] current log_task status(palf_id=1, self="172.77.0.7:2882", log_id=8607756, log_task={header:{begin_lsn:{lsn:1604090090}, end_lsn:{lsn:1604090203}, log_id:8607756, min_scn:{val:1755161749741193001, v:0}, max_scn:{val:1755161749741193001, v:0}, data_len:57, proposal_id:50, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:1604090090}, data_checksum:2031283194, accum_checksum:759186326, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1755161750716165, freeze_ts:1755161750716173, submit_ts:1755161750716183, flushed_ts:1755161853765770, gen_to_freeze cost time:8, gen_to_submit cost time:18, submit_to_flush cost time:103049587}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161974728707, last_flushed_end_lsn:{lsn:1604090331}}]) [2025-08-14 09:01:24.755893] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:01:24.759343] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=35] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:24.759366] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=16] temporary file module use no blocks [2025-08-14 09:01:24.759787] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=13] gc stale ls task succ [2025-08-14 09:01:24.756248] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:24.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:24.757072] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=24] blacklist_loop exec finished(cost_time=25, is_enabled=true, send_cnt=0) [2025-08-14 09:01:24.745604] WDIAG [STORAGE.DUP_TABLE] refresh_dup_ls_ (ob_dup_table_util.cpp:110) [699][T1001_DupTbLeas][T1001][Y0-0000000000000000-0-0] [lt=966578][errcode=-4012] get duplicate ls status info failed(ret=-4012, tmp_dup_ls_status_info={tenant_id:0, ls_id:{id:-1}, ls_group_id:18446744073709551615, status:"UNKNOWN", unit_group_id:18446744073709551615, primary_zone:"", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}) [2025-08-14 09:01:24.759375] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:24.759934] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [804][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4389] timer task too much delay in priority queue(token={this:0x7f991b7a5d18, timer:0x7f99200ff180, task:0x7f99200ff1a8, task_type:"N9oceanbase8observer18ObTableLoadService17ObCheckTenantTaskE", scheduled_time:1755162049756485, pushed_time:1755162051780465, delay:1000000}, delay_in_priority_queue=2023980, threashold=1000000) [2025-08-14 09:01:24.759945] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=567] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:24.759956] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [804][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a5d18, timer:0x7f99200ff180, task:0x7f99200ff1a8, task_type:"N9oceanbase8observer18ObTableLoadService17ObCheckTenantTaskE", scheduled_time:1755162049756485, pushed_time:1755162051780465, delay:1000000}, delay_in_thread_pool=31982740, threashold=500000) [2025-08-14 09:01:24.759967] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=22] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43622986198970, flush_io_finished_ret_:0, flush_io_finished_round_:26266, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:24.760445] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:24.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:24.760793] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:341) [342][LSMetaTblUp0][T0][YB42AC4D0007-00063C495B39A91A-0-0] [lt=19][errcode=-4012] fail to update replica(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, ls_id={id:1}, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.77.0.7:2882", sql_port:2881, role:2, member_list:[{server:"172.77.0.7:2882", timestamp:1}], replica_type:0, proposal_id:50, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-08-14 09:01:24.761106] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7277230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f72ea690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:24.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:24.761130] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7272330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919a5090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:24.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:24.761150] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919822b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891970390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:24.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:24.761162] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891969a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:24.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:24.761182] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d71b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891969890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:24.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:24.761203] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f989196b390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:24.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:24.761216] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a27b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f72ed810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:24.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:24.761246] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a40b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7283690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:24.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:24.761259] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7faf30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7281590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:24.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:24.740129] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6175) [14780][T1001_TimerWK4_SSTableGC][T1001][YB42AC4D0007-00063C4974099C4E-0-0] [lt=8598] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:50408}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1755136957224657000}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1755159231184436000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1755159231184436000) [2025-08-14 09:01:24.742155] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1958707] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136826952869001, v:0}, clog_base_lsn:{lsn:5099962368}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1755071646732636000, v:0}, all_id_meta:{id_meta:[{limited_id:1755161555795565000, latest_log_ts:{val:1755161545796642001, v:0}}, {limited_id:22000001, latest_log_ts:{val:1755135645787806003, v:0}}, {limited_id:60000001, latest_log_ts:{val:1755135646400653001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:50, palf_env_:0x7f9950bf4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-08-14 09:01:24.746167] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=969744] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136829134139001, v:0}, clog_base_lsn:{lsn:8186781696}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1753840867372367034, v:0}, all_id_meta:{id_meta:[{limited_id:1755161729775179000, latest_log_ts:{val:1755161719775179001, v:0}}, {limited_id:18000001, latest_log_ts:{val:1755157440846556001, v:0}}, {limited_id:60000001, latest_log_ts:{val:1755135657266496001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:46, palf_env_:0x7f99321fa030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-08-14 09:01:24.764603] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:24.765805] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=1982] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:24.765840] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [804][T1002_TimerWK0_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=12] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):258, sequence:1}) [2025-08-14 09:01:24.765872] WDIAG [PALF] runTimerTask (block_gc_timer_task.cpp:94) [804][T1002_TimerWK0_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4389] try_recycle_blocks cost too much time(ret=-4389, cost_time_us=5876, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.77.0.7:2882", log_dir:"/root/ob/observer/store/clog/tenant_1002", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):258, sequence:1}, log_alloc_mgr_:{flying_log_task:1, flying_meta_task:2}}) [2025-08-14 09:01:24.765897] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:97) [804][T1002_TimerWK0_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=25] BlockGCTimerTask success(ret=0, cost_time_us=5876, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.77.0.7:2882", log_dir:"/root/ob/observer/store/clog/tenant_1002", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):258, sequence:1}, log_alloc_mgr_:{flying_log_task:1, flying_meta_task:2}}) [2025-08-14 09:01:24.765918] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [804][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a4818, timer:0x7f99335d9ee0, task:0x7f991c5e9398, task_type:"N9oceanbase4palf16BlockGCTimerTaskE", scheduled_time:1755162051809789, pushed_time:1755162051809857, delay:30000}, delay_in_thread_pool=32950130, threashold=500000) [2025-08-14 09:01:24.765954] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [804][T1002_TimerWK0_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=13] get_min_unreplayed_log_info(lsn={lsn:1604088821}, scn={val:1755100948296325001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873479, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:12492179, ret_code_:0}}, next_to_submit_lsn_:{lsn:1604088821}, next_to_submit_scn_:{val:1755100948296325001, v:0}, base_lsn_:{lsn:1409200128}, base_scn_:{val:1753840802540914000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:1604088821}, end_lsn:{lsn:1604088821}, read_buf:{buf_len_:3747840, buf_:0x7f98f6a05000}, block_size:67104768, log_storage_:0x7f990bfebbf0, log_storage_:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:42, read_io_size:147709952, read_disk_cost_ts:1061450, start_lsn:{lsn:1409200128}}}}}}) [2025-08-14 09:01:24.766035] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1085) [804][T1002_TimerWK0_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=78] replay status is not follower(min_unreplayed_lsn={lsn:1604088821}, base_lsn={lsn:1409200128}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873479, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:12492179, ret_code_:0}}, next_to_submit_lsn_:{lsn:1604088821}, next_to_submit_scn_:{val:1755100948296325001, v:0}, base_lsn_:{lsn:1409200128}, base_scn_:{val:1753840802540914000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:1604088821}, end_lsn:{lsn:1604088821}, read_buf:{buf_len_:3747840, buf_:0x7f98f6a05000}, block_size:67104768, log_storage_:0x7f990bfebbf0, log_storage_:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:42, read_io_size:147709952, read_disk_cost_ts:1061450, start_lsn:{lsn:1409200128}}}}}}) [2025-08-14 09:01:24.766085] INFO [CLOG] operator() (ob_log_replay_service.cpp:1678) [804][T1002_TimerWK0_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=49] get_replay_process success(id={id:1}, submitted_log_size=194888693, unsubmitted_log_size=0, replayed_log_size=194888693, unreplayed_log_size=0) [2025-08-14 09:01:24.766100] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [804][T1002_TimerWK0_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=11] get_min_unreplayed_log_info(lsn={lsn:7323685240}, scn={val:1755100946796590002, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873485, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:8946764, ret_code_:0}}, next_to_submit_lsn_:{lsn:7323685240}, next_to_submit_scn_:{val:1755100946796590002, v:0}, base_lsn_:{lsn:7180210176}, base_scn_:{val:1753840803166611000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:7323685240}, end_lsn:{lsn:7323685240}, read_buf:{buf_len_:3747840, buf_:0x7f98f7a05000}, block_size:67104768, log_storage_:0x7f9901131bf0, log_storage_:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:28, read_io_size:98508800, read_disk_cost_ts:645339, start_lsn:{lsn:7180210176}}}}}}) [2025-08-14 09:01:24.766146] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1085) [804][T1002_TimerWK0_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=46] replay status is not follower(min_unreplayed_lsn={lsn:7323685240}, base_lsn={lsn:7180210176}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873485, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:8946764, ret_code_:0}}, next_to_submit_lsn_:{lsn:7323685240}, next_to_submit_scn_:{val:1755100946796590002, v:0}, base_lsn_:{lsn:7180210176}, base_scn_:{val:1753840803166611000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:7323685240}, end_lsn:{lsn:7323685240}, read_buf:{buf_len_:3747840, buf_:0x7f98f7a05000}, block_size:67104768, log_storage_:0x7f9901131bf0, log_storage_:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:28, read_io_size:98508800, read_disk_cost_ts:645339, start_lsn:{lsn:7180210176}}}}}}) [2025-08-14 09:01:24.766192] INFO [CLOG] operator() (ob_log_replay_service.cpp:1678) [804][T1002_TimerWK0_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=46] get_replay_process success(id={id:1001}, submitted_log_size=143475064, unsubmitted_log_size=0, replayed_log_size=143475064, unreplayed_log_size=0) [2025-08-14 09:01:24.766220] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:174) [804][T1002_TimerWK0_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=8] dump tenant replay process(tenant_id=1002, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=322, replayed_log_size(MB)=322, last_submitted_log_size(MB)=322, last_replayed_log_size(MB)=322, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-08-14 09:01:24.766240] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [804][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a49d8, timer:0x7f9922d47110, task:0x7f9922eb44d0, task_type:"N9oceanbase10logservice17ReplayProcessStatE", scheduled_time:1755162052761108, pushed_time:1755162052761182, delay:10000000}, delay_in_thread_pool=32004758, threashold=500000) [2025-08-14 09:01:24.766766] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=13] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:01:24.767014] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=546014][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:01:24.767039] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=15] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:24.767052] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=12] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3692) [2025-08-14 09:01:24.767061] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:01:24.770286] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:24.770322] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=36] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3693) [2025-08-14 09:01:24.770334] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:01:24.771161] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7352330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f989196fc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:24.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:24.771883] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=11] gc stale ls task succ [2025-08-14 09:01:24.773249] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=14] current monitor number(seq_=-1) [2025-08-14 09:01:24.775275] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=22] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:24.775304] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=25] temporary file module use no blocks [2025-08-14 09:01:24.775313] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:24.775323] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=8] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:24.775394] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:24.765, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:24.780589] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=18] [COMPACTION DAG STATUS] (start_time=1755162074773275, end_time=1755162084780588, dag_status={}) [2025-08-14 09:01:24.781207] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919aca90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:24.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:24.781249] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f734fc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891918610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:24.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:24.783264] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:24.789685] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:24.791271] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891969110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:24.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:24.791308] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f989196c510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:24.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:24.801333] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f72eb110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:24.794, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:24.812489] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f989196b690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:24.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:24.812577] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=85][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7282d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:24.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:24.812593] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7289710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:24.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:24.812612] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7397b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:24.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:24.812625] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918299b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891916210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:24.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:24.813141] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:24.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:24.819312] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:24.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:24.819646] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:24.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:24.829713] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd070, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fd350, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:5.00s, expected_run_time:09:01:24.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:24.829751] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:01:24.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:24.832645] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f72ec910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:24.825, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:24.832680] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f72ec790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:24.826, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:24.832696] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7282f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:24.820, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:24.832733] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72879b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f728a010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:24.828, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:24.834504] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:24.852780] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7289590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:24.846, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:24.870878] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=38] PNIO [ratelimit] time: 1755162084870875, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007602, add_bytes: 0 [2025-08-14 09:01:24.919377] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:24.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:24.919708] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:24.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:24.934567] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:24.993577] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:24.989, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:24.994453] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=3610][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:24.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.013652] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:25.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.019439] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:24.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.019768] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:24.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.027005] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:25.151, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.034631] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:25.094135] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:25.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.119504] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:25.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.119815] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:25.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.120334] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=24] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:01:25.134654] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:25.214972] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:25.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.219566] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:25.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.219871] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:25.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.234698] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:25.237552] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=13] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:25.237579] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162085237547], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:25.237604] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162085237547], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:25.245985] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:25.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.256037] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:25.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.270405] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:25.270431] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=26] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3694) [2025-08-14 09:01:25.270452] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:01:25.270464] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=6] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:25.270476] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=13] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3695) [2025-08-14 09:01:25.270484] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:01:25.270660] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:25.270672] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=12] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3696) [2025-08-14 09:01:25.270687] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:01:25.278524] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:25.265, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.308710] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:25.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.319934] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:25.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.419973] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:25.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.497410] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:25.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.520022] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:25.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.530022] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:25.515, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.620083] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:25.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.670894] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:01:25.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.688630] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EE93-0-0] [lt=12] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:01:25.718763] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=29] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:25.719462] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=28] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:01:25.719493] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:25.719931] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:25.720147] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:25.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:24.770559] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2826) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8571] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:682, [delay/us]: prepare:0, schedule:0, submit:600000000, device:8907, total:8917) [2025-08-14 09:01:25.723504] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=951477] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:01:24.775335] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43622986211134, flush_io_finished_ret_:0, flush_io_finished_round_:26234, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:25.727001] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:25.729368] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72760b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919b5b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.719, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.736065] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] get_fetch_log_dst failed or invalid(ret=0, palf_id=1, self="172.77.0.7:2882", fetch_log_dst="172.77.0.7:2882") [2025-08-14 09:01:25.737368] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:25.234484] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:01:25.737642] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=502392][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:01:25.737665] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:01:25.737680] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:00:41.841, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=08:59:41.841], idx=471, next_allow_print_ts="09:01:11.841") [2025-08-14 09:01:25.737699] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:01:25.737717] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:01:25.737731] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:01:25.737746] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:01:25.737760] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:00:52.998, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:22.998], idx=771, next_allow_print_ts="09:01:22.998") [2025-08-14 09:01:25.737779] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=08:59:57.942, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:27.942], idx=868, next_allow_print_ts="09:00:27.942") [2025-08-14 09:01:25.737954] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:01:25.737969] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:01:25.737990] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:01:25.245159] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=52][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:25.239, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.738126] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=492219][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:25.245|207=492.96ms|210=0us|222=2us|total=492.98ms) [2025-08-14 09:01:25.738142] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] timer task use too much time(end=1755162085738141, start=1755162085245158, delta=492983) [2025-08-14 09:01:25.738155] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=482666, expected_run_ts_us_=1755162085255487, ObClockGenerator::getRealClock()=1755162085738154, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:25.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.738170] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:25.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.738189] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=148872, expected_run_ts_us_=1755162085589316, ObClockGenerator::getRealClock()=1755162085738189, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:25.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.738206] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:25.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.738221] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:25.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.365070] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:25.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.738301] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=372665][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:25.365|207=373.23ms|210=0us|222=1us|total=373.24ms) [2025-08-14 09:01:25.738313] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task use too much time(end=1755162085738312, start=1755162085365068, delta=373244) [2025-08-14 09:01:25.738338] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:25.738358] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=15] temporary file module use no blocks [2025-08-14 09:01:25.738368] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:25.738389] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=19] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:25.738403] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43622986207118, flush_io_finished_ret_:0, flush_io_finished_round_:26260, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:25.237634] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=26] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162085237547]}) [2025-08-14 09:01:25.738499] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=500105][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:01:25.738521] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:01:25.738542] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=10] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:25.738553] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162085738537], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:25.738567] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162085738537], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:25.738589] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162085738537]}) [2025-08-14 09:01:25.738747] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:01:25.738760] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=13] get number(ret=-4038, service_type_=0, range=1, base_id=1755162085737458000, start_id=0, end_id=140295968341544) [2025-08-14 09:01:25.738778] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:01:25.738794] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162085738743], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:25.738820] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=24] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162085738743]}) [2025-08-14 09:01:25.738830] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts statistics(tenant_id=1002, gts_rpc_cnt=0, get_gts_cache_cnt=0, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:01:25.738946] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:25.740647] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=20335] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:25.740674] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] try to retry start_working [2025-08-14 09:01:25.741633] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:25.319628] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:25.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.742480] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=422209][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:25.319|207=422.85ms|210=0us|222=2us|total=422.87ms) [2025-08-14 09:01:25.742499] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4389] timer task use too much time(end=1755162085742499, start=1755162085319626, delta=422873) [2025-08-14 09:01:25.334757] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:25.742554] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=407176][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:25.744719] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:25.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.723523] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:01:25.745074] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21512] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:01:25.745912] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7283810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.745939] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f728b690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.745953] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739be30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f728b810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.745966] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7281e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.746256] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=16] [COMPACTION DAG STATUS] (start_time=1755162074780406, end_time=1755162085746255, dag_status={}) [2025-08-14 09:01:25.724624] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:213) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-08-14 09:01:25.746848] INFO [COMMON] handle (memory_dump.cpp:544) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=22187] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-08-14 09:01:25.747763] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=14] update_check_range(min_check_version=2540, max_check_version=2540, global_version=2634) [2025-08-14 09:01:25.749753] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:25.749785] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:25.749816] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:25.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.750181] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:25.750222] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:25.750273] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:24.780557] INFO [ARCHIVE] do_thread_task_ (ob_archive_timer.cpp:125) [862][T1002_ArcTimer][T1002][YB42AC4D0007-00063C495E599C16-0-0] [lt=2829015] archive round not in doing status, just skip(key={incarnation:-1, dest_id:-1, round:-1}, state={status:"INVALID"}) [2025-08-14 09:01:25.754340] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:25.754800] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:25.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:24.924032] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=762434][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a5df8, timer:0x7f994b3feee0, task:0x7f994b3fef50, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162023758648, pushed_time:0, delay:1000000}, running_task_token={this:0x7f994e1a5c38, timer:0x7f994b3feee0, task:0x7f994b3fef20, task_type:"N9oceanbase8observer18ObTableLoadService15ObHeartBeatTaskE", scheduled_time:1755162023721549, pushed_time:1755162047763044, delay:10000000}, total_delay=61165382, threashold=1000000, tenant_id=1) [2025-08-14 09:01:25.755081] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:25.755972] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7edc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7f0910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.756012] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ebe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7f0a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.756025] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891971110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.756213] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:01:25.756280] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=43] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:01:25.757942] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:25.757974] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=11] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:25.758017] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:23.763697] EDIAG [ELECT] operator() (election_proposer.cpp:232) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1994091, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1994080, ObMalloc#start=1, ObMalloc#end=4, ALLOC_END=0, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xfa07f77 0xfa07a14 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:25.760849] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=1994091][errcode=-4389] run this task use too much time(*p_task_={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:0, time_interval:500.00ms, expected_run_time:09:01:25.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.760874] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:01:25.760|total=2.00s) [2025-08-14 09:01:25.760894] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] task executed too delay(delay_time=1976464, *p_task_={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:0, time_interval:250.00ms, expected_run_time:09:01:25.765, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.762527] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=18] gc stale ls task succ [2025-08-14 09:01:25.762620] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:25.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.764325] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:25.766039] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891968c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.750, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.766105] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=62][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7283990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.766119] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728d1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7291110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.766407] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=17] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:25.768304] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=12] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:01:24.734352] EDIAG [SHARE] refresh (ob_task_define.cpp:396) [280][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=991339][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1033482, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1033473, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1fd6cd17 0x96c37e0 0x1fda2168 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:24.734352] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [280][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=991339][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:114, tid:768}]) [2025-08-14 09:01:25.770757] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:25.770786] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=27] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3697) [2025-08-14 09:01:25.770798] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:01:25.770821] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=18] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:25.770830] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3698) [2025-08-14 09:01:25.770837] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:01:25.771066] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:25.771079] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=13] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3699) [2025-08-14 09:01:25.771086] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:01:25.772273] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:01:25.774429] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=25] current monitor number(seq_=-1) [2025-08-14 09:01:25.776104] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919b4790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.776135] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919afe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919b9990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.776151] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d62b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891968f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.776164] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891970210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.784300] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:25.739215] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=11482] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:01:25.787923] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891968b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.787958] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f989196b210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.787971] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7280990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.787995] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7397a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.789687] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:25.797192] INFO handle (memory_dump.cpp:671) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=15] statistics: tenant_cnt: 6, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 168 270 0 1 2 110 1 0 1 3 7 0 0 1 4 1 1 0 1 5 86 6 0 1 6 13 0 0 1 7 22 2 0 1 8 53 1 0 1 12 17 3 0 1 13 10 1 0 1 14 1 1 0 1 15 1 1 0 1 16 4 6 0 1 18 4 3 0 1 20 2 1 0 1 22 1 1 0 1 24 1 0 0 1 25 6 0 0 500 0 561 274 0 500 7 188 6 0 500 8 41 1 0 500 9 2 1 0 500 10 2 1 0 500 16 49 4 0 500 17 52 7 0 500 22 38 64 0 500 23 27 10 0 508 0 2 7 0 508 8 5 1 0 509 0 2 7 0 509 8 5 1 0 1001 0 147 249 0 1001 2 26 1 0 1001 3 5 0 0 1001 4 1 1 0 1001 5 85 6 0 1001 6 11 0 0 1001 7 30 2 0 1001 8 36 1 0 1001 12 17 3 0 1001 13 3 1 0 1001 14 1 1 0 1001 16 6 2 0 1001 18 4 3 0 1001 20 4 1 0 1001 25 8 0 0 1002 0 262 278 0 1002 2 198 1 0 1002 3 4 0 0 1002 4 1 1 0 1002 5 101 6 0 1002 6 14 0 0 1002 7 6 2 0 1002 8 73 1 0 1002 12 18 3 0 1002 13 7 1 0 1002 14 1 2 0 1002 16 7 8 0 1002 18 9 3 0 1002 20 2 1 0 1002 21 1 0 0 1002 24 1 0 0 1002 25 7 0 0 cost_time: 49459 [2025-08-14 09:01:25.797992] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7280690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.798030] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7efe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.798042] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7f0010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.798439] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=20] [MEMORY] tenant: 1, limit: 2147483648 hold: 1006882816 cache_hold: 56623104 cache_used: 56623104 cache_item_count: 27 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 444993536 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 230686720 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 1163264 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 483328 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 53714944 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 1458176 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 14204928 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 111149056 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35651584 limit= 429496720 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 8003584 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= PS_CACHE_CTX_ID hold_bytes= 65536 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 6438912 limit= 9223372036854775807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 3481600 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3964928 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 212992 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 884736 limit= 9223372036854775807 [2025-08-14 09:01:25.807906] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EE83-0-0] [lt=18] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-08-14 09:01:25.808052] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7282c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.818117] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f728b210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.802, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.820213] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:25.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.828154] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f728a190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.814, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.828190] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7282010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.817, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.828201] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7ee810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.828216] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7ee990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.819, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.828250] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f73a0a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.813, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.835475] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:01:25.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.838208] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919831b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891970c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.828, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.842563] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:25.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.842628] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:25.848257] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891973c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.835, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.855057] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=927676][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:01:25.858329] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891968210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.847, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.858360] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891970b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.844, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.858377] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7282190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:25.844, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:25.859479] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c0b90, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571c0e70, time_wheel:0x7f9959fef990, thread_pool:0x7f9959fef170, is_running:1, time_interval:3.00s, expected_run_time:09:01:25.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.878496] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=32] PNIO [ratelimit] time: 1755162085878493, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007618, add_bytes: 0 [2025-08-14 09:01:25.920277] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=49][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:25.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.938545] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:25.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.942626] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:25.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.998376] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=59][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:25.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:25.999670] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:25.989, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.019798] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:26.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.020311] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:25.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.038614] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:25.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.042687] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:25.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.090158] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:26.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.120354] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:26.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.120917] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=17] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:01:26.138681] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:26.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.142749] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:26.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.210803] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:26.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.220405] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:26.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.238174] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:01:26.238745] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:26.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.239123] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=13] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:26.239153] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162086239118], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:26.239178] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162086239118], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:26.239211] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=26] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162086239118]}) [2025-08-14 09:01:26.239249] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=18] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:26.239262] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162086239247], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:26.240959] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:26.239, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.249888] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:26.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.259968] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:26.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.261036] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:26.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.271182] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:26.271211] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=28] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3700) [2025-08-14 09:01:26.271219] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:01:26.271229] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=6] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:26.271263] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:227) [1031][PxTargetMgr0][T1001][Y0-0000000000000000-0-0] [lt=31] refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3701) [2025-08-14 09:01:26.271274] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:01:26.271464] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:26.320456] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:26.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.338810] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:26.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.420466] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:26.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.438875] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:26.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.475278] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=64][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fa190, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa470, time_wheel:0x7f99341cdbf0, thread_pool:0x7f99341cd3d0, is_running:1, time_interval:2.00s, expected_run_time:09:01:26.334, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.491408] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=52][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:26.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.492334] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:26.489, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.512475] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:26.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.520518] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:26.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.538939] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:26.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.585778] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fe410, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe6f0, time_wheel:0x7f9922df9d40, thread_pool:0x7f9922df9520, is_running:1, time_interval:2.00s, expected_run_time:09:01:26.432, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.592964] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:26.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.620564] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:26.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.622548] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ebbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7ee510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:26.615, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:26.639008] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:26.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.675430] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:01:26.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.713517] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=52][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:26.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.719441] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=27] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:26.719497] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:26.719960] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:26.720633] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:26.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.724691] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=947843] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:26.724727] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=29] temporary file module use no blocks [2025-08-14 09:01:26.724737] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:26.724751] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:26.724774] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=22] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43622988164381, flush_io_finished_ret_:0, flush_io_finished_round_:26235, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:26.728019] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=25] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:01:26.728359] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=31] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:25.807005] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [851][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=15] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=16, total_destructed_detector=16, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f9922fee2b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-08-14 09:01:26.239273] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162086239247]}) [2025-08-14 09:01:26.732902] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=492591][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:01:26.732920] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:01:26.732938] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162086732894], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:26.732954] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162086732894], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:26.732970] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162086732894]}) [2025-08-14 09:01:25.786318] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [723][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=27] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=16, total_destructed_detector=16, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f99333562b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-08-14 09:01:26.238209] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:01:26.733499] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=494532][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:01:26.733535] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:01:26.733548] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:00:41.841, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=08:59:41.841], idx=471, next_allow_print_ts="09:01:11.841") [2025-08-14 09:01:26.733565] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:01:26.733583] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:01:26.733576] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [14881][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=1995579][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f9970fc5ed8, timer:0x7f9970f43820, task:0x564d727182c8, task_type:"N9oceanbase5share20ObActiveSessHistTaskE", scheduled_time:1755162047740015, pushed_time:1755162047740087, delay:1000000}, delay_in_thread_pool=31003408, threashold=500000) [2025-08-14 09:01:26.733596] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:01:26.733612] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:01:26.733625] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:00:52.998, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:22.998], idx=771, next_allow_print_ts="09:01:22.998") [2025-08-14 09:01:26.733644] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=08:59:57.942, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:27.942], idx=868, next_allow_print_ts="09:00:27.942") [2025-08-14 09:01:26.733810] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:01:26.733825] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:01:26.733838] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:01:26.242813] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:26.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.733944] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=490385][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:26.242|207=491.13ms|210=0us|222=2us|total=491.14ms) [2025-08-14 09:01:26.733971] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4389] timer task use too much time(end=1755162086733970, start=1755162086242812, delta=491158) [2025-08-14 09:01:26.271484] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=20] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3702) [2025-08-14 09:01:26.734017] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=461833][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:01:25.746013] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:147) [804][T1002_TimerWK0_][T1002][Y0-0000000000000000-0-0] [lt=12] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-08-14 09:01:26.734841] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [804][T1002_TimerWK0_][T1002][Y0-0000000000000000-0-0] [lt=987328] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-08-14 09:01:26.734858] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [804][T1002_TimerWK0_][T1002][Y0-0000000000000000-0-0] [lt=16] [tabletchange] task check ls(ls->get_ls_id()={id:1001}, tablet_persist_trigger=0) [2025-08-14 09:01:25.739937] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [719][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571f9d70, caller:ob_table_lock_service.cpp:start:276, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE6$_1273, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa090, time_wheel:0x7f99375f7d10, thread_pool:0x7f99375f74f0, is_running:0, time_interval:10.00s, expected_run_time:09:01:32.332, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.735499] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [719][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=994056][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:01:26.735|total=4.31s) [2025-08-14 09:01:26.735540] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919abd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:26.721, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:26.735562] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919abb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:26.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:26.735575] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919b4f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:26.726, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:26.735589] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fdc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7282310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:26.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:26.735602] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7282490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:26.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:26.742218] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:26.743658] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:26.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.743805] WDIAG [DETECT] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [723][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=945645][errcode=0] cost too much time:ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t), (this=|threshold=100.00ms|start at 09:01:26.743|total=963.49ms) [2025-08-14 09:01:26.744068] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [14780][T1001_TimerWK4_SSTableGC][T1001][YB42AC4D0007-00063C4974099C4E-0-0] [lt=21505][errcode=0] cost too much time:mds_table_base.cpp:int oceanbase::storage::mds::MdsTableBase::advance_state_to(oceanbase::storage::mds::MdsTableBase::State) const, (this=|threshold=1.00ms|start at 09:01:25.763|total=980.81ms) [2025-08-14 09:01:26.744858] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:26.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.746578] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:26.746658] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f728be10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:26.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:26.746742] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:26.746779] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=24] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:26.748487] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:26.752329] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:26.752452] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:26.752483] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:26.752509] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=24] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:26.752531] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=15] temporary file module use no blocks [2025-08-14 09:01:26.752540] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:26.752553] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:26.752568] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43622988178196, flush_io_finished_ret_:0, flush_io_finished_round_:26261, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:26.752895] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:26.752918] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:26.753929] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:26.733595] WDIAG alarm_if_necessary (ob_timer_service.cpp:143) [14881][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4389] timer task's elapsed_time is greater than period(token={this:0x7f9970fc5ed8, timer:0x7f9970f43820, task:0x564d727182c8, task_type:"N9oceanbase5share20ObActiveSessHistTaskE", scheduled_time:1755162047740015, pushed_time:1755162047740087, delay:1000000}, elapsed_time=7990080, period=1000000) [2025-08-14 09:01:26.755066] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:26.755416] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=60][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:26.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.755816] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=27] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:26.755887] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:26.756407] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=16667] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:26.756459] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=37] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:25.786473] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=97] get wrs ts(ls_id={id:1}, delta=537998942, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:26.757179] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72847b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f728a310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:26.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:26.758452] WDIAG [DETECT] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [851][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=924198][errcode=0] cost too much time:ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t), (this=|threshold=100.00ms|start at 09:01:26.758|total=951.47ms) [2025-08-14 09:01:26.760786] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=32] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:26.760817] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26] temporary file module use no blocks [2025-08-14 09:01:26.760823] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:26.760837] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:26.760848] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43622988200485, flush_io_finished_ret_:0, flush_io_finished_round_:26268, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:26.767358] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919b9e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:26.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:26.767403] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891806f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:26.751, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:26.767419] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f739fb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:26.750, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:26.767607] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:26.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.769043] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a5d18, timer:0x7f99200ff180, task:0x7f99200ff1a8, task_type:"N9oceanbase8observer18ObTableLoadService17ObCheckTenantTaskE", scheduled_time:1755162085759970, pushed_time:0, delay:1000000}, running_task_token={this:0x7f991b7a5fb8, timer:0x7f99200ff180, task:0x7f99200ff1f0, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162050755138, pushed_time:1755162084760013, delay:1000000}, total_delay=1009072, threashold=1000000, tenant_id=1002) [2025-08-14 09:01:26.769348] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=23] gc stale ls task succ [2025-08-14 09:01:26.769393] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:26.769489] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=25] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:26.769531] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=22] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:01:26.773243] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:01:26.777423] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73539b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7396b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:26.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:26.784529] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:26.789538] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:26.807569] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=61][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7f1510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:26.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:26.817629] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d22b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f72eaf90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:26.808, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:26.820686] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:26.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.827696] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f72ec010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:26.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:26.827730] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891972790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:26.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:26.834016] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:26.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.847816] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919ab290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:26.831, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:26.847849] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f73a1c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:26.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:26.853039] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:01:26.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.856263] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:233) [827][T1002_RFLWorker][T1002][YB42AC4D0007-00063C495D099C16-0-0] [lt=8938] ObRemoteFetchWorker is running(thread_index=0) [2025-08-14 09:01:26.856487] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=17] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:26.857205] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:01:26.857231] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:01:26.857878] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f989196ac10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:26.849, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:26.867938] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f73a0910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:26.852, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:26.877982] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891972610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:26.861, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:26.886139] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=30] PNIO [ratelimit] time: 1755162086886136, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007643, add_bytes: 0 [2025-08-14 09:01:26.908134] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919859b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891972490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:26.896, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:26.920750] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:26.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.932693] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=837407][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb4798, timer:0x7f9966ee3f40, task:0x7f99333e53a0, task_type:"N9oceanbase5table20ObTableClientInfoMgr27ObTableClientInfoRetireTaskE", scheduled_time:1755161724792698, pushed_time:0, delay:60000000}, running_task_token={this:0x7f9931bb4178, timer:0x7f9966ee3f40, task:0x7f99333d2fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161723481487, pushed_time:1755161737766048, delay:900000000}, total_delay=362139994, threashold=1000000, tenant_id=1001) [2025-08-14 09:01:26.934080] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:26.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.956557] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=16] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:26.997243] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:26.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.999189] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:26.989, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.019269] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:27.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.020817] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:26.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.034141] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:26.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.056613] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=22] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:27.076928] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=16][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:01:27.099712] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:27.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.108309] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [882][T1002_ArcSender][T1002][YB42AC4D0007-00063C495EC99C16-0-0] [lt=29] ObArchiveSender is running(thread_index=6) [2025-08-14 09:01:27.120878] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:27.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.121557] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=20] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:01:27.134209] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:27.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.156668] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=23] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:27.210366] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:27.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.220933] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:27.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.232943] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb2b98, timer:0x7f9966ee3f40, task:0x7f99322ce030, task_type:"N9oceanbase8observer20ObTableQueryASyncMgrE", scheduled_time:1755161725751508, pushed_time:0, delay:0}, running_task_token={this:0x7f9931bb4178, timer:0x7f9966ee3f40, task:0x7f99333d2fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161723481487, pushed_time:1755161737766048, delay:900000000}, total_delay=361481433, threashold=1000000, tenant_id=1001) [2025-08-14 09:01:27.233253] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=11] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:27.233287] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=30][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162087233247], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:27.233313] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162087233247], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:27.233335] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162087233247]}) [2025-08-14 09:01:27.233353] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162087233247]}) [2025-08-14 09:01:27.233370] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:01:27.233376] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:01:27.233383] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:27.233391] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162087233382], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:27.233400] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162087233382]}) [2025-08-14 09:01:27.233408] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162087233382]}) [2025-08-14 09:01:27.233600] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:01:27.233617] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:01:27.233631] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162087233596], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:27.233644] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162087233596]}) [2025-08-14 09:01:27.233649] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162087233596]}) [2025-08-14 09:01:27.234042] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:01:27.234076] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:01:27.234100] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:27.234110] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:01:27.234123] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:00:41.841, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=08:59:41.841], idx=471, next_allow_print_ts="09:01:11.841") [2025-08-14 09:01:27.234130] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=22] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3703) [2025-08-14 09:01:27.234139] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:01:27.234142] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:01:27.234160] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=13] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:27.234165] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:01:27.234170] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3704) [2025-08-14 09:01:27.234179] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:01:27.234184] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:01:27.234203] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:01:27.234222] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:00:52.998, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:22.998], idx=771, next_allow_print_ts="09:01:22.998") [2025-08-14 09:01:27.234259] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=08:59:57.942, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:27.942], idx=868, next_allow_print_ts="09:00:27.942") [2025-08-14 09:01:27.234378] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:27.234395] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=16] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3705) [2025-08-14 09:01:27.234400] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:01:27.234443] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:01:27.234464] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:01:27.234482] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:01:27.234582] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:27.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.240542] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:27.239, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.248692] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:27.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.256690] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=22] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:27.258760] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:27.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.260640] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:27.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.270698] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:27.265, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.300882] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:27.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.321001] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:27.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.334643] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=59][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:27.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.356747] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=24] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:27.421047] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:27.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.434705] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:27.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.456802] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=26] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:27.490018] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=50][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:27.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.492041] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:27.489, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.512170] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:27.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.521110] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:27.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.522160] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:27.515, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.534737] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:27.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.556863] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=25] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:27.592606] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:27.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.621154] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:27.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.634798] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:27.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.656925] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=25] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:27.675951] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571634a0, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957163780, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:5.00s, expected_run_time:09:01:27.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.675989] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:01:27.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.713221] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:27.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:24.759907] EDIAG [STORAGE.DUP_TABLE] execute_for_dup_ls_ (ob_dup_table_util.cpp:202) [699][T1001_DupTbLeas][T1001][Y0-0000000000000000-0-0] [lt=14277][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2956804, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2956799, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1d53f102 0x1d53d6c8 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:25.942687] EDIAG [RS] get_valid_sts_after (ob_tenant_info_loader.cpp:454) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1776778, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1776771, ObMalloc#start=0, ObMalloc#end=3, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1001d942 0x996d843 0x996d15a 0x996cfd8 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:27.720442] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:27.720483] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:27.720577] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=79] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:27.720665] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=911263][errcode=-4016] task executed too delay(delay_time=691399, *p_task_={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:0, time_interval:250.00ms, expected_run_time:09:01:27.765, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:24.759907] WDIAG [STORAGE.DUP_TABLE] execute_for_dup_ls_ (ob_dup_table_util.cpp:202) [699][T1001_DupTbLeas][T1001][Y0-0000000000000000-0-0] [lt=14277][errcode=0] refresh dup ls failed(tmp_ret=-4012, this={tenant_id_:1001, dup_table_scan_timer_:0x7f9932a3dae8, dup_loop_worker_:0x7f9932d066f0, min_dup_ls_status_info_:{tenant_id:0, ls_id:{id:-1}, ls_group_id:18446744073709551615, status:"UNKNOWN", unit_group_id:18446744073709551615, primary_zone:"", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}, tenant_schema_dup_tablet_set_.size():0, scan_task_execute_interval_:10000000, last_dup_ls_refresh_time_:0, last_dup_schema_refresh_time_:0, last_scan_task_succ_time_:1755161988765267, max_execute_interval_:51022952}) [2025-08-14 09:01:27.721870] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:27.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.722183] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1776778] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:01:27.722204] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:27.722216] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:27.722457] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=22] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 444993536 used= 389486368 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 70 washed_blocks= 287 washed_size= 70221824 [MEMORY] request_cached_chunk_cnt= 4 [MEMORY] hold= 389486368 used= 379240963 count= 32868 avg_used= 11538 mod=SUMMARY [2025-08-14 09:01:27.722504] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=26] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 230686720 used= 228884480 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 block_cnt= 110 chunk_cnt= 110 mod=Memstore [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:01:27.722521] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1 ctx_id= EXECUTE_CTX_ID hold= 1163264 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 15 washed_size= 13516800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:27.722537] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:01:27.722559] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 53714944 used= 47487424 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 84 washed_blocks= 334 washed_size= 126640128 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 47487424 used= 43098388 count= 3994 avg_used= 10790 mod=SUMMARY [2025-08-14 09:01:27.722574] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= WORK_AREA hold= 1458176 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 13 washed_blocks= 66 washed_size= 25804800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:27.722589] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 14204928 used= 10970224 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 277 washed_size= 31932416 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10970224 used= 6078924 count= 18573 avg_used= 327 mod=SUMMARY [2025-08-14 09:01:27.722605] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 111149056 used= 108896256 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 block_cnt= 211 chunk_cnt= 53 mod=CoStack [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 mod=SUMMARY [2025-08-14 09:01:27.722625] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 56623104 used= 56623104 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 56623104 used= 0 count= 27 avg_used= 0 mod=SUMMARY [2025-08-14 09:01:27.722640] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 34365376 limit= 429496720 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 34365376 used= 31086608 count= 475 avg_used= 65445 mod=SUMMARY [2025-08-14 09:01:27.722655] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 8003584 used= 4947392 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 10 washed_blocks= 23 washed_size= 12967936 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4947392 used= 4793344 count= 604 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:27.722671] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:27.722686] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 65536 used= 31024 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 2031616 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 31024 used= 27058 count= 20 avg_used= 1352 mod=SUMMARY [2025-08-14 09:01:27.722701] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= RPC_CTX_ID hold= 6438912 used= 1548288 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1548288 used= 1015424 count= 67 avg_used= 15155 mod=SUMMARY [2025-08-14 09:01:27.722719] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 3481600 used= 2391232 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 10 washed_size= 4907008 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2391232 used= 2367040 count= 36 avg_used= 65751 mod=SUMMARY [2025-08-14 09:01:27.722736] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 3964928 used= 3080192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 3 washed_size= 229376 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3080192 used= 2983936 count= 376 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:27.722752] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= SCHEMA_SERVICE hold= 212992 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1884160 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:27.722768] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:27.722782] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= MERGE_NORMAL_CTX_ID hold= 884736 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 6 washed_blocks= 9 washed_size= 11698176 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:27.722805] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant: 500, limit: 9223372036854775807 hold: 684924928 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 326643712 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 44957696 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 85983232 limit= 9223372036854775807 [MEMORY] ctx_id= LIBEASY hold_bytes= 3145728 limit= 9223372036854775807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 6291456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 2760704 limit= 9223372036854775807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 37232640 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 78925824 limit= 9223372036854775807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 98983936 limit= 9223372036854775807 [2025-08-14 09:01:27.722840] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 326643712 used= 300846672 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 502 washed_blocks= 2234 washed_size= 929439744 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 300846672 used= 293477495 count= 12622 avg_used= 23251 mod=SUMMARY [2025-08-14 09:01:27.722866] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= GLIBC hold= 44957696 used= 21968496 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 176 washed_blocks= 807 washed_size= 347209728 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 21968496 used= 10367285 count= 131706 avg_used= 78 mod=SUMMARY [2025-08-14 09:01:27.722882] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= CO_STACK hold= 85983232 used= 83091456 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 83091456 used= 82936896 count= 161 avg_used= 515136 block_cnt= 161 chunk_cnt= 41 mod=CoStack [MEMORY] hold= 83091456 used= 82936896 count= 161 avg_used= 515136 mod=SUMMARY [2025-08-14 09:01:27.722896] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= LIBEASY hold= 3145728 used= 3070816 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1048576 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3070816 used= 2859784 count= 223 avg_used= 12824 mod=SUMMARY [2025-08-14 09:01:27.722909] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 500 ctx_id= LOGGER_CTX_ID hold= 6291456 used= 4161536 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4161536 used= 4159488 count= 2 avg_used= 2079744 mod=SUMMARY [2025-08-14 09:01:27.722945] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=19] [MEMORY] tenant_id= 500 ctx_id= RPC_CTX_ID hold= 2760704 used= 1310720 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 49 washed_blocks= 61 washed_size= 99999744 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1310720 used= 865536 count= 54 avg_used= 16028 mod=SUMMARY [2025-08-14 09:01:27.722960] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 500 ctx_id= PKT_NIO hold= 37232640 used= 35792384 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 42 washed_blocks= 63 washed_size= 82591744 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 35792384 used= 34871392 count= 116 avg_used= 300615 mod=SUMMARY [2025-08-14 09:01:27.722991] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 500 ctx_id= SCHEMA_SERVICE hold= 78925824 used= 68904576 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 22 washed_size= 5767168 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 68904576 used= 67642943 count= 4856 avg_used= 13929 mod=SUMMARY [2025-08-14 09:01:27.723015] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=17] [MEMORY] tenant_id= 500 ctx_id= UNEXPECTED_IN_500 hold= 98983936 used= 97125328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 97125328 used= 86744896 count= 3004 avg_used= 28876 mod=SUMMARY [2025-08-14 09:01:27.723030] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 508, limit: 1073741824 hold: 15613952 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 5128192 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10485760 limit= 9223372036854775807 [2025-08-14 09:01:27.723041] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 508 ctx_id= DEFAULT_CTX_ID hold= 5128192 used= 3765328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3765328 used= 3723912 count= 84 avg_used= 44332 mod=SUMMARY [2025-08-14 09:01:27.723066] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 508 ctx_id= CO_STACK hold= 10485760 used= 10321920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10321920 used= 10302720 count= 20 avg_used= 515136 mod=SUMMARY [2025-08-14 09:01:27.723105] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant: 509, limit: 1073741824 hold: 15613952 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 5128192 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10485760 limit= 9223372036854775807 [2025-08-14 09:01:27.723129] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 509 ctx_id= DEFAULT_CTX_ID hold= 5128192 used= 3765328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3765328 used= 3723912 count= 84 avg_used= 44332 mod=SUMMARY [2025-08-14 09:01:27.723152] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 509 ctx_id= CO_STACK hold= 10485760 used= 10321920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10321920 used= 10302720 count= 20 avg_used= 515136 mod=SUMMARY [2025-08-14 09:01:27.723211] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant: 1001, limit: 1073741824 hold: 606650368 cache_hold: 44040192 cache_used: 44040192 cache_item_count: 21 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 302309376 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 54525952 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 3080192 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 483328 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 45514752 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 5521408 limit= 157286400 [MEMORY] ctx_id= GLIBC hold_bytes= 15499264 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 73424896 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35651584 limit= 214748360 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 3252224 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 12582912 limit= 1073741800 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 5570560 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3866624 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 1179648 limit= 9223372036854775807 [2025-08-14 09:01:27.723253] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1001 ctx_id= DEFAULT_CTX_ID hold= 302309376 used= 269117392 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 81 washed_blocks= 406 washed_size= 113410048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 269117392 used= 262997190 count= 3795 avg_used= 69300 mod=SUMMARY [2025-08-14 09:01:27.723267] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= MEMSTORE_CTX_ID hold= 54525952 used= 54099968 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 54099968 used= 54075216 count= 26 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:01:27.723277] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= EXECUTE_CTX_ID hold= 3080192 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 8 washed_size= 7405568 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:27.723287] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:01:27.723298] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= PLAN_CACHE_CTX_ID hold= 45514752 used= 36449984 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 77 washed_blocks= 201 washed_size= 132743168 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 36449984 used= 32916258 count= 3312 avg_used= 9938 mod=SUMMARY [2025-08-14 09:01:27.723308] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= WORK_AREA hold= 5521408 used= 0 limit= 157286400 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 9 washed_blocks= 40 washed_size= 17547264 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:27.723322] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1001 ctx_id= GLIBC hold= 15499264 used= 10889728 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 25 washed_blocks= 164 washed_size= 47415296 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10889728 used= 6020141 count= 18313 avg_used= 328 mod=SUMMARY [2025-08-14 09:01:27.723334] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= CO_STACK hold= 73424896 used= 71221248 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 32 washed_blocks= 34 washed_size= 2072576 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 71221248 used= 71088768 count= 138 avg_used= 515136 block_cnt= 138 chunk_cnt= 36 mod=CoStack [MEMORY] hold= 71221248 used= 71088768 count= 138 avg_used= 515136 mod=SUMMARY [2025-08-14 09:01:27.723348] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= KVSTORE_CACHE_ID hold= 44040192 used= 44040192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 44040192 used= 0 count= 21 avg_used= 0 mod=SUMMARY [2025-08-14 09:01:27.723368] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 33906624 limit= 214748360 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33906624 used= 30629648 count= 468 avg_used= 65447 mod=SUMMARY [2025-08-14 09:01:27.723379] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= TX_CALLBACK_CTX_ID hold= 3252224 used= 2342592 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 3 washed_blocks= 8 washed_size= 3039232 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2342592 used= 2269696 count= 286 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:27.723389] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:27.723401] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= RPC_CTX_ID hold= 12582912 used= 294912 limit= 1073741800 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 294912 used= 195072 count= 12 avg_used= 16256 mod=SUMMARY [2025-08-14 09:01:27.723414] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= TX_DATA_TABLE hold= 5570560 used= 2342784 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 8 washed_size= 2818048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2342784 used= 2319424 count= 30 avg_used= 77314 mod=SUMMARY [2025-08-14 09:01:27.723427] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= MDS_DATA_ID hold= 3866624 used= 2785280 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 5 washed_size= 4521984 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2785280 used= 2698240 count= 340 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:27.723443] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= MERGE_NORMAL_CTX_ID hold= 1179648 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 8 washed_blocks= 24 washed_size= 15597568 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:27.723469] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant: 1002, limit: 8589934592 hold: 1941843968 cache_hold: 48234496 cache_used: 48234496 cache_item_count: 23 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 1007513600 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 415236096 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 1114112 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 950272 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 211812352 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 1441792 limit= 429496725 [MEMORY] ctx_id= GLIBC hold_bytes= 12582912 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 149913600 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 37748736 limit= 1717986900 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 7766016 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 3883008 limit= 8589934500 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 5718016 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3047424 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 1032192 limit= 9223372036854775807 [2025-08-14 09:01:27.723500] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1002 ctx_id= DEFAULT_CTX_ID hold= 1007513600 used= 916257904 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 15 [MEMORY] hold= 185860096 used= 185534336 count= 114 avg_used= 1627494 block_cnt= 114 chunk_cnt= 104 mod=MysqlRequesReco [MEMORY] hold= 167792640 used= 167772192 count= 1 avg_used= 167772192 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 83927040 used= 83886080 count= 2 avg_used= 41943040 block_cnt= 2 chunk_cnt= 2 mod=LogGroupBuffer [MEMORY] hold= 67190784 used= 67109344 count= 4 avg_used= 16777336 block_cnt= 4 chunk_cnt= 4 mod=MrgMemPoolChk [MEMORY] hold= 916257904 used= 899064929 count= 9453 avg_used= 95108 mod=SUMMARY [2025-08-14 09:01:27.723514] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= MEMSTORE_CTX_ID hold= 415236096 used= 411992064 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 411992064 used= 411803568 count= 198 avg_used= 2079816 block_cnt= 198 chunk_cnt= 198 mod=Memstore [MEMORY] hold= 411992064 used= 411803568 count= 198 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:01:27.723524] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= EXECUTE_CTX_ID hold= 1114112 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 7 washed_size= 7274496 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:27.723534] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= TRANS_CTX_MGR_ID hold= 950272 used= 933888 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1146880 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 933888 used= 921984 count= 2 avg_used= 460992 mod=SUMMARY [2025-08-14 09:01:27.723545] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= PLAN_CACHE_CTX_ID hold= 211812352 used= 201995008 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 176930880 used= 160228962 count= 13450 avg_used= 11912 block_cnt= 13450 chunk_cnt= 101 mod=SqlPhyPlan [MEMORY] hold= 201995008 used= 183048143 count= 17906 avg_used= 10222 mod=SUMMARY [2025-08-14 09:01:27.723557] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1002 ctx_id= WORK_AREA hold= 1441792 used= 0 limit= 429496725 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 14 washed_blocks= 54 washed_size= 27918336 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:27.723567] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= GLIBC hold= 12582912 used= 11425280 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 11425280 used= 6322556 count= 19216 avg_used= 329 mod=SUMMARY [2025-08-14 09:01:27.723579] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= CO_STACK hold= 149913600 used= 147087360 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 68 washed_blocks= 70 washed_size= 3178496 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 147087360 used= 146813760 count= 285 avg_used= 515136 block_cnt= 285 chunk_cnt= 73 mod=CoStack [MEMORY] hold= 147087360 used= 146813760 count= 285 avg_used= 515136 mod=SUMMARY [2025-08-14 09:01:27.723593] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= KVSTORE_CACHE_ID hold= 48234496 used= 48234496 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 48234496 used= 0 count= 23 avg_used= 0 mod=SUMMARY [2025-08-14 09:01:27.723603] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= META_OBJ_CTX_ID hold= 37748736 used= 35741632 limit= 1717986900 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 35741632 used= 32457488 count= 496 avg_used= 65438 mod=SUMMARY [2025-08-14 09:01:27.723614] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= TX_CALLBACK_CTX_ID hold= 7766016 used= 6934400 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 17 washed_size= 6914048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 6934400 used= 6729728 count= 848 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:27.723626] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1002 ctx_id= LOB_CTX_ID hold= 147456 used= 16384 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 16384 used= 15872 count= 2 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:27.723638] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= RPC_CTX_ID hold= 3883008 used= 991232 limit= 8589934500 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 6 washed_blocks= 6 washed_size= 10797056 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 991232 used= 645504 count= 45 avg_used= 14344 mod=SUMMARY [2025-08-14 09:01:27.723650] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= TX_DATA_TABLE hold= 5718016 used= 4521856 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 9 washed_blocks= 16 washed_size= 13156352 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4521856 used= 4480128 count= 40 avg_used= 112003 mod=SUMMARY [2025-08-14 09:01:27.723662] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= MDS_DATA_ID hold= 3047424 used= 1998848 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 4 washed_size= 1146880 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1998848 used= 1936384 count= 244 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:27.723672] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= MDS_CTX_ID hold= 147456 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:27.723684] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:27.723694] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= MERGE_NORMAL_CTX_ID hold= 1032192 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 12 washed_size= 13647872 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:27.728141] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=27] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:01:27.728292] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:27.728543] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=35] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:27.728604] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=56] temporary file module use no blocks [2025-08-14 09:01:27.728614] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:27.728631] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=15] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:27.728649] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=17] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43622989164578, flush_io_finished_ret_:0, flush_io_finished_round_:26236, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:27.733788] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891970090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:27.724, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:27.733888] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:27.733911] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162087733884], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:27.733960] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162087733884], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:27.733976] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162087733884]}) [2025-08-14 09:01:27.734022] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=41] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:27.734030] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162087734020], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:27.734060] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162087734020]}) [2025-08-14 09:01:27.734406] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:01:27.734432] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:01:27.734446] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162087734402], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:27.734473] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=24] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162087734402]}) [2025-08-14 09:01:27.734666] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:01:27.734694] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:01:27.734713] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:01:27.734726] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:00:41.841, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=08:59:41.841], idx=471, next_allow_print_ts="09:01:11.841") [2025-08-14 09:01:27.734747] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:27.734752] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:01:27.734763] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=15] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3706) [2025-08-14 09:01:27.734772] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:01:27.734771] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:01:27.734784] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:01:27.734801] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:01:27.734814] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:00:52.998, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:22.998], idx=771, next_allow_print_ts="09:01:22.998") [2025-08-14 09:01:27.734786] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:27.734835] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=48] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3707) [2025-08-14 09:01:27.734833] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=08:59:57.942, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:27.942], idx=868, next_allow_print_ts="09:00:27.942") [2025-08-14 09:01:27.734843] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:01:27.735017] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:27.735028] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=11] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3708) [2025-08-14 09:01:27.735035] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:01:27.735033] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:01:27.735051] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:01:27.735065] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:01:27.735171] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:27.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.736643] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:27.737261] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:27.742895] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:27.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.743149] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=41] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:27.744046] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:27.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.745403] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:27.745439] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [699][T1001_DupTbLeas][T1001][Y0-0000000000000000-0-0] [lt=2956804][errcode=-4389] timer task use too much time(end=1755162087745438, start=1755162039788215, delta=47957223) [2025-08-14 09:01:27.746439] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:27.749740] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:27.749771] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:27.753188] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:27.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.753599] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=15] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:27.753881] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:27.754150] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=66][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7ef590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:27.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:27.754186] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7ef710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:27.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:27.755253] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:27.755275] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=49] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:27.759255] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:27.760968] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=39] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:27.761006] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=35] temporary file module use no blocks [2025-08-14 09:01:27.761014] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:27.761028] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:27.761039] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43622989200655, flush_io_finished_ret_:0, flush_io_finished_round_:26269, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:26.752610] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1007509, owner: ObLog, click_count: 7, time dist:[FORMAT_END=1007500, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=5, ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=1], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:26.762863] EDIAG [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:275) [804][T1002_TimerWK0_][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1009509, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1009503, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1c45261f 0x1c44fbfd 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:26.752610] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:27.762473] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [726][T1001_ObXAHbWor][T1001][Y0-0000000000000000-0-0] [lt=978735] XA scheduler heartbeat task statistics(avg_time=2) [2025-08-14 09:01:27.762968] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=23] tx gc loop thread is running(MTL_ID()=1001) [2025-08-14 09:01:27.762988] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=18] try gc retain ctx [2025-08-14 09:01:27.763012] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=25] try to retry start_working [2025-08-14 09:01:27.763424] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:27.764706] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=53][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7281410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:27.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:27.765807] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:27.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:26.754946] INFO pn_ratelimit (group.c:653) [14881][TimerWK4_IngressService][T0][Y0-0000000000000000-0-0] [lt=21292] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-08-14 09:01:27.768320] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=689793][errcode=0] EASY SLOW: thread: 7f9954179300, lastrun: 1755162086.752603 cost: 0.168541 loop:26457, slowcnt: 1 [2025-08-14 09:01:27.769421] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=28] gc stale ls task succ [2025-08-14 09:01:27.769585] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=17] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:01:27.769630] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:26.762863] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:275) [804][T1002_TimerWK0_][T1002][Y0-0000000000000000-0-0] [lt=7] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=2, times=1289) [2025-08-14 09:01:27.773933] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [804][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=1009509][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a5378, timer:0x7f9922ffd708, task:0x7f9922ffd730, task_type:"N9oceanbase7storage10checkpoint17ObTabletGCService18ObTabletChangeTaskE", scheduled_time:1755162053754432, pushed_time:1755162053754691, delay:5000000}, delay_in_thread_pool=31011570, threashold=500000) [2025-08-14 09:01:27.776266] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=121][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919ab110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:27.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:27.776295] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891969710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:27.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:27.776319] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d36b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f72ebb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:27.767, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:27.776331] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891971d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:27.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:27.776564] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1007509][errcode=0] epoll handler too much time :1009096 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:27.776591] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:27.777393] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=42] gc stale ls task succ [2025-08-14 09:01:27.777443] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=20] current monitor number(seq_=-1) [2025-08-14 09:01:27.784271] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:27.786330] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919b4a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:27.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:27.786363] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72739b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919a5e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:27.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:27.786379] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72779b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f72eab10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:27.777, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:27.786392] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f728bc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:27.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:27.796421] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919902b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f989196c390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:27.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:27.796457] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f72ec610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:27.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:27.796496] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891973810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:27.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:27.806489] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7277730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f72ea990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:27.794, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:27.806526] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891825c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:27.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:27.821802] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [571][T1_ArcSender][T1][YB42AC4D0007-00063C495BA99C16-0-0] [lt=658559][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:01:27.821928] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:27.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.826598] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7283e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:27.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:27.835225] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:27.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.836666] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7288390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:27.820, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:27.846727] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7288210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:27.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:27.854167] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=51][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:01:27.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.857809] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:01:27.857840] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=26] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:01:27.893725] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=34] PNIO [ratelimit] time: 1755162087893722, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007586, add_bytes: 0 [2025-08-14 09:01:27.921977] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:27.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.935269] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=53][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:27.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.935329] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=753579][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162087.789242 cost: 0.146084 loop:26436, slowcnt: 1 [2025-08-14 09:01:27.967355] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f989196aa90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:27.952, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:27.967388] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7288e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:27.953, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:27.987477] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891992fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f989196de90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:27.970, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:27.991794] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:27.989, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:27.995726] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:27.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.011911] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:28.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.022045] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=55][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:27.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.035325] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:27.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.092410] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=58][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:28.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.122111] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:28.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.122300] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=17] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:01:28.135386] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:28.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.213030] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=50][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:28.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.222173] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:28.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.234737] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=11] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:28.234779] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=39][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162088234730], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:28.234811] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162088234730], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:28.234829] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162088234730]}) [2025-08-14 09:01:28.234851] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:01:28.234875] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=24] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:01:28.234886] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=7] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:28.234899] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162088234884], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:28.234916] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162088234884]}) [2025-08-14 09:01:28.235117] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:01:28.235132] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:01:28.235122] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:28.235143] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=20] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3709) [2025-08-14 09:01:28.235147] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162088235112], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:28.235153] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:01:28.235161] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162088235112]}) [2025-08-14 09:01:28.235168] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:28.235177] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3710) [2025-08-14 09:01:28.235185] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:01:28.235388] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:01:28.235409] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=19] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:28.235419] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:01:28.235427] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:227) [1031][PxTargetMgr0][T1002][Y0-0000000000000000-0-0] [lt=15] refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3711) [2025-08-14 09:01:28.235437] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:01:28.235441] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:01:28.235456] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:00:41.841, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=08:59:41.841], idx=471, next_allow_print_ts="09:01:11.841") [2025-08-14 09:01:28.235456] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:28.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.243211] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:28.239, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.246966] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:28.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.257023] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:28.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.263334] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:28.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.322247] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=50][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:28.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.335506] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:28.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.422289] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:28.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.435535] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:28.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.476521] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fa190, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa470, time_wheel:0x7f99341cdbf0, thread_pool:0x7f99341cd3d0, is_running:1, time_interval:2.00s, expected_run_time:09:01:28.334, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.494575] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:28.489, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.504312] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:28.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.514703] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:28.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.522322] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:28.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.535596] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:28.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.586872] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fe410, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe6f0, time_wheel:0x7f9922df9d40, thread_pool:0x7f9922df9520, is_running:1, time_interval:2.00s, expected_run_time:09:01:28.432, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.595191] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:28.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.622375] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:28.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.635620] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:28.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.677862] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:01:28.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.690602] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EE95-0-0] [lt=26] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:01:28.716020] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:28.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.719344] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:28.720609] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:28.720663] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:28.722437] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:28.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.727761] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=131] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:28.728831] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:28.728866] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27] temporary file module use no blocks [2025-08-14 09:01:28.728878] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:28.728895] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=15] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:28.728919] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=23] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43622990168446, flush_io_finished_ret_:0, flush_io_finished_round_:26237, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:27.768362] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:288) [732][T1001_ArcFetche][T1001][YB42AC4D0007-00063C495CA99C16-0-0] [lt=5936] ObArchiveFetcher is running(thread_index=0) [2025-08-14 09:01:28.735419] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:28.735442] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162088735415], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:28.735462] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162088735415], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:28.735483] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162088735415]}) [2025-08-14 09:01:28.735509] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=21] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:28.735527] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162088735507], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:28.735543] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162088735507]}) [2025-08-14 09:01:28.735748] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:01:28.735765] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:01:28.735780] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162088735745], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:28.735792] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162088735745]}) [2025-08-14 09:01:28.735966] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:28.735995] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=28] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3712) [2025-08-14 09:01:28.736011] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:01:28.736025] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:28.736038] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=13] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3713) [2025-08-14 09:01:28.736055] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:01:28.736111] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=125][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc190, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc470, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:10.00s, expected_run_time:09:01:28.720, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.736265] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:28.736282] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=16] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3714) [2025-08-14 09:01:28.736287] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:01:28.736321] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:28.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.736382] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:28.737102] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:28.738904] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=35] LogLoopThread round_cost_time(us)(round_cost_time=7) [2025-08-14 09:01:27.822245] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:01:27.789253] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:28.739392] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=948702][errcode=0] epoll handler too much time :950149 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:28.739384] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=915750][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:28.739407] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:28.235491] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:01:28.739428] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=503165][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:01:28.739449] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:01:28.739465] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:01:28.739479] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:01:28.739492] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:00:52.998, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:22.998], idx=771, next_allow_print_ts="09:01:22.998") [2025-08-14 09:01:28.739510] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=08:59:57.942, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:27.942], idx=868, next_allow_print_ts="09:00:27.942") [2025-08-14 09:01:28.739681] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:01:28.739696] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:01:28.739710] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:01:25.772315] WDIAG [COMMON] wait (ob_io_define.cpp:880) [816][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=-4012] fail to wait result condition(ret=-4012, wait_ms=300000, *this={is_inited_:true, is_finished_:false, is_canceled_:false, has_estimated_:false, complete_size_:0, offset_:9269248, size_:4096, timeout_us_:300000000, result_ref_cnt_:2, out_ref_cnt_:1, flag_:{mode:"WRITE", group_id_:0, func_type_:12, wait_event_id_:16, is_sync_:true, is_unlimited_:false, is_detect_:false, is_write_through_:false, is_sealed_:true, is_time_detect_:false, need_close_dev_and_fd_:false, reserved_:0}, ret_code_:{io_ret_:0, fs_errno_:0}, tenant_id_:1002, tenant_io_mgr_:{ptr:0x7f9922e86030}, user_data_buf_:null, buf_:0x7f98ffe05000, io_callback_:null, time_log:{begin_ts:1755161785772262, enqueue_used:3, dequeue_used:-1, submit_used:-1, return_used:-1, callback_enqueue_used:-1, callback_dequeue_used:-1, callback_finish_used:-1, end_used:-1}}) [2025-08-14 09:01:28.740450] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [14780][T1001_TimerWK4_SSTableGC][T1001][YB42AC4D0007-00063C4974099C4E-0-0] [lt=24][errcode=0] cost too much time:mds_row.ipp:int oceanbase::storage::mds::MdsRow::get_latest_committed(READ_OP &&) const [K = oceanbase::storage::mds::DummyKey, V = oceanbase::..., (this=|threshold=5.00ms|start at 09:01:27.754|541=0us|total=986.07ms) [2025-08-14 09:01:28.740489] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [14780][T1001_TimerWK4_SSTableGC][T1001][YB42AC4D0007-00063C4974099C4E-0-0] [lt=31][errcode=0] cost too much time:mds_unit.ipp:int oceanbase::storage::mds::MdsUnit::get_latest_committed(OP &&) const [K = oceanbase::storage::mds::DummyKey, V = oceanbase::sto..., (this=|threshold=10.00ms|start at 09:01:27.754|910=1us|911=0us|total=986.11ms) [2025-08-14 09:01:27.722061] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=969167] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=32, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:27.722061] EDIAG [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=969167][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1017512, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1017500, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=8], seq:[0, 1, 2, 3, 4], BACKTRACE: 0xfa06846 0xfa05f62 0xfa05c84 0xfa05a50 0x1b51bd85 0x1b51b9c4 0x97a8d82 0x97a88d2 0x97a7481 0x97a6a43 0x97a42c5 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:28.741389] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [804][T1002_TimerWK0_TabletShell][T1002][Y0-0000000000000000-0-0] [lt=25] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-08-14 09:01:28.741414] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [804][T1002_TimerWK0_TabletShell][T1002][Y0-0000000000000000-0-0] [lt=18] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=2, times=1280) [2025-08-14 09:01:28.741428] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [804][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a5458, timer:0x7f9922ffd748, task:0x7f9922ffd770, task_type:"N9oceanbase7storage10checkpoint16ObEmptyShellTaskE", scheduled_time:1755162053754488, pushed_time:1755162053754696, delay:5000000}, delay_in_thread_pool=34019273, threashold=500000) [2025-08-14 09:01:28.742330] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:28.742430] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:28.730479] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=25] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:01:28.743563] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919b9690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:28.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:28.743603] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891968e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:28.737, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:28.743617] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e71b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891973210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:28.739, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:28.743630] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918119b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f73a1090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:28.737, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:28.745717] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:28.747257] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:28.748392] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:28.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.754421] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72774b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f72ea810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:28.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:28.754460] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fbe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7280c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:28.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:28.755495] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:28.755654] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8030, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8310, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:10.00s, expected_run_time:09:01:28.730, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.755696] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:28.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.755719] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:28.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.755802] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:28.755830] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:28.755881] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:28.755890] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:28.755898] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] temporary file module use no blocks [2025-08-14 09:01:28.755906] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:28.755916] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=9] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:28.755927] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43622990192430, flush_io_finished_ret_:0, flush_io_finished_round_:26263, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:28.756016] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:28.756043] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=17] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:28.756081] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:28.754120] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:355) [835][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19] finish check and clear obj lock(ls->get_ls_id()={id:1001}) [2025-08-14 09:01:28.764197] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [835][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10053][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99319fe030, caller:ob_table_lock_service.cpp:start:276, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE6$_1273, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe310, time_wheel:0x7f99535f3990, thread_pool:0x7f99535f3170, is_running:0, time_interval:10.00s, expected_run_time:09:01:36.426, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.764222] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [835][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:01:28.764|total=2.24s) [2025-08-14 09:01:28.764873] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891968990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:28.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:28.764908] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891973090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:28.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:28.764924] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7280b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:28.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:28.765972] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:28.769465] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:28.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.769667] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=15] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:01:28.772555] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:28.742816] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=939988] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:01:28.742597] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=964292] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:01:28.742340] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=939488] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:01:28.773348] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [14780][T1001_TimerWK4_SSTableGC][T1001][YB42AC4D0007-00063C4974099C4E-0-0] [lt=17][errcode=0] cost too much time:mds_table_impl.ipp:virtual int oceanbase::storage::mds::MdsTableImpl, oceanbase::st..., (this=|threshold=5.00ms|start at 09:01:25.746|663=0us|total=3.03s) [2025-08-14 09:01:28.740174] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [869][T1002_MultiVers][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1515759, expected_run_ts_us_=1755162086435309, ObClockGenerator::getRealClock()=1755162087951069, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fe960, caller:ob_multi_version_garbage_collector.cpp:start:115, function_type:ZN9oceanbase19concurrency_control30ObMultiVersionGarbageCollector5startEvE6$_1159, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fec40, time_wheel:0x7f99535f7990, thread_pool:0x7f99535f7170, is_running:1, time_interval:60.00s, expected_run_time:09:01:26.435, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.774030] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [869][T1002_MultiVers][T1002][Y0-0000000000000000-0-0] [lt=33801][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fe960, caller:ob_multi_version_garbage_collector.cpp:start:115, function_type:ZN9oceanbase19concurrency_control30ObMultiVersionGarbageCollector5startEvE6$_1159, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fec40, time_wheel:0x7f99535f7990, thread_pool:0x7f99535f7170, is_running:1, time_interval:60.00s, expected_run_time:09:01:26.435, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.774050] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [869][T1002_MultiVers][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:27.951|207=822.98ms|210=0us|222=2us|total=822.99ms) [2025-08-14 09:01:28.774064] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [869][T1002_MultiVers][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] timer task use too much time(end=1755162088774063, start=1755162087951068, delta=822995) [2025-08-14 09:01:28.774598] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=16] gc stale ls task succ [2025-08-14 09:01:28.774957] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919aef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919b9090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:28.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:28.774984] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7270a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919a4190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:28.762, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:28.774999] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891971710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:28.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:28.775022] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f73a0f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:28.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:28.775032] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7ef890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:28.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:28.777424] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=19] gc stale ls task succ [2025-08-14 09:01:28.777550] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=23] current monitor number(seq_=-1) [2025-08-14 09:01:28.784306] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:28.785017] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f734de30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891919810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:28.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:28.785056] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891972f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:28.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:28.785068] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f989196b090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:28.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:28.785090] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d59b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f72ed090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:28.773, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:28.785107] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7efa10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:28.777, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:28.788480] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:28.795071] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f72eba10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:28.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:28.795110] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7f0190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:28.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:28.805130] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f728a790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:28.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:28.805162] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7281110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:28.790, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:28.808853] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EE85-0-0] [lt=23] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-08-14 09:01:28.822498] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:28.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.835297] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7f1690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:28.827, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:28.836385] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:28.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.839468] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:28.845356] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739af30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f989196bc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:28.837, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:28.848474] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=829680][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a5fb8, timer:0x7f994b3feee0, task:0x7f994b3fef80, task_type:"N9oceanbase8observer18ObTableLoadService21ObClientTaskPurgeTaskE", scheduled_time:1755162027715356, pushed_time:0, delay:1000000}, running_task_token={this:0x7f994e1a5c38, timer:0x7f994b3feee0, task:0x7f994b3fef20, task_type:"N9oceanbase8observer18ObTableLoadService15ObHeartBeatTaskE", scheduled_time:1755162023721549, pushed_time:1755162047763044, delay:10000000}, total_delay=61133116, threashold=1000000, tenant_id=1) [2025-08-14 09:01:28.854791] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:01:28.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.855418] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f739e510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:28.841, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:28.855451] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7f0310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:28.849, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:28.855466] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7280f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:28.849, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:28.858990] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:01:28.859034] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=38] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:01:28.864248] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c0b90, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571c0e70, time_wheel:0x7f9959fef990, thread_pool:0x7f9959fef170, is_running:1, time_interval:3.00s, expected_run_time:09:01:28.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.901342] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=26] PNIO [ratelimit] time: 1755162088901339, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007617, add_bytes: 0 [2025-08-14 09:01:28.905718] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f728a610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:28.892, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:28.922531] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:28.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.925834] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7ee090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:28.917, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:28.935759] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb3c38, timer:0x7f99333cb180, task:0x7f99333cb1c0, task_type:"N9oceanbase8observer18ObTableLoadService15ObHeartBeatTaskE", scheduled_time:1755161967770316, pushed_time:0, delay:10000000}, running_task_token={this:0x7f9931bb3ed8, timer:0x7f99333cb180, task:0x7f99333cb208, task_type:"N9oceanbase8observer18ObTableLoadService25ObClientTaskAutoAbortTaskE", scheduled_time:1755161959782824, pushed_time:1755162045729074, delay:1000000}, total_delay=121165440, threashold=1000000, tenant_id=1001) [2025-08-14 09:01:28.935880] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7281a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:28.922, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:28.936449] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:28.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.939529] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:28.990110] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:28.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.990892] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:28.989, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.011021] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:29.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.022595] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:28.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.036513] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:28.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.039625] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:29.091397] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:29.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.122658] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:29.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.123028] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=26] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:01:29.136576] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:29.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.139613] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:29.212048] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:29.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.222696] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:29.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.236018] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:29.236055] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=34][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162089236014], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:29.236075] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162089236014], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:29.236092] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162089236014]}) [2025-08-14 09:01:29.236107] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:01:29.236127] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=20] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:01:29.236143] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=12] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:29.236156] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162089236142], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:29.236180] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=23] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162089236142]}) [2025-08-14 09:01:29.236402] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:01:29.236421] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:01:29.236432] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162089236397], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:29.236455] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162089236397]}) [2025-08-14 09:01:29.236636] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:29.236656] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=18] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3715) [2025-08-14 09:01:29.236662] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:01:29.236672] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:29.236685] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=13] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3716) [2025-08-14 09:01:29.236690] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:01:29.236894] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=23] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:29.236913] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=19] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3717) [2025-08-14 09:01:29.236918] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:01:29.236934] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:29.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.239673] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:29.239888] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:01:29.239913] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:01:29.239936] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:01:29.239950] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:00:41.841, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=08:59:41.841], idx=471, next_allow_print_ts="09:01:11.841") [2025-08-14 09:01:29.239971] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:01:29.239992] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:01:29.240017] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:01:29.240030] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:01:29.240046] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:01:29.240058] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:01:29.240070] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:00:52.998, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:22.998], idx=771, next_allow_print_ts="09:01:22.998") [2025-08-14 09:01:29.240088] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=08:59:57.942, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:27.942], idx=868, next_allow_print_ts="09:00:27.942") [2025-08-14 09:01:29.240265] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:01:29.240280] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:01:29.240292] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:01:29.241415] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:29.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.242158] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:29.239, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.251429] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:29.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.262239] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:29.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.277893] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:29.265, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.322759] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:29.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.337014] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:29.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.339700] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:29.422834] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:29.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.437072] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=52][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:29.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.439742] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:29.492594] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:29.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.493351] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:29.489, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.513476] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:29.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.522905] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:29.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.529100] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:29.515, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.537122] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:29.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.539805] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:29.593994] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:29.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.622960] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:29.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.637185] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:29.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.639864] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:29.678461] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:01:29.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:00:06.777042] EDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [957][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8EF5-0-0] [lt=15062][errcode=-4341] LOGGER COST TOO MUCH TIME, cost: 1946294, owner: ObLog, click_count: 5, time dist:[FORMAT_END=983689, ObMalloc#start=1, ObMalloc#end=962601, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0x1c028aa9 0x1c028835 0x1c031b7e 0x1c03844f 0x984d055 0x9280839 0x926c154 0x12c5a0c9 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:29.714628] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:29.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.718565] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:29.718599] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=26] tx gc loop thread is running(MTL_ID()=1) [2025-08-14 09:01:29.718609] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] try gc retain ctx [2025-08-14 09:01:29.719380] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:29.720732] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:29.720828] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:29.723067] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:29.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.726704] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:29.729031] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=31] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:29.729061] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=25] temporary file module use no blocks [2025-08-14 09:01:29.729071] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:29.729084] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:29.729099] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43622991168719, flush_io_finished_ret_:0, flush_io_finished_round_:26238, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:29.731574] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919b9390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:29.728, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:29.731623] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f734c030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891807390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:29.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:29.732572] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [556][T1_OBJLockGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fafd0, caller:ob_table_lock_service.cpp:start:276, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE6$_1273, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fb2b0, time_wheel:0x7f994b19db50, thread_pool:0x7f994b19d330, is_running:1, time_interval:10.00s, expected_run_time:09:01:29.696, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.736683] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:29.736707] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162089736678], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:29.736727] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162089736678], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:29.736743] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162089736678]}) [2025-08-14 09:01:29.736758] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=5, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:01:29.736775] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=12] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:29.736783] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162089736773], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:29.736795] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162089736773]}) [2025-08-14 09:01:29.736957] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:01:29.736980] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:01:29.736994] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162089736953], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:29.737007] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162089736953]}) [2025-08-14 09:01:29.737193] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:29.737209] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=15] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3718) [2025-08-14 09:01:29.737217] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:01:29.737230] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:29.737252] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=21] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3719) [2025-08-14 09:01:29.737259] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:01:29.737227] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:29.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.737373] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:29.737427] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:29.737440] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=13] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3720) [2025-08-14 09:01:29.737448] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:01:29.737473] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:29.718574] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:01:29.718633] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15] get_max_applied_scn(scn={val:1755161547738516001, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:50, palf_committed_end_lsn_:{lsn:5282737655}, palf_committed_end_scn_:{val:1755161547738516001, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161547738516001, v:0}}) [2025-08-14 09:01:28.763779] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=8] get_max_applied_scn(scn={val:1755161719775179002, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:45, palf_committed_end_lsn_:{lsn:8461098908}, palf_committed_end_scn_:{val:1755161719775179002, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161719775179002, v:0}}) [2025-08-14 09:01:29.739928] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:29.740478] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:01:29.740506] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:01:29.740527] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:01:29.740544] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:00:41.841, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=08:59:41.841], idx=471, next_allow_print_ts="09:01:11.841") [2025-08-14 09:01:29.740566] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:01:29.740583] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:01:29.740606] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:01:29.740623] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:01:29.740642] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:01:29.740655] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:01:29.740664] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:00:52.998, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:22.998], idx=771, next_allow_print_ts="09:01:22.998") [2025-08-14 09:01:29.740693] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=08:59:57.942, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:27.942], idx=868, next_allow_print_ts="09:00:27.942") [2025-08-14 09:01:29.740853] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:01:29.740874] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:01:29.740883] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:01:29.741336] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=82] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:00:06.757378] EDIAG [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6175) [511][T1_TimerWK2_SSTableGC][T1][YB42AC4D0007-00063C495D6A02D9-0-0] [lt=15][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1977939, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1977937, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x1bb90396 0x975a989 0x975972e 0x9758a1f 0x1aee8055 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:29.742430] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:29.742431] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919aae10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:29.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:29.742465] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891971b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:29.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:29.742480] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e67b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891972c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:29.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:29.742493] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f739e390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:29.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:29.742639] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=12003] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:01:29.743974] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:29.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.744599] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:29.745035] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:29.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.747224] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:29.749258] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:29.749263] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:29.752535] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891968690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:29.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:29.752568] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891968510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:29.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:29.752583] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891972a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:29.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:29.752596] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f72ed390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:29.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:29.754037] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=51][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:29.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:28.959086] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] get_fetch_log_dst failed or invalid(ret=0, palf_id=1, self="172.77.0.7:2882", fetch_log_dst="172.77.0.7:2882") [2025-08-14 09:01:29.755868] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:29.755897] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=14] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:29.755931] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:29.756316] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:29.756333] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] temporary file module use no blocks [2025-08-14 09:01:29.756342] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:29.756354] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:29.756367] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43622991195723, flush_io_finished_ret_:0, flush_io_finished_round_:26264, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:29.756437] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:29.759659] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:29.761257] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=30] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:29.761282] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=21] temporary file module use no blocks [2025-08-14 09:01:29.761289] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=5] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:29.761313] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=23] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:29.761332] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=19] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43622991200946, flush_io_finished_ret_:0, flush_io_finished_round_:26271, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:29.763890] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:29.765298] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:29.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.765694] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7280090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:29.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:29.769742] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=21] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:01:29.772695] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=57] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:29.774692] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:01:29.775756] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:29.765, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.777492] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=19] gc stale ls task succ [2025-08-14 09:01:29.777639] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=17] current monitor number(seq_=-1) [2025-08-14 09:01:29.784251] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:29.785872] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72707b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919a4010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:29.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:29.789362] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:29.795926] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919b4d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:29.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:29.805941] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:29.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.805995] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739acb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f989196bb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:29.798, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:29.816055] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891973690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:29.806, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:29.816089] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891973510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:29.809, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:29.823093] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:29.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.826115] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f989196b510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:29.815, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:29.837290] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=77][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:29.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.839980] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:29.846256] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891973390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:29.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:29.846319] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=60][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f72ed210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:29.832, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:29.856319] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f72eb410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:29.841, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:29.856548] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc4578, timer:0x7f9970f40f80, task:0x564d6d07a498, task_type:"N9oceanbase8observer26ObRemoteMasterRsUpdateTaskE", scheduled_time:1755162028742204, pushed_time:0, delay:600000000}, running_task_token={this:0x7f9970fc5fb8, timer:0x7f9970f40f80, task:0x564d727182f0, task_type:"N9oceanbase5share16ObAshRefreshTaskE", scheduled_time:1755161611717613, pushed_time:1755161614757742, delay:120000000}, total_delay=61114343, threashold=1000000, tenant_id=500) [2025-08-14 09:01:29.857586] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=75][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd070, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fd350, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:5.00s, expected_run_time:09:01:29.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.857617] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:01:29.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.866385] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f989196af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:29.859, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:29.908959] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=26] PNIO [ratelimit] time: 1755162089908957, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007618, add_bytes: 0 [2025-08-14 09:01:29.923117] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:29.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.937355] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:29.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.940043] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:29.996007] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:29.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:29.996604] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:29.989, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.016666] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:30.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.023182] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:29.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.027179] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:30.151, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.037376] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:29.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.040116] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:30.097075] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:30.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.123245] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:30.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.123688] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=19] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:01:30.137437] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:30.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.140175] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:30.217753] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:30.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.223315] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:30.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.237230] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=17] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:30.237266] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=34][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162090237226], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:30.237288] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162090237226], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:30.237308] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162090237226]}) [2025-08-14 09:01:30.237322] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162090237226]}) [2025-08-14 09:01:30.237337] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:01:30.237351] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:01:30.237361] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=5] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:30.237367] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162090237359], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:30.237383] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162090237359]}) [2025-08-14 09:01:30.237393] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] gts statistics(tenant_id=1001, gts_rpc_cnt=0, get_gts_cache_cnt=11, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:01:30.237409] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162090237359]}) [2025-08-14 09:01:30.237457] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:30.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.237526] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:30.237547] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:227) [1031][PxTargetMgr0][T1][Y0-0000000000000000-0-0] [lt=18] refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3721) [2025-08-14 09:01:30.237557] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:01:30.237572] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:30.237582] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3722) [2025-08-14 09:01:30.237601] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:01:30.237598] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:01:30.237620] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=21] get number(ret=-4038, service_type_=0, range=1, base_id=1755162090237152000, start_id=0, end_id=140295968341544) [2025-08-14 09:01:30.237634] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:01:30.237786] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:30.237799] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=12] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3723) [2025-08-14 09:01:30.237808] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:01:30.240212] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:30.241054] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:01:30.241080] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:01:30.241105] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:01:30.241121] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:00:41.841, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=08:59:41.841], idx=471, next_allow_print_ts="09:01:11.841") [2025-08-14 09:01:30.241143] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:01:30.241159] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:01:30.241190] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:01:30.241208] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:01:30.241220] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:01:30.241248] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:01:30.241260] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:00:52.998, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:22.998], idx=771, next_allow_print_ts="09:01:22.998") [2025-08-14 09:01:30.241278] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=08:59:57.942, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:27.942], idx=868, next_allow_print_ts="09:00:27.942") [2025-08-14 09:01:30.241452] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:01:30.241466] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:01:30.241489] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:01:30.247743] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:30.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.247930] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:30.239, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.257805] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:30.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.268050] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:30.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.278559] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:30.265, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.308742] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:30.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.323344] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:30.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.337517] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:30.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.340270] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:30.423372] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:30.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.437550] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:30.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.440330] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:30.477704] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fa190, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa470, time_wheel:0x7f99341cdbf0, thread_pool:0x7f99341cd3d0, is_running:1, time_interval:2.00s, expected_run_time:09:01:30.334, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.499145] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:30.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.499282] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:30.489, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.519416] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:30.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.523433] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:30.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.529901] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:30.515, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.537611] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:30.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.540390] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:30.587951] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fe410, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe6f0, time_wheel:0x7f9922df9d40, thread_pool:0x7f9922df9520, is_running:1, time_interval:2.00s, expected_run_time:09:01:30.432, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.599881] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:30.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.623500] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:30.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.637653] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:30.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.678995] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:01:30.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.710484] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:30.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.719294] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:30.719758] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:30.721547] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:30.721605] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73516b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f989196f510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:30.716, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:30.724219] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:30.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.728169] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=56] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:30.729202] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:30.729230] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=22] temporary file module use no blocks [2025-08-14 09:01:30.729253] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=22] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:30.729378] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=123] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:30.729392] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43622992168893, flush_io_finished_ret_:0, flush_io_finished_round_:26239, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:30.640450] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:30.730192] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19036] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:01:30.731374] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=1129] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:01:30.731429] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:01:30.731470] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:01:30.731495] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=1, sync_total=0, async_cnt= 1 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:01:30.731509] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:01:30.731820] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919b5990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:30.720, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:30.731852] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919b5810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:30.727, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:30.731867] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f72ed690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:30.727, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:30.731886] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72851b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7289410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:30.721, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:30.726678] INFO [PALF] check_parent_health (log_config_mgr.cpp:2875) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=28] re_register_parent reason(palf_id=1, self="172.77.0.7:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1755162084734527, last_first_register_time_us=1755158728414267, register_time_us=1755162084734527, curr_time_us=1755162088838979) [2025-08-14 09:01:30.737818] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:30.739489] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:30.739537] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:30.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.739664] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:30.739683] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=17] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3724) [2025-08-14 09:01:30.739692] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:01:30.739706] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:30.739715] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3725) [2025-08-14 09:01:30.739723] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:01:30.739910] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:30.739921] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=11] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3726) [2025-08-14 09:01:30.739929] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:01:30.740513] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=89756] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:01:30.740538] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:30.741689] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:01:30.741713] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:01:30.741732] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:01:30.741744] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:00:41.841, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=08:59:41.841], idx=471, next_allow_print_ts="09:01:11.841") [2025-08-14 09:01:30.741762] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:01:30.741774] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:01:30.741793] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:01:30.741807] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:01:30.741824] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:01:30.741837] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:01:30.741849] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:00:52.998, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:22.998], idx=771, next_allow_print_ts="09:01:22.998") [2025-08-14 09:01:30.741874] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=08:59:57.942, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:27.942], idx=868, next_allow_print_ts="09:00:27.942") [2025-08-14 09:01:30.741862] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:30.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:30.741938] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f739e210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:30.737, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:30.741965] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98f7281d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:30.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:30.742119] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:01:30.742135] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:01:30.742148] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:01:30.742505] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:30.742546] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=30] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:30.743842] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=30] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:01:30.744968] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:30.745004] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [14780][T1001_TimerWK4_SSTableGC][T1001][YB42AC4D0007-00063C4974099C4E-0-0] [lt=41][errcode=0] cost too much time:ob_i_tablet_mds_interface.ipp:int oceanbase::storage::ObITabletMdsInterface::get_mds_data_from_tablet(const K &, const share::SCN &, const int64_t, const common::ObFunction &) const [K = oceanbase::storage::mds::Du..., (this=|threshold=10.00ms|start at 09:01:30.730|160=1us|total=14.75ms) [2025-08-14 09:01:30.745053] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [14780][T1001_TimerWK4_SSTableGC][T1001][YB42AC4D0007-00063C4974099C4E-0-0] [lt=41][errcode=0] cost too much time:ob_i_tablet_mds_interface.ipp:int oceanbase::storage::ObITabletMdsInterface::get_latest_committed(OP &&) const [T = oceanbase::storage::ObTabletCreateDeleteMdsUserData, OP = oceanbase::storage::MdsDefaultDeepCopyOperation, oceanbase::st..., (this=|threshold=5.00ms|start at 09:01:31.754|663=0us|total=990.18ms) [2025-08-14 09:01:32.744977] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [511][T1_TimerWK2_SSTableGC][T1][YB42AC4D0007-00063C495D6A02D9-0-0] [lt=12][errcode=0] cost too much time:ob_i_tablet_mds_interface.ipp:int oceanbase::storage::ObITabletMdsInterface::get_latest_committed(OP &&) const [T = oceanbase::storage::ObTabletCreateDeleteMdsUserData, OP = oceanbase::storage::MdsDefaultDeepCopyOperationget_ls_id()={id:1001}) [2025-08-14 09:01:37.724874] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36651] get wrs ts(ls_id={id:1}, delta=549020218, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:37.725050] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=55][errcode=-4389] run this task use too much time(*p_task_={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:0, time_interval:500.00ms, expected_run_time:09:01:37.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:37.725073] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:01:37.725|total=1.92s) [2025-08-14 09:01:37.725447] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=26] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:37.729410] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=164] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:37.714371] WDIAG [RPC] handle_resp (ob_poc_rpc_proxy.cpp:192) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] async_cb handler cost too much time(total_time=1041215, decode_time=-1755162095715539, process_time=0, session_destroy_time=1755162096756754, ret=0, pcode=5395) [2025-08-14 09:01:37.730371] WDIAG stat_cleanup (debug.c:35) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=15973][errcode=0] PNIO [delay_warn] cost too much time: 2015014us, procedure: pn_pktc_resp_cb [2025-08-14 09:01:37.730389] WDIAG eloop_delay_warn (debug.h:93) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] PNIO [delay_warn] eloop handle events delay high: 2015014, malloc=0/0 write=0/0 read=0/1 server_process=0/0 client_cb=2015014/1 [2025-08-14 09:01:37.714436] WDIAG [RPC] handle_resp (ob_poc_rpc_proxy.cpp:192) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=4177][errcode=0] async_cb handler cost too much time(total_time=12315, decode_time=-1755162096741303, process_time=0, session_destroy_time=1755162096753618, ret=0, pcode=5384) [2025-08-14 09:01:37.730949] WDIAG stat_cleanup (debug.c:35) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=16487][errcode=0] PNIO [delay_warn] cost too much time: 989007us, procedure: pn_pktc_resp_cb [2025-08-14 09:01:37.730956] WDIAG eloop_delay_warn (debug.h:93) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] PNIO [delay_warn] eloop handle events delay high: 989007, malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=989007/1 [2025-08-14 09:01:37.733109] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10163] get wrs ts(ls_id={id:1}, delta=347017541, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:37.722598] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=3042018] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:37.736591] WDIAG pkts_handle_req_queue (pkts_post.h:51) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] PNIO [delay_warn] delay high: 973007 [2025-08-14 09:01:37.736616] WDIAG pkts_wq_flush (packet_server.c:52) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] PNIO rpc resp is expired, expire_us=1755162096756050, sock_id=65535 [2025-08-14 09:01:37.736631] WDIAG pkts_handle_req_queue (pkts_post.h:51) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] PNIO [delay_warn] delay high: 966007 [2025-08-14 09:01:37.736683] INFO pktc_resp_cb_on_msg (pktc_resp.h:72) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO resp cb not found: packet_id=559650 [2025-08-14 09:01:37.738161] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:37.738284] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:37.740420] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=19] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:37.740439] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] temporary file module use no blocks [2025-08-14 09:01:37.740446] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:37.740458] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:37.740470] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43622999180116, flush_io_finished_ret_:0, flush_io_finished_round_:26246, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:37.742257] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:37.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:37.742321] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:37.742544] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:37.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:37.743471] WDIAG [RPC] handle_resp (ob_poc_rpc_proxy.cpp:192) [320][pnio1][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] async_cb handler cost too much time(total_time=984115, decode_time=-1755162096759355, process_time=0, session_destroy_time=1755162097743470, ret=0, pcode=1815) [2025-08-14 09:01:37.743485] WDIAG stat_cleanup (debug.c:35) [320][pnio1][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] PNIO [delay_warn] cost too much time: 984007us, procedure: pn_pktc_resp_cb [2025-08-14 09:01:37.743505] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [320][pnio1][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162004737259", warnings:[]}, tenant_id_=1, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:01:37.743528] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [320][pnio1][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162005956015", warnings:[]}, tenant_id_=1, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:01:37.743538] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [320][pnio1][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559534, sock=0x7f97e7e11048, code=-47 [2025-08-14 09:01:37.743548] WDIAG delay_warn (debug.h:103) [320][pnio1][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] PNIO [delay_warn] pktc_flush_cb delay high: 984007, start_us=1755162096758696 [2025-08-14 09:01:37.743557] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [320][pnio1][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162007846478", warnings:[]}, tenant_id_=1, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:01:37.743567] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [320][pnio1][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559542, sock=0x7f97e7e11048, code=-47 [2025-08-14 09:01:37.743573] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [320][pnio1][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162010734094", warnings:[]}, tenant_id_=1, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:01:37.743583] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [320][pnio1][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559558, sock=0x7f97e7e11048, code=-47 [2025-08-14 09:01:37.743590] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [320][pnio1][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559550, sock=0x7f97e7e11048, code=-47 [2025-08-14 09:01:37.743596] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [320][pnio1][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162013826181", warnings:[]}, tenant_id_=1, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:01:37.743605] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [320][pnio1][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559566, sock=0x7f97e7e11048, code=-47 [2025-08-14 09:01:37.743617] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [320][pnio1][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559573, sock=0x7f97e7e11048, code=-47 [2025-08-14 09:01:37.743624] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [320][pnio1][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559597, sock=0x7f97e7e11048, code=-47 [2025-08-14 09:01:37.743631] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [320][pnio1][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559582, sock=0x7f97e7e11048, code=-47 [2025-08-14 09:01:37.743638] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [320][pnio1][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162027383174", warnings:[]}, tenant_id_=1, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:01:37.743646] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [320][pnio1][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559589, sock=0x7f97e7e11048, code=-47 [2025-08-14 09:01:37.743652] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [320][pnio1][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162029064698", warnings:[]}, tenant_id_=1, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:01:37.743661] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [320][pnio1][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559606, sock=0x7f97e7e11048, code=-47 [2025-08-14 09:01:37.743667] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [320][pnio1][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162031145288", warnings:[]}, tenant_id_=1, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:01:37.743675] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [320][pnio1][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559614, sock=0x7f97e7e11048, code=-47 [2025-08-14 09:01:37.743682] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [320][pnio1][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162032739406", warnings:[]}, tenant_id_=1, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:01:37.743691] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [320][pnio1][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559622, sock=0x7f97e7e11048, code=-47 [2025-08-14 09:01:37.743697] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [320][pnio1][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559630, sock=0x7f97e7e11048, code=-47 [2025-08-14 09:01:37.726852] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:233) [827][T1002_RFLWorker][T1002][YB42AC4D0007-00063C495D099C16-0-0] [lt=25] ObRemoteFetchWorker is running(thread_index=0) [2025-08-14 09:01:37.745062] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891968390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:37.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:37.745112] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919ac790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:37.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:37.745125] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891991430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f989196ce10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:37.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:37.746291] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9734] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:37.746876] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:37.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:37.748439] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:37.748473] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:37.748500] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=16] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:01:37.749280] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:37.749305] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:37.751785] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:37.751802] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=16] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3766) [2025-08-14 09:01:37.751809] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:01:37.751821] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=6] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:37.751827] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3767) [2025-08-14 09:01:37.751832] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:01:37.751993] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:37.752003] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3768) [2025-08-14 09:01:37.752008] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:01:37.752603] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:37.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:37.755125] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919aac90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:37.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:37.755160] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919b9510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:37.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:37.755173] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f98919ad090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:37.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:37.755185] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e62b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f9891972910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:37.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:37.755213] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73940b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:414, func_shared_ptr_.ptr:0x7f994b7ee690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:37.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:37.755518] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:37.755544] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:37.755567] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:37.755583] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=9] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:37.755856] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:37.758207] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:01:37.758228] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:01:37.758256] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:01:37.758265] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:00:41.841, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=08:59:41.841], idx=471, next_allow_print_ts="09:01:11.841") [2025-08-14 09:01:37.758280] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:01:37.758289] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:01:37.758311] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:01:37.758321] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:01:37.758333] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:01:37.758343] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:01:37.758352] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:00:52.998, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:22.998], idx=771, next_allow_print_ts="09:01:22.998") [2025-08-14 09:01:37.758367] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=08:59:57.942, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:27.942], idx=868, next_allow_print_ts="09:00:27.942") [2025-08-14 09:01:37.758553] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:01:37.758565] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:01:37.758575] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:01:37.721319] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [834][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99319fe030, caller:ob_table_lock_service.cpp:start:276, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE6$_1273, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe310, time_wheel:0x7f99535f3990, thread_pool:0x7f99535f3170, is_running:0, time_interval:10.00s, expected_run_time:09:01:46.426, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:37.759020] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [834][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=37642][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:01:37.759|total=1.22s) [2025-08-14 09:01:37.759017] WDIAG [RPC] handle_resp (ob_poc_rpc_proxy.cpp:192) [320][pnio1][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] async_cb handler cost too much time(total_time=15312, decode_time=-1755162097743703, process_time=0, session_destroy_time=1755162097759015, ret=0, pcode=5387) [2025-08-14 09:01:37.759051] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [320][pnio1][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559639, sock=0x7f97e7e11048, code=-47 [2025-08-14 09:01:37.733266] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:288) [570][T1_ArcFetcher][T1][YB42AC4D0007-00063C495B999C16-0-0] [lt=4375] ObArchiveFetcher is running(thread_index=0) [2025-08-14 09:01:37.744903] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=6] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:37.760380] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=15450][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162097744899], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:37.760397] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162097744899], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:37.760414] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162097744899]}) [2025-08-14 09:01:37.760426] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:01:37.760435] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:01:37.760447] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=5] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:37.760452] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162097760445], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:37.760461] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162097760445]}) [2025-08-14 09:01:37.760639] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:01:37.760647] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:01:37.760659] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162097760635], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:37.760668] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162097760635]}) [2025-08-14 09:01:37.760676] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7] gts statistics(tenant_id=1002, gts_rpc_cnt=0, get_gts_cache_cnt=0, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:01:37.756157] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:37.752184] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:37.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:37.733263] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:953) [740][T1001_Occam][T1001][YB42AC4D0007-00063C4966399E1E-0-0] [lt=987094][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2025-08-14 09:01:37.762214] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [740][T1001_Occam][T1001][YB42AC4D0007-00063C4966399E1E-0-0] [lt=28901] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:37.762223] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [740][T1001_Occam][T1001][YB42AC4D0007-00063C4966399E1E-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:01:37.762501] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [740][T1001_Occam][T1001][YB42AC4D0007-00063C4966399E1E-0-0] [lt=276][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:01:37.762802] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=842547] [sql nio session](*s={this:0x7f98f72d0030, session_id:3221589415, trace_id:YB42AC4D0007-00063C4965FA43DC-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:245}, err:0, last_decode_time:1755161751714366, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1662, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:01:37.762827] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=25] [sql nio session](*s={this:0x7f98f72d5030, session_id:3221590200, trace_id:YB42AC4D0007-00063C4965FA43E0-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:248}, err:0, last_decode_time:1755161776777167, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1593, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:01:37.762869] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 444993536 used= 389502752 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 70 washed_blocks= 287 washed_size= 70221824 [MEMORY] request_cached_chunk_cnt= 4 [MEMORY] hold= 41963520 used= 41943040 count= 1 avg_used= 41943040 block_cnt= 1 chunk_cnt= 1 mod=LogGroupBuffer [MEMORY] hold= 41963520 used= 41943072 count= 1 avg_used= 41943072 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 29130752 used= 29116416 count= 14 avg_used= 2079744 block_cnt= 14 chunk_cnt= 14 mod=CACHE_MAP_NODE [MEMORY] hold= 25640960 used= 25394048 count= 37 avg_used= 686325 block_cnt= 37 chunk_cnt= 27 mod=MysqlRequesReco [MEMORY] hold= 20615168 used= 20545536 count= 4 avg_used= 5136384 block_cnt= 4 chunk_cnt= 4 mod=ServerObjecPool [MEMORY] hold= 16908288 used= 16777728 count= 16 avg_used= 1048608 block_cnt= 16 chunk_cnt= 16 mod=DiagnosticInfos [MEMORY] hold= 16797696 used= 16777336 count= 1 avg_used= 16777336 block_cnt= 1 chunk_cnt= 1 mod=MrgMemPoolChk [MEMORY] hold= 16083712 used= 15926225 count= 112 avg_used= 142198 block_cnt= 38 chunk_cnt= 9 mod=OmtTenant [MEMORY] hold= 10510336 used= 10491392 count= 1 avg_used= 10491392 block_cnt= 1 chunk_cnt= 1 mod=ASHListBuffer [MEMORY] hold= 9457664 used= 9437784 count= 1 avg_used= 9437784 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 8777728 used= 8760064 count= 2 avg_used= 4380032 block_cnt= 2 chunk_cnt= 2 mod=FreeTbltStream [MEMORY] hold= 8540160 used= 8519680 count= 1 avg_used= 8519680 block_cnt= 1 chunk_cnt= 1 mod=ArcFetchQueue [MEMORY] hold= 8540160 used= 8519680 count= 1 avg_used= 8519680 block_cnt= 1 chunk_cnt= 1 mod=RCSrv [MEMORY] hold= 6856704 used= 6788160 count= 9 avg_used= 754240 block_cnt= 9 chunk_cnt= 9 mod=ResourceGroup [MEMORY] hold= 6365184 used= 5636096 count= 89 avg_used= 63326 block_cnt= 89 chunk_cnt= 30 mod=[T]char [MEMORY] hold= 5505024 used= 4822272 count= 168 avg_used= 28704 block_cnt= 168 chunk_cnt= 31 mod=DiagnosticInfo [MEMORY] hold= 5386240 used= 5368320 count= 1 avg_used= 5368320 block_cnt= 1 chunk_cnt= 1 mod=RespTimeColl [MEMORY] hold= 4943872 used= 4915456 count= 2 avg_used= 2457728 block_cnt= 2 chunk_cnt= 2 mod=HashBuckDmId [MEMORY] hold= 4739072 used= 4718712 count= 1 avg_used= 4718712 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmChe [MEMORY] hold= 4472832 used= 2529072 count= 546 avg_used= 4632 block_cnt= 132 chunk_cnt= 55 mod=WorkerMap [MEMORY] hold= 4235264 used= 4194304 count= 2 avg_used= 2097152 block_cnt= 2 chunk_cnt= 2 mod=SHARE_BLOCK [MEMORY] hold= 3865600 used= 3705600 count= 800 avg_used= 4632 block_cnt= 800 chunk_cnt= 13 mod=CkptDgnMem [MEMORY] hold= 3865600 used= 3705600 count= 800 avg_used= 4632 block_cnt= 800 chunk_cnt= 13 mod=CkptDgnMemCU [MEMORY] hold= 3772416 used= 3751944 count= 1 avg_used= 3751944 block_cnt= 1 chunk_cnt= 1 mod=IteratorStorage [MEMORY] hold= 3723264 used= 3698704 count= 2 avg_used= 1849352 block_cnt= 2 chunk_cnt= 2 mod=LogDIOAligned [MEMORY] hold= 3162112 used= 3145808 count= 2 avg_used= 1572904 block_cnt= 2 chunk_cnt= 2 mod=TenantDLMgr [MEMORY] hold= 3145728 used= 2228224 count= 128 avg_used= 17408 block_cnt= 128 chunk_cnt= 6 mod=SqlDtlQueue [MEMORY] hold= 3047424 used= 3013468 count= 10 avg_used= 301346 block_cnt= 10 chunk_cnt= 2 mod=LDIOSetup [MEMORY] hold= 3031040 used= 3013952 count= 1 avg_used= 3013952 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2850816 used= 2832384 count= 12 avg_used= 236032 block_cnt= 12 chunk_cnt= 10 mod=DiagnosticSum [MEMORY] hold= 2379776 used= 2359608 count= 1 avg_used= 2359608 block_cnt= 1 chunk_cnt= 1 mod=MediumTabletMap [MEMORY] hold= 2379776 used= 2359608 count= 1 avg_used= 2359608 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMonMap [MEMORY] hold= 2379776 used= 2359608 count= 1 avg_used= 2359608 block_cnt= 1 chunk_cnt= 1 mod=FastFrezCkr [MEMORY] hold= 2375680 used= 2359536 count= 2 avg_used= 1179768 block_cnt= 2 chunk_cnt= 2 mod=HashBuckLCSta [MEMORY] hold= 2248704 used= 2228224 count= 1 avg_used= 2228224 block_cnt= 1 chunk_cnt= 1 mod=LogIOCb [MEMORY] hold= 2198720 used= 416700 count= 25277 avg_used= 16 block_cnt= 272 chunk_cnt= 3 mod=Number [MEMORY] hold= 2197056 used= 2174312 count= 77 avg_used= 28237 block_cnt= 77 chunk_cnt= 2 mod=TabletMap [MEMORY] hold= 2064384 used= 1835008 count= 28 avg_used= 65536 block_cnt= 28 chunk_cnt= 11 mod=MajorMergeMgr [MEMORY] hold= 1843200 used= 1770696 count= 9 avg_used= 196744 block_cnt= 9 chunk_cnt= 5 mod=DagMap [MEMORY] hold= 1309440 used= 1305600 count= 20 avg_used= 65280 block_cnt= 20 chunk_cnt= 10 mod=ResultSet [MEMORY] hold= 1294336 used= 1280384 count= 2 avg_used= 640192 block_cnt= 2 chunk_cnt= 1 mod=XATimeWheel [MEMORY] hold= 1294336 used= 1280384 count= 2 avg_used= 640192 block_cnt= 2 chunk_cnt= 2 mod=TransTimeWheel [MEMORY] hold= 1187840 used= 1179768 count= 1 avg_used= 1179768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPlanCac [MEMORY] hold= 1187840 used= 1179768 count= 1 avg_used= 1179768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsInfo [MEMORY] hold= 1187840 used= 1179768 count= 1 avg_used= 1179768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsCache [MEMORY] hold= 1187840 used= 1179768 count= 1 avg_used= 1179768 block_cnt= 1 chunk_cnt= 1 mod=RewriteRuleMap [MEMORY] hold= 1114112 used= 1079296 count= 131 avg_used= 8238 block_cnt= 80 chunk_cnt= 44 mod=ParamCharsType [MEMORY] hold= 1064960 used= 1061408 count= 5 avg_used= 212281 block_cnt= 5 chunk_cnt= 5 mod=IoControl [MEMORY] hold= 1015808 used= 983040 count= 4 avg_used= 245760 block_cnt= 4 chunk_cnt= 3 mod=SqlExecutor [MEMORY] hold= 958464 used= 950272 count= 1 avg_used= 950272 block_cnt= 1 chunk_cnt= 1 mod=IOWorkerLQ [MEMORY] hold= 933888 used= 931072 count= 1 avg_used= 931072 block_cnt= 1 chunk_cnt= 1 mod=ArcSenderQueue [MEMORY] hold= 933888 used= 931072 count= 1 avg_used= 931072 block_cnt= 1 chunk_cnt= 1 mod=TransService [MEMORY] hold= 884736 used= 856896 count= 18 avg_used= 47605 block_cnt= 12 chunk_cnt= 9 mod=CommSysVarFac [MEMORY] hold= 829632 used= 821248 count= 2 avg_used= 410624 block_cnt= 2 chunk_cnt= 2 mod=LSMap [MEMORY] hold= 802816 used= 800000 count= 1 avg_used= 800000 block_cnt= 1 chunk_cnt= 1 mod=SqlFltSpanRec [MEMORY] hold= 798112 used= 789624 count= 3 avg_used= 263208 block_cnt= 2 chunk_cnt= 2 mod=PartitTableTask [MEMORY] hold= 786688 used= 524352 count= 33 avg_used= 15889 block_cnt= 33 chunk_cnt= 4 mod=LogAlloc [MEMORY] hold= 786432 used= 386304 count= 48 avg_used= 8048 block_cnt= 48 chunk_cnt= 20 mod=[T]ObPerfEventR [MEMORY] hold= 679872 used= 658688 count= 83 avg_used= 7936 block_cnt= 10 chunk_cnt= 7 mod=TltMemtablMgr [MEMORY] hold= 671744 used= 663552 count= 1 avg_used= 663552 block_cnt= 1 chunk_cnt= 1 mod=ApplySrv [MEMORY] hold= 666880 used= 404416 count= 89 avg_used= 4544 block_cnt= 66 chunk_cnt= 32 mod=[T]ObTraceEvent [MEMORY] hold= 663552 used= 655360 count= 1 avg_used= 655360 block_cnt= 1 chunk_cnt= 1 mod=FetchLog [MEMORY] hold= 663552 used= 659200 count= 1 avg_used= 659200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 655360 used= 626816 count= 4 avg_used= 156704 block_cnt= 4 chunk_cnt= 4 mod=SQLSessionInfo [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=MultiVersionGC [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=MdsT [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=OBJLockGC [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=CoordTF [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=ElectTimer [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=DupTbLease [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=CoordTR [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=FrzTrigger [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=DetectorTimer [MEMORY] hold= 598016 used= 590232 count= 1 avg_used= 590232 block_cnt= 1 chunk_cnt= 1 mod=DagNetIdMap [MEMORY] hold= 516096 used= 507904 count= 1 avg_used= 507904 block_cnt= 1 chunk_cnt= 1 mod=CommitVersions [MEMORY] hold= 483328 used= 477216 count= 1 avg_used= 477216 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMon [MEMORY] hold= 475136 used= 458816 count= 2 avg_used= 229408 block_cnt= 2 chunk_cnt= 2 mod=ts_token [MEMORY] hold= 442368 used= 393216 count= 6 avg_used= 65536 block_cnt= 6 chunk_cnt= 6 mod=MinorMergeMgr [MEMORY] hold= 409600 used= 401408 count= 1 avg_used= 401408 block_cnt= 1 chunk_cnt= 1 mod=ReplaySrv [MEMORY] hold= 401408 used= 393504 count= 1 avg_used= 393504 block_cnt= 1 chunk_cnt= 1 mod=ClogGe [MEMORY] hold= 385024 used= 375520 count= 2 avg_used= 187760 block_cnt= 2 chunk_cnt= 2 mod=bf_queue [MEMORY] hold= 376832 used= 223376 count= 23 avg_used= 9712 block_cnt= 23 chunk_cnt= 17 mod=[T]ObDSActionAr [MEMORY] hold= 311104 used= 303904 count= 6 avg_used= 50650 block_cnt= 6 chunk_cnt= 3 mod=PoolFreeList [MEMORY] hold= 303104 used= 294936 count= 1 avg_used= 294936 block_cnt= 1 chunk_cnt= 1 mod=RSCompactMap [MEMORY] hold= 303104 used= 294936 count= 1 avg_used= 294936 block_cnt= 1 chunk_cnt= 1 mod=DmlStatHashMap [MEMORY] hold= 303104 used= 294936 count= 1 avg_used= 294936 block_cnt= 1 chunk_cnt= 1 mod=ColUsagHashMap [MEMORY] hold= 303104 used= 294936 count= 1 avg_used= 294936 block_cnt= 1 chunk_cnt= 1 mod=RSCompStMap [MEMORY] hold= 286720 used= 280192 count= 1 avg_used= 280192 block_cnt= 1 chunk_cnt= 1 mod=MetaMemMgr [MEMORY] hold= 278528 used= 270468 count= 2 avg_used= 135234 block_cnt= 2 chunk_cnt= 2 mod=CsSstableReader [MEMORY] hold= 247120 used= 3318 count= 3061 avg_used= 1 block_cnt= 31 chunk_cnt= 1 mod=CharsetUtil [MEMORY] hold= 212992 used= 211072 count= 1 avg_used= 211072 block_cnt= 1 chunk_cnt= 1 mod=TimerWK [MEMORY] hold= 204800 used= 196616 count= 1 avg_used= 196616 block_cnt= 1 chunk_cnt= 1 mod=TenDLBucket [MEMORY] hold= 204800 used= 196616 count= 1 avg_used= 196616 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 204800 used= 196616 count= 1 avg_used= 196616 block_cnt= 1 chunk_cnt= 1 mod=T3MBucket [MEMORY] hold= 204800 used= 196744 count= 1 avg_used= 196744 block_cnt= 1 chunk_cnt= 1 mod=DagNetMap [MEMORY] hold= 155648 used= 150784 count= 19 avg_used= 7936 block_cnt= 19 chunk_cnt= 15 mod=LinearHashMapNo [MEMORY] hold= 147456 used= 139264 count= 1 avg_used= 139264 block_cnt= 1 chunk_cnt= 1 mod=RFLTaskQueue [MEMORY] hold= 139264 used= 134912 count= 17 avg_used= 7936 block_cnt= 10 chunk_cnt= 7 mod=SqlSession [MEMORY] hold= 131072 used= 130560 count= 2 avg_used= 65280 block_cnt= 2 chunk_cnt= 2 mod=MemTblObj [MEMORY] hold= 122880 used= 110608 count= 2 avg_used= 55304 block_cnt= 2 chunk_cnt= 2 mod=TabletIDSetBkt [MEMORY] hold= 114688 used= 106496 count= 1 avg_used= 106496 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlMgr [MEMORY] hold= 114688 used= 111096 count= 1 avg_used= 111096 block_cnt= 1 chunk_cnt= 1 mod=DbmsSched_Job [MEMORY] hold= 114688 used= 111096 count= 1 avg_used= 111096 block_cnt= 1 chunk_cnt= 1 mod=AsyncFreezeTabl [MEMORY] hold= 114688 used= 111104 count= 14 avg_used= 7936 block_cnt= 13 chunk_cnt= 10 mod=QueryAllocator [MEMORY] hold= 102512 used= 27016 count= 370 avg_used= 73 block_cnt= 74 chunk_cnt= 31 mod=Coro [MEMORY] hold= 98304 used= 91968 count= 1 avg_used= 91968 block_cnt= 1 chunk_cnt= 1 mod=tenant_mf_mgr [MEMORY] hold= 98304 used= 95520 count= 1 avg_used= 95520 block_cnt= 1 chunk_cnt= 1 mod=LSSvr [MEMORY] hold= 88000 used= 85888 count= 11 avg_used= 7808 block_cnt= 11 chunk_cnt= 1 mod=TabletIDSetNode [MEMORY] hold= 81920 used= 73896 count= 1 avg_used= 73896 block_cnt= 1 chunk_cnt= 1 mod=RSCompPairCache [MEMORY] hold= 81824 used= 73472 count= 2 avg_used= 36736 block_cnt= 2 chunk_cnt= 2 mod=ts_cancel_set [MEMORY] hold= 81824 used= 73472 count= 2 avg_used= 36736 block_cnt= 2 chunk_cnt= 2 mod=ts_queue [MEMORY] hold= 81824 used= 73472 count= 2 avg_used= 36736 block_cnt= 2 chunk_cnt= 2 mod=ts_run_set [MEMORY] hold= 81536 used= 78080 count= 10 avg_used= 7808 block_cnt= 10 chunk_cnt= 8 mod=HashNodeLCSta [MEMORY] hold= 73728 used= 72736 count= 1 avg_used= 72736 block_cnt= 1 chunk_cnt= 1 mod=LogSharedQueueT [MEMORY] hold= 73728 used= 72736 count= 1 avg_used= 72736 block_cnt= 1 chunk_cnt= 1 mod=LogSubmitLogQue [MEMORY] hold= 73728 used= 66176 count= 1 avg_used= 66176 block_cnt= 1 chunk_cnt= 1 mod=Rpc [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=FlushMeta [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=FlushLog [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=LockMemObj [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=DEVICE_MANAGER [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=TxDataMemObj [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=TxCtxMemObj [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=StorMetaCacheIO [MEMORY] hold= 57344 used= 51328 count= 1 avg_used= 51328 block_cnt= 1 chunk_cnt= 1 mod=palf_env [MEMORY] hold= 56448 used= 48384 count= 14 avg_used= 3456 block_cnt= 11 chunk_cnt= 9 mod=CACHE_INST [MEMORY] hold= 52672 used= 34944 count= 52 avg_used= 672 block_cnt= 36 chunk_cnt= 17 mod=[T]ObWarningBuf [MEMORY] hold= 49152 used= 47616 count= 2 avg_used= 23808 block_cnt= 2 chunk_cnt= 1 mod=IOUsageInfo [MEMORY] hold= 49152 used= 47616 count= 6 avg_used= 7936 block_cnt= 6 chunk_cnt= 5 mod=SqlSessiVarMap [MEMORY] hold= 49152 used= 45184 count= 1 avg_used= 45184 block_cnt= 1 chunk_cnt= 1 mod=IO_MGR [MEMORY] hold= 49152 used= 47616 count= 6 avg_used= 7936 block_cnt= 6 chunk_cnt= 5 mod=PlanVaIdx [MEMORY] hold= 48960 used= 44840 count= 2 avg_used= 22420 block_cnt= 2 chunk_cnt= 2 mod=DynamicFactor [MEMORY] hold= 48112 used= 5016 count= 209 avg_used= 24 block_cnt= 74 chunk_cnt= 29 mod=[T]MemoryContex [MEMORY] hold= 46400 used= 32160 count= 67 avg_used= 480 block_cnt= 25 chunk_cnt= 8 mod=OMT_Worker [MEMORY] hold= 42096 used= 28792 count= 59 avg_used= 488 block_cnt= 30 chunk_cnt= 12 mod=[T]ObSQLMockedT [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=TLD_TblCtxMap [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=Autoincrement [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoBkt [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletPtrMap [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=DagWarnHisBkt [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CliTaskMap [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CTBriefMap [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=HTblRkMapBkt [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=HashBucCliInFo [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=TableAQueryBkt [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=HB_SERVICE [MEMORY] hold= 40896 used= 39680 count= 5 avg_used= 7936 block_cnt= 5 chunk_cnt= 5 mod=MdsTableMgr [MEMORY] hold= 40768 used= 39040 count= 5 avg_used= 7808 block_cnt= 5 chunk_cnt= 5 mod=HashNodePlanCac [MEMORY] hold= 40640 used= 39680 count= 5 avg_used= 7936 block_cnt= 5 chunk_cnt= 1 mod=ObDMMDL [MEMORY] hold= 36096 used= 33216 count= 3 avg_used= 11072 block_cnt= 3 chunk_cnt= 3 mod=DI_CONTAINER [MEMORY] hold= 32768 used= 15872 count= 4 avg_used= 3968 block_cnt= 4 chunk_cnt= 4 mod=CkpDgn [MEMORY] hold= 32768 used= 27960 count= 3 avg_used= 9320 block_cnt= 3 chunk_cnt= 3 mod=PlGdbHelper [MEMORY] hold= 32768 used= 31744 count= 4 avg_used= 7936 block_cnt= 4 chunk_cnt= 4 mod=LSIter [MEMORY] hold= 32768 used= 24688 count= 2 avg_used= 12344 block_cnt= 2 chunk_cnt= 2 mod=TLD_AssignedMgr [MEMORY] hold= 32704 used= 25191 count= 2 avg_used= 12595 block_cnt= 2 chunk_cnt= 2 mod=CGReadInfoMgr [MEMORY] hold= 24576 used= 19776 count= 1 avg_used= 19776 block_cnt= 1 chunk_cnt= 1 mod=LogApplyStatus [MEMORY] hold= 24576 used= 16392 count= 1 avg_used= 16392 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 24576 used= 20136 count= 2 avg_used= 10068 block_cnt= 2 chunk_cnt= 2 mod=TabletStats [MEMORY] hold= 17024 used= 8448 count= 3 avg_used= 2816 block_cnt= 2 chunk_cnt= 2 mod=BaseLogWriter [MEMORY] hold= 16384 used= 15584 count= 2 avg_used= 7792 block_cnt= 2 chunk_cnt= 2 mod=CkptDgnCUNode [MEMORY] hold= 16384 used= 14016 count= 1 avg_used= 14016 block_cnt= 1 chunk_cnt= 1 mod=LogReplayStatus [MEMORY] hold= 16384 used= 8200 count= 1 avg_used= 8200 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=MaBlkLoadMap [MEMORY] hold= 16384 used= 8448 count= 1 avg_used= 8448 block_cnt= 1 chunk_cnt= 1 mod=PalfEnv [MEMORY] hold= 16384 used= 9792 count= 1 avg_used= 9792 block_cnt= 1 chunk_cnt= 1 mod=TSQLSessionMgr [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=MedCheckSet [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=backupTaskSched [MEMORY] hold= 16384 used= 10248 count= 1 avg_used= 10248 block_cnt= 1 chunk_cnt= 1 mod=TabletSvrBucket [MEMORY] hold= 16384 used= 13896 count= 1 avg_used= 13896 block_cnt= 1 chunk_cnt= 1 mod=task_cache [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletMap [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=TLD_ResourceMgr [MEMORY] hold= 16256 used= 15872 count= 2 avg_used= 7936 block_cnt= 2 chunk_cnt= 2 mod=DasRefrLoca [MEMORY] hold= 12864 used= 12288 count= 3 avg_used= 4096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapCn [MEMORY] hold= 12864 used= 12288 count= 3 avg_used= 4096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapDi [MEMORY] hold= 11008 used= 10624 count= 2 avg_used= 5312 block_cnt= 1 chunk_cnt= 1 mod=Scheduler [MEMORY] hold= 9456 used= 9064 count= 2 avg_used= 4532 block_cnt= 2 chunk_cnt= 2 mod=GROUP_INDEX_MAP [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=RSCompPairs [MEMORY] hold= 8192 used= 7792 count= 1 avg_used= 7792 block_cnt= 1 chunk_cnt= 1 mod=CkptDgnMemNode [MEMORY] hold= 8192 used= 7792 count= 1 avg_used= 7792 block_cnt= 1 chunk_cnt= 1 mod=ColUsagNode [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=FillCache [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=GetMediumList [MEMORY] hold= 8192 used= 7728 count= 1 avg_used= 7728 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsCache [MEMORY] hold= 8192 used= 7712 count= 1 avg_used= 7712 block_cnt= 1 chunk_cnt= 1 mod=DmlStatNode [MEMORY] hold= 8192 used= 7808 count= 1 avg_used= 7808 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsInfo [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=GetKeptShotInfo [MEMORY] hold= 8192 used= 7352 count= 1 avg_used= 7352 block_cnt= 1 chunk_cnt= 1 mod=mrg_prog_cker [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=HandleSubmit [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=ls_id [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=LsElectRefInfo [MEMORY] hold= 8192 used= 7632 count= 1 avg_used= 7632 block_cnt= 1 chunk_cnt= 1 mod=CheckMapNode [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=MediumList [MEMORY] hold= 8192 used= 7632 count= 1 avg_used= 7632 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoNode [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=REPLAY_STATUS [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=leakChecker [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=APPLY_STATUS [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=CPUUSage [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=LCLSender [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=MdsDebugMap [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=HTableLockMap [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=DASIDCache [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=ShareBlksMap [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=MatchLsnMap [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=LSInfo [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=PurgeThrottle [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=LockWaitMgr [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileMap [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileBlkMgr [MEMORY] hold= 6880 used= 5600 count= 7 avg_used= 800 block_cnt= 7 chunk_cnt= 6 mod=DICollector [MEMORY] hold= 5216 used= 1976 count= 13 avg_used= 152 block_cnt= 2 chunk_cnt= 2 mod=ObTxLSLogCb [MEMORY] hold= 5056 used= 4656 count= 2 avg_used= 2328 block_cnt= 2 chunk_cnt= 2 mod=MViewService [MEMORY] hold= 4832 used= 4632 count= 1 avg_used= 4632 block_cnt= 1 chunk_cnt= 1 mod=PxPoolBkt [MEMORY] hold= 4832 used= 4632 count= 1 avg_used= 4632 block_cnt= 1 chunk_cnt= 1 mod=ResGrpIdMap [MEMORY] hold= 4832 used= 4632 count= 1 avg_used= 4632 block_cnt= 1 chunk_cnt= 1 mod=ExTabletCntMap [MEMORY] hold= 4832 used= 4632 count= 1 avg_used= 4632 block_cnt= 1 chunk_cnt= 1 mod=ResRuleIdMap [MEMORY] hold= 4608 used= 2544 count= 6 avg_used= 424 block_cnt= 6 chunk_cnt= 5 mod=SequenceIdMap [MEMORY] hold= 4288 used= 4096 count= 1 avg_used= 4096 block_cnt= 1 chunk_cnt= 1 mod=MacroFile [MEMORY] hold= 4224 used= 3072 count= 6 avg_used= 512 block_cnt= 5 chunk_cnt= 4 mod=SqlSessiQuerSql [MEMORY] hold= 4160 used= 3520 count= 4 avg_used= 880 block_cnt= 3 chunk_cnt= 3 mod=PoolArenaArray [MEMORY] hold= 4096 used= 2328 count= 1 avg_used= 2328 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoMap [MEMORY] hold= 3840 used= 2544 count= 6 avg_used= 424 block_cnt= 6 chunk_cnt= 5 mod=PackStateMap [MEMORY] hold= 3840 used= 2544 count= 6 avg_used= 424 block_cnt= 6 chunk_cnt= 5 mod=SequenceMap [MEMORY] hold= 3840 used= 2544 count= 6 avg_used= 424 block_cnt= 6 chunk_cnt= 5 mod=ContextsMap [MEMORY] hold= 3328 used= 1728 count= 4 avg_used= 432 block_cnt= 4 chunk_cnt= 4 mod=NegParamIdex [MEMORY] hold= 3328 used= 1792 count= 8 avg_used= 224 block_cnt= 1 chunk_cnt= 1 mod=LogIOTask [MEMORY] hold= 3312 used= 3112 count= 1 avg_used= 3112 block_cnt= 1 chunk_cnt= 1 mod=TFileFLCtx [MEMORY] hold= 3312 used= 3112 count= 1 avg_used= 3112 block_cnt= 1 chunk_cnt= 1 mod=PendingFreeMap [MEMORY] hold= 3200 used= 3008 count= 1 avg_used= 3008 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlDfc [MEMORY] hold= 2528 used= 2328 count= 1 avg_used= 2328 block_cnt= 1 chunk_cnt= 1 mod=MedCheckMap [MEMORY] hold= 2528 used= 2328 count= 1 avg_used= 2328 block_cnt= 1 chunk_cnt= 1 mod=IRMMemHashBuck [MEMORY] hold= 2528 used= 2328 count= 1 avg_used= 2328 block_cnt= 1 chunk_cnt= 1 mod=VecIdxLSMgr [MEMORY] hold= 2528 used= 2328 count= 1 avg_used= 2328 block_cnt= 1 chunk_cnt= 1 mod=DASMemHashBuck [MEMORY] hold= 2240 used= 2048 count= 1 avg_used= 2048 block_cnt= 1 chunk_cnt= 1 mod=CreateContext [MEMORY] hold= 2048 used= 1856 count= 1 avg_used= 1856 block_cnt= 1 chunk_cnt= 1 mod=LogIOWS [MEMORY] hold= 2048 used= 1272 count= 1 avg_used= 1272 block_cnt= 1 chunk_cnt= 1 mod=LSCheckMap [MEMORY] hold= 2016 used= 1824 count= 1 avg_used= 1824 block_cnt= 1 chunk_cnt= 1 mod=PxResMgr [MEMORY] hold= 1920 used= 1728 count= 1 avg_used= 1728 block_cnt= 1 chunk_cnt= 1 mod=timer_service [MEMORY] hold= 1808 used= 1424 count= 2 avg_used= 712 block_cnt= 2 chunk_cnt= 2 mod=DetectManager [MEMORY] hold= 1744 used= 1544 count= 1 avg_used= 1544 block_cnt= 1 chunk_cnt= 1 mod=TXBFRetryId [MEMORY] hold= 1744 used= 1544 count= 1 avg_used= 1544 block_cnt= 1 chunk_cnt= 1 mod=TabStatMgrLock [MEMORY] hold= 1472 used= 1272 count= 1 avg_used= 1272 block_cnt= 1 chunk_cnt= 1 mod=HashBucApiGroup [MEMORY] hold= 1472 used= 1272 count= 1 avg_used= 1272 block_cnt= 1 chunk_cnt= 1 mod=DUP_LS_SET [MEMORY] hold= 1472 used= 1272 count= 1 avg_used= 1272 block_cnt= 1 chunk_cnt= 1 mod=GCMemtableMap [MEMORY] hold= 1280 used= 1088 count= 1 avg_used= 1088 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 1248 used= 1056 count= 1 avg_used= 1056 block_cnt= 1 chunk_cnt= 1 mod=LOG_HASH_MAP [MEMORY] hold= 1232 used= 1032 count= 1 avg_used= 1032 block_cnt= 1 chunk_cnt= 1 mod=BFLoadTask [MEMORY] hold= 1200 used= 1000 count= 1 avg_used= 1000 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletMapLk [MEMORY] hold= 1120 used= 120 count= 5 avg_used= 24 block_cnt= 2 chunk_cnt= 2 mod=FreezeTask [MEMORY] hold= 1024 used= 832 count= 1 avg_used= 832 block_cnt= 1 chunk_cnt= 1 mod=tenant_rm_mgr [MEMORY] hold= 976 used= 776 count= 1 avg_used= 776 block_cnt= 1 chunk_cnt= 1 mod=DepTabletSet [MEMORY] hold= 896 used= 696 count= 1 avg_used= 696 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfo [MEMORY] hold= 752 used= 552 count= 1 avg_used= 552 block_cnt= 1 chunk_cnt= 1 mod=LSFreeze [MEMORY] hold= 704 used= 208 count= 1 avg_used= 208 block_cnt= 1 chunk_cnt= 1 mod=[T]ObMacroBlock [MEMORY] hold= 672 used= 472 count= 1 avg_used= 472 block_cnt= 1 chunk_cnt= 1 mod=Election [MEMORY] hold= 640 used= 224 count= 2 avg_used= 112 block_cnt= 1 chunk_cnt= 1 mod=Coordinator [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=MSTXCTX [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=VectorIndexVsag [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=HAScheduler [MEMORY] hold= 544 used= 144 count= 2 avg_used= 72 block_cnt= 2 chunk_cnt= 2 mod=TntSrvObjPool [MEMORY] hold= 512 used= 120 count= 2 avg_used= 60 block_cnt= 1 chunk_cnt= 1 mod=UserResourceMgr [MEMORY] hold= 448 used= 64 count= 2 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfFSCbNode [MEMORY] hold= 416 used= 2 count= 2 avg_used= 1 block_cnt= 2 chunk_cnt= 2 mod=ObLogEXTHandler [MEMORY] hold= 400 used= 200 count= 1 avg_used= 200 block_cnt= 1 chunk_cnt= 1 mod=ExTabletMapLk [MEMORY] hold= 384 used= 102 count= 1 avg_used= 102 block_cnt= 1 chunk_cnt= 1 mod=TxLSLogBuf [MEMORY] hold= 304 used= 104 count= 1 avg_used= 104 block_cnt= 1 chunk_cnt= 1 mod=RedoLogBuckLock [MEMORY] hold= 256 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 256 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=[T]TSILastOper [MEMORY] hold= 256 used= 48 count= 1 avg_used= 48 block_cnt= 1 chunk_cnt= 1 mod=[T]TSITabletTab [MEMORY] hold= 240 used= 41 count= 1 avg_used= 41 block_cnt= 1 chunk_cnt= 1 mod=KeepAliveBuf [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=RebuildCbNode [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfRCCbNode [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=logservice [MEMORY] hold= 389502752 used= 379256835 count= 32870 avg_used= 11538 mod=SUMMARY [2025-08-14 09:01:37.762911] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=20] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 230686720 used= 228884480 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 block_cnt= 110 chunk_cnt= 110 mod=Memstore [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:01:37.762923] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= EXECUTE_CTX_ID hold= 1163264 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 15 washed_size= 13516800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:37.762936] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 block_cnt= 1 chunk_cnt= 1 mod=PartTranCtxMgr [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:01:37.762953] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 53714944 used= 47487424 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 84 washed_blocks= 334 washed_size= 126640128 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 40274240 used= 36480727 count= 2830 avg_used= 12890 block_cnt= 2830 chunk_cnt= 79 mod=SqlPhyPlan [MEMORY] hold= 3768320 used= 3427728 count= 321 avg_used= 10678 block_cnt= 84 chunk_cnt= 49 mod=SqlPlanCache [MEMORY] hold= 1524288 used= 1400877 count= 158 avg_used= 8866 block_cnt= 158 chunk_cnt= 11 mod=SqlPhyPlObj [MEMORY] hold= 1382080 used= 1263616 count= 617 avg_used= 2048 block_cnt= 223 chunk_cnt= 58 mod=CreateContext [MEMORY] hold= 536448 used= 523776 count= 66 avg_used= 7936 block_cnt= 66 chunk_cnt= 24 mod=OB_SQLSTAT_OBJ [MEMORY] hold= 2048 used= 1664 count= 2 avg_used= 832 block_cnt= 1 chunk_cnt= 1 mod=PlanCache [MEMORY] hold= 47487424 used= 43098388 count= 3994 avg_used= 10790 mod=SUMMARY [2025-08-14 09:01:37.762964] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1 ctx_id= WORK_AREA hold= 1458176 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 13 washed_blocks= 66 washed_size= 25804800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:37.762976] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 14204928 used= 10970224 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 277 washed_size= 31932416 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10953744 used= 6065548 count= 18570 avg_used= 326 block_cnt= 372 chunk_cnt= 12 mod=PlJit [MEMORY] hold= 16480 used= 13376 count= 3 avg_used= 4458 block_cnt= 2 chunk_cnt= 2 mod=PlCodeGen [MEMORY] hold= 10970224 used= 6078924 count= 18573 avg_used= 327 mod=SUMMARY [2025-08-14 09:01:37.762988] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 111149056 used= 108896256 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 block_cnt= 211 chunk_cnt= 53 mod=CoStack [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 mod=SUMMARY [2025-08-14 09:01:37.763015] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 56623104 used= 56623104 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 56623104 used= 0 count= 27 avg_used= 0 block_cnt= 0 chunk_cnt= 0 mod=KvstorCacheMb [MEMORY] hold= 56623104 used= 0 count= 27 avg_used= 0 mod=SUMMARY [2025-08-14 09:01:37.763028] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 34365376 limit= 429496720 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 29491200 used= 26236400 count= 400 avg_used= 65591 block_cnt= 400 chunk_cnt= 15 mod=L_TabletPool [MEMORY] hold= 4784128 used= 4765440 count= 73 avg_used= 65280 block_cnt= 73 chunk_cnt= 3 mod=N_TabletPool [MEMORY] hold= 90048 used= 84768 count= 2 avg_used= 42384 block_cnt= 2 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 34365376 used= 31086608 count= 475 avg_used= 65445 mod=SUMMARY [2025-08-14 09:01:37.763083] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [740][T1001_Occam][T1001][YB42AC4D0007-00063C4966399E1E-0-0] [lt=10] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:37.763093] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [740][T1001_Occam][T1001][YB42AC4D0007-00063C4966399E1E-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:01:37.763102] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 8003584 used= 4947392 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 10 washed_blocks= 23 washed_size= 12967936 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4947392 used= 4793344 count= 604 avg_used= 7936 block_cnt= 66 chunk_cnt= 8 mod=MemtableCallbac [MEMORY] hold= 4947392 used= 4793344 count= 604 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:37.763102] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [740][T1001_Occam][T1001][YB42AC4D0007-00063C4966399E1E-0-0] [lt=9][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:01:37.763117] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=LobPersist [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:37.763130] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 65536 used= 31024 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 2031616 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 31024 used= 27058 count= 20 avg_used= 1352 block_cnt= 6 chunk_cnt= 1 mod=SqlPsCache [MEMORY] hold= 31024 used= 27058 count= 20 avg_used= 1352 mod=SUMMARY [2025-08-14 09:01:37.763171] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= RPC_CTX_ID hold= 6438912 used= 1572864 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1253376 used= 829056 count= 51 avg_used= 16256 block_cnt= 51 chunk_cnt= 1 mod=OB_WRS_CLUSTER_ [MEMORY] hold= 245760 used= 162560 count= 10 avg_used= 16256 block_cnt= 10 chunk_cnt= 3 mod=OB_LOG_REGISTER [MEMORY] hold= 24576 used= 16256 count= 1 avg_used= 16256 block_cnt= 1 chunk_cnt= 1 mod=OB_LOG_GET_STAT [MEMORY] hold= 16384 used= 7936 count= 2 avg_used= 3968 block_cnt= 2 chunk_cnt= 1 mod=[L]OB_WRS_CLUST [MEMORY] hold= 16384 used= 7936 count= 2 avg_used= 3968 block_cnt= 2 chunk_cnt= 2 mod=[L]OB_LOG_REGIS [MEMORY] hold= 16384 used= 7936 count= 2 avg_used= 3968 block_cnt= 1 chunk_cnt= 1 mod=[L]OB_DETECT_MA [MEMORY] hold= 1572864 used= 1031680 count= 68 avg_used= 15171 mod=SUMMARY [2025-08-14 09:01:37.763188] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 3481600 used= 2391232 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 10 washed_size= 4907008 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2121728 used= 2105152 count= 3 avg_used= 701717 block_cnt= 3 chunk_cnt= 3 mod=MEMTABLE_ARENA [MEMORY] hold= 261376 used= 253952 count= 32 avg_used= 7936 block_cnt= 21 chunk_cnt= 2 mod=TX_DATA_SLICE [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=TX_DATA_ARENA [MEMORY] hold= 2391232 used= 2367040 count= 36 avg_used= 65751 mod=SUMMARY [2025-08-14 09:01:37.763208] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 3964928 used= 3080192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 3 washed_size= 229376 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3080192 used= 2983936 count= 376 avg_used= 7936 block_cnt= 24 chunk_cnt= 2 mod=MdsTable [MEMORY] hold= 3080192 used= 2983936 count= 376 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:37.763221] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1 ctx_id= SCHEMA_SERVICE hold= 212992 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1884160 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=NonPartTenCac [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:37.763453] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:37.763468] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= MERGE_NORMAL_CTX_ID hold= 884736 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 6 washed_blocks= 9 washed_size= 11698176 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:37.763485] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant: 500, limit: 9223372036854775807 hold: 705314816 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 328740864 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 48611328 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 85983232 limit= 9223372036854775807 [MEMORY] ctx_id= LIBEASY hold_bytes= 3145728 limit= 9223372036854775807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 20971520 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 2719744 limit= 9223372036854775807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 37232640 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 78925824 limit= 9223372036854775807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 98983936 limit= 9223372036854775807 [2025-08-14 09:01:37.763843] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 328740864 used= 300923120 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 502 washed_blocks= 2234 washed_size= 929439744 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 52641792 used= 50996736 count= 3317 avg_used= 15374 block_cnt= 1499 chunk_cnt= 42 mod=TZInfoArray [MEMORY] hold= 35672064 used= 35651584 count= 1 avg_used= 35651584 block_cnt= 1 chunk_cnt= 1 mod=qdtable [MEMORY] hold= 33292288 used= 33275904 count= 16 avg_used= 2079744 block_cnt= 16 chunk_cnt= 16 mod=ObTxDesc [MEMORY] hold= 25064000 used= 24981120 count= 159 avg_used= 157113 block_cnt= 105 chunk_cnt= 49 mod=TenantCtxAlloca [MEMORY] hold= 12873728 used= 12856400 count= 1 avg_used= 12856400 block_cnt= 1 chunk_cnt= 1 mod=MemDumpContext [MEMORY] hold= 7397376 used= 7340256 count= 7 avg_used= 1048608 block_cnt= 7 chunk_cnt= 7 mod=ConcurHashMap [MEMORY] hold= 7275040 used= 6334546 count= 1107 avg_used= 5722 block_cnt= 571 chunk_cnt= 6 mod=CharsetInit [MEMORY] hold= 5373952 used= 5236736 count= 32 avg_used= 163648 block_cnt= 32 chunk_cnt= 15 mod=Iterator &) const, (this=|threshold=10.00ms|start at 09:01:41.736|total=12.37ms) [2025-08-14 09:01:41.749424] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [385]OB_MALLOC COST TOO MUCH TIME, cost_time=965539, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=965537], seq:[0] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=965574, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=965568], seq:[0, 1] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=979226, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=979178, ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=996428, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=996417], seq:[0] [857]OB_MALLOC COST TOO MUCH TIME, cost_time=908785, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=1005797, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=996441], seq:[0, 1] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=1005480, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1005473], seq:[0] [388]OB_MALLOC COST TOO MUCH TIME, cost_time=1003895, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1003885, ALLOC_CHUNK_END=5], seq:[0, 1] [1031]OB_MALLOC COST TOO MUCH TIME, cost_time=1003681, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1003671, ALLOC_CHUNK_END=5], seq:[0, 1] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=1003691, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1003681, ALLOC_CHUNK_END=6], seq:[0, 1] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=998406, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=998379, ALLOC_CHUNK_END=22], seq:[0, 1] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=1006214, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=979251, WASH_BLOCK_END=410, ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=16831], seq:[0, 1, 2, 3, 4] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=997390, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=997380, ALLOC_CHUNK_END=4], seq:[0, 1] [737]OB_MALLOC COST TOO MUCH TIME, cost_time=995377, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=995365, ALLOC_CHUNK_END=5], seq:[0, 1] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=987593, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=987583, ALLOC_CHUNK_END=5], seq:[0, 1] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=986104, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=986084, ALLOC_CHUNK_END=5], seq:[0, 1] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=984514, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=984503, ALLOC_CHUNK_END=5], seq:[0, 1] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=984209, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=984197, ALLOC_CHUNK_END=5], seq:[0, 1] [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=981946, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=981931, ALLOC_CHUNK_END=5], seq:[0, 1] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=981151, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=981128, ALLOC_CHUNK_END=14], seq:[0, 1] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=971498, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=971486, ALLOC_CHUNK_END=5], seq:[0, 1] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=969780, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=969770, ALLOC_CHUNK_END=4], seq:[0, 1] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=938254, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=938243, ALLOC_CHUNK_END=5], seq:[0, 1] [575]OB_MALLOC COST TOO MUCH TIME, cost_time=924499, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=924489, ALLOC_CHUNK_END=4], seq:[0, 1] [656]OB_MALLOC COST TOO MUCH TIME, cost_time=880029, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=880020, ALLOC_CHUNK_END=5], seq:[0, 1] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=760701, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=760687, ALLOC_CHUNK_END=5], seq:[0, 1] [381]OB_MALLOC COST TOO MUCH TIME, cost_time=522754, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=522741, ALLOC_CHUNK_END=5], seq:[0, 1] [387]OB_MALLOC COST TOO MUCH TIME, cost_time=1007433, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=979040, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=28370, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [2025-08-14 09:01:41.753669] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1007623, owner: ObLog, click_count: 5, time dist:[FORMAT_END=6, ObMalloc#start=0, ObMalloc#end=1005812, ALLOC_END=1804, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [772]OB_MALLOC COST TOO MUCH TIME, cost_time=1012663, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=997557, ALLOC_CHUNK_END=15099], seq:[0, 1] [2025-08-14 09:01:41.749498] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:41.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:41.749552] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:41.752992] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72847b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728a310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:41.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:41.754543] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:41.763103] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:41.763146] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b9e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:41.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:41.763168] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891806f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:41.751, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:41.763180] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739fb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:41.750, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:41.773251] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73539b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7396b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:41.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:41.774288] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:41.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:41.775892] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:41.775913] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=18] temporary file module use no blocks [2025-08-14 09:01:41.775921] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:41.775932] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:41.775945] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623003215589, flush_io_finished_ret_:0, flush_io_finished_round_:26283, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:41.779034] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16] LogLoopThread round_cost_time(us)(round_cost_time=9) [2025-08-14 09:01:41.779057] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] get_fetch_log_dst failed or invalid(ret=0, palf_id=1, self="172.77.0.7:2882", fetch_log_dst="172.77.0.7:2882") [2025-08-14 09:01:41.784209] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:41.803406] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f1510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:41.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:41.804472] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:41.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:41.813488] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d22b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72eaf90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:41.808, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:41.823576] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ec010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:41.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:41.823611] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891972790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:41.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:41.843696] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ab290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:41.831, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:41.843721] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a1c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:41.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:41.849554] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:41.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:41.853765] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196ac10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:41.849, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:41.863825] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a0910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:41.852, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:41.873894] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891972610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:41.861, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:41.879110] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:01:41.879129] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:01:41.904121] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919859b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891972490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:41.896, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:41.949621] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:41.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:42.025802] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:42.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:42.049689] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:41.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:42.148295] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=24] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:01:42.149747] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:42.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:42.249808] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:42.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:42.277395] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:42.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:42.307578] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:42.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:42.349871] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:42.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:42.377414] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=16][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:01:42.449946] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:42.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:42.492215] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fa190, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa470, time_wheel:0x7f99341cdbf0, thread_pool:0x7f99341cd3d0, is_running:1, time_interval:2.00s, expected_run_time:09:01:42.334, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:42.503677] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fe410, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe6f0, time_wheel:0x7f9922df9d40, thread_pool:0x7f9922df9520, is_running:1, time_interval:2.00s, expected_run_time:09:01:42.432, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:42.528958] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:42.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:42.549996] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:42.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:42.650035] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:42.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:41.749531] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:42.717907] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=729567][errcode=0] EASY SLOW: thread: 7f99544fb300, lastrun: 1755162101.749522 cost: 0.111721 loop:26467, slowcnt: 1 [2025-08-14 09:01:42.724602] WDIAG [STORAGE] reset (ob_access_service.cpp:42) [693][T1001_TimerWK3_FreInfoReload][T1001][YB42AC4D0007-00063C495DC9FC50-0-0] [lt=9][errcode=-4389] guard used too much time(guard_used_us=210020734, ls_id={id:1}, lbt()="0x26a1806d 0x1c3e21a6 0x940a506 0x9407229 0x939ba72 0x9399e3d 0x9399084 0x93973b0 0x93942f7 0x9725f91 0x124dfd77 0x96a5a4d 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x1fa4b892 0x1aec706b 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3") [2025-08-14 09:01:42.733205] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:42.733255] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891970090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:42.724, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:42.733548] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=965592][errcode=0] epoll handler too much time :967062 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:42.745648] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:42.747435] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:42.748596] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:42.750114] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:42.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:42.750842] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:42.751542] WDIAG [COORDINATOR] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=0] cost too much time:election_priority_impl.cpp:int oceanbase::logservice::coordinator::SerializeFunctor::operator()(const T &) [T = oceanbase::logservice::coordinator::PriorityV1], (this=|threshold=1.00s|start at 09:01:42.751|37=2us|39=0us|41=1us|total=1.01s) [2025-08-14 09:01:42.751568] WDIAG [COORDINATOR] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] cost too much time:election_priority_impl.cpp:virtual int oceanbase::logservice::coordinator::ElectionPriorityImpl::serialize(char *, const int64_t, int64_t &) const, (this=|threshold=1.00s|start at 09:01:41.727|96=0us|98=1.02s|total=1.02s) [2025-08-14 09:01:42.753381] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7ef590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:42.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:42.753408] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7ef710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:42.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:42.754507] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:41.753669] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:42.762855] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1007623][errcode=0] epoll handler too much time :1009190 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:42.762884] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [772]OB_MALLOC COST TOO MUCH TIME, cost_time=1963815, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1012677], seq:[0, 1] [2025-08-14 09:01:41.749535] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1963870, owner: ObLog, click_count: 5, time dist:[FORMAT_END=12, ObMalloc#start=0, ObMalloc#end=1963851, ALLOC_END=6, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [474]OB_MALLOC COST TOO MUCH TIME, cost_time=1959467, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1005487], seq:[0, 1] [2025-08-14 09:01:41.754007] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1959513, owner: ObLog, click_count: 5, time dist:[FORMAT_END=19, ObMalloc#start=1, ObMalloc#end=1959491, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=1935177, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=981955, WASH_BLOCK_END=953221], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=1713754, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=760708, WASH_BLOCK_END=953046], seq:[0, 1, 2] [768]OB_MALLOC COST TOO MUCH TIME, cost_time=1021904, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=68886, WASH_BLOCK_END=953017], seq:[0, 1, 2] [2025-08-14 09:01:42.000577] EDIAG eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1713803, owner: ObLog, click_count: 5, time dist:[FORMAT_END=13, ObMalloc#start=1, ObMalloc#end=1713762, ALLOC_END=27, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x27a8a8c8 0x97b7e1a 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:42.693470] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1021964, owner: ObLog, click_count: 5, time dist:[FORMAT_END=28, ObMalloc#start=1, ObMalloc#end=1021914, ALLOC_END=21, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [686]OB_MALLOC COST TOO MUCH TIME, cost_time=981398, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=28410, WASH_BLOCK_END=952988], seq:[0, 1, 2] [386]OB_MALLOC COST TOO MUCH TIME, cost_time=978589, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=25626, WASH_BLOCK_END=952963], seq:[0, 1, 2] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=1952587, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=998415, WASH_BLOCK_END=954167, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=3], seq:[0, 1, 2, 3, 4] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=973262, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=20301, WASH_BLOCK_END=952961], seq:[0, 1, 2] [832]OB_MALLOC COST TOO MUCH TIME, cost_time=973281, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=20307, WASH_BLOCK_END=952974], seq:[0, 1, 2] [864]OB_MALLOC COST TOO MUCH TIME, cost_time=966154, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=13195, WASH_BLOCK_END=952959], seq:[0, 1, 2] [1031]OB_MALLOC COST TOO MUCH TIME, cost_time=1972876, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1003688, WASH_BLOCK_END=969187], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=1973006, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1003699, WASH_BLOCK_END=969307], seq:[0, 1, 2] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=1966043, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=997403, WASH_BLOCK_END=968639], seq:[0, 1, 2] [737]OB_MALLOC COST TOO MUCH TIME, cost_time=1964037, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=995386, WASH_BLOCK_END=968651], seq:[0, 1, 2] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=1956368, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=987603, WASH_BLOCK_END=968765], seq:[0, 1, 2] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=1954912, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=986111, WASH_BLOCK_END=968801], seq:[0, 1, 2] [316]OB_MALLOC COST TOO MUCH TIME, cost_time=996585, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=29686], seq:[0, 1] [2025-08-14 09:01:41.761325] EDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] LOGGER COST TOO MUCH TIME, cost: 1968986, owner: ObLog, click_count: 5, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=1952790, ALLOC_END=16183, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b52e837 0x996d04c 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [477]OB_MALLOC COST TOO MUCH TIME, cost_time=1954491, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=984524, WASH_BLOCK_END=969967], seq:[0, 1, 2] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=1954167, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=984216, WASH_BLOCK_END=969951], seq:[0, 1, 2] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=1941383, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=971512, WASH_BLOCK_END=969871], seq:[0, 1, 2] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=1939768, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=969788, WASH_BLOCK_END=969980], seq:[0, 1, 2] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=1951226, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=981171, WASH_BLOCK_END=970055], seq:[0, 1, 2] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=1908258, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=938263, WASH_BLOCK_END=969995], seq:[0, 1, 2] [2025-08-14 09:01:41.822633] EDIAG [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1908325, owner: ObLog, click_count: 5, time dist:[FORMAT_END=22, ObMalloc#start=1, ObMalloc#end=1908268, ALLOC_END=34, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1c03a517 0x97a8d73 0x97a88d2 0x97a7481 0x97a6a43 0x97a42c5 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [575]OB_MALLOC COST TOO MUCH TIME, cost_time=1894591, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=924507, WASH_BLOCK_END=970084], seq:[0, 1, 2] [2025-08-14 09:01:41.836437] EDIAG [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=19][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1894642, owner: ObLog, click_count: 5, time dist:[FORMAT_END=13, ObMalloc#start=1, ObMalloc#end=1894599, ALLOC_END=29, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1bf2ec62 0x986ebd7 0x986bee6 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [656]OB_MALLOC COST TOO MUCH TIME, cost_time=1850182, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=880037, WASH_BLOCK_END=970145], seq:[0, 1, 2] [317]OB_MALLOC COST TOO MUCH TIME, cost_time=1009801, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=39752, WASH_BLOCK_END=970047], seq:[0, 1, 2] [2025-08-14 09:01:41.881007] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1850257, owner: ObLog, click_count: 5, time dist:[FORMAT_END=21, ObMalloc#start=1, ObMalloc#end=1850206, ALLOC_END=28, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [389]OB_MALLOC COST TOO MUCH TIME, cost_time=999703, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=29562, WASH_BLOCK_END=970141], seq:[0, 1, 2] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=999265, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=29079, WASH_BLOCK_END=970185], seq:[0, 1, 2] [381]OB_MALLOC COST TOO MUCH TIME, cost_time=1492829, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=522765, WASH_BLOCK_END=970064], seq:[0, 1, 2] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=998777, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=28505, WASH_BLOCK_END=970271], seq:[0, 1, 2] [387]OB_MALLOC COST TOO MUCH TIME, cost_time=1977751, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1007445, WASH_BLOCK_END=970306], seq:[0, 1, 2] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=994951, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=24670, WASH_BLOCK_END=970281], seq:[0, 1, 2] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=990243, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=20024, WASH_BLOCK_END=970219], seq:[0, 1, 2] [2025-08-14 09:01:41.753530] EDIAG [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1982058, owner: ObLog, click_count: 5, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=1006223, ALLOC_END=975824, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a1f867 0x27a135c3 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:42.763946] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7281410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:42.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:42.773434] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:42.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:42.773934] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=30] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:42.773957] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=16] temporary file module use no blocks [2025-08-14 09:01:42.773963] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:42.773975] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=9] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:42.773998] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=23] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623004213620, flush_io_finished_ret_:0, flush_io_finished_round_:26277, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:42.774111] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ab110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:42.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:42.774130] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891969710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:42.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:42.774141] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d36b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ebb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:42.767, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:42.774151] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891971d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:42.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:42.776039] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=24] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:42.776062] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=20] temporary file module use no blocks [2025-08-14 09:01:42.776070] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:42.776080] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:42.776091] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623004215738, flush_io_finished_ret_:0, flush_io_finished_round_:26284, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:42.783479] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:42.784205] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b4a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:42.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:42.784248] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72739b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919a5e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:42.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:42.784259] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72779b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72eab10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:42.777, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:42.784269] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728bc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:42.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:42.794282] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919902b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196c390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:42.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:42.794306] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ec610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:42.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:42.794317] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891973810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:42.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:42.803601] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:42.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:42.804335] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7277730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ea990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:42.794, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:42.804357] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891825c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:42.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:42.824419] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7283e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:42.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:42.834490] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7288390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:42.820, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:42.844554] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7288210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:42.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:42.850180] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:42.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:42.879615] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=6] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:01:42.879634] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:01:42.880441] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:42.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:42.950211] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:42.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:42.965253] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196aa90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:42.952, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:42.965280] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7288e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:42.953, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:42.980502] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:42.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:42.985411] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891992fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196de90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:42.970, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:43.024862] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:43.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.050273] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:42.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.080537] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:42.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.122209] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [571][T1_ArcSender][T1][YB42AC4D0007-00063C495BA99C16-0-0] [lt=31][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:01:43.150305] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:43.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.180586] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:43.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.250380] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:43.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.276353] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:43.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.280631] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:43.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.306492] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:43.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.350440] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:43.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.380692] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:43.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.450458] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:43.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.480767] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:43.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.527593] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:43.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.550481] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:43.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.580830] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:43.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.650544] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:43.564, task_priority:1, with_handle_protected:1}) [857]OB_MALLOC COST TOO MUCH TIME, cost_time=991387, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=8630, WASH_BLOCK_END=982757], seq:[0, 1, 2] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=998738, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=15941, WASH_BLOCK_END=982793, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=3], seq:[0, 1, 2, 3, 4] [15515]OB_MALLOC COST TOO MUCH TIME, cost_time=985364, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=985355], seq:[0] [699]OB_MALLOC COST TOO MUCH TIME, cost_time=1007262, tenant_id=500, label=HeapVar, ctx_id=0, prio=0, size=24600, owner: ObMalloc, click_count: 10, time dist:[ALLOC_CHUNK_END=983198, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9] [657]OB_MALLOC COST TOO MUCH TIME, cost_time=1950843, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=32896, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1950836], seq:[0] [15515]OB_MALLOC COST TOO MUCH TIME, cost_time=1980541, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=997943], seq:[0, 1] [388]OB_MALLOC COST TOO MUCH TIME, cost_time=1016153, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1016145], seq:[0] [857]OB_MALLOC COST TOO MUCH TIME, cost_time=991164, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [2025-08-14 09:01:43.680870] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:43.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:41.749535] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:41.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.716406] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1963870][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:41.749|207=1.97s|210=0us|222=2us|total=1.97s) [2025-08-14 09:01:41.754007] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:41.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.716522] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1959513][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:41.754|207=1.96s|210=0us|222=1us|total=1.96s) [2025-08-14 09:01:41.778757] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=21] current monitor number(seq_=-1) [2025-08-14 09:01:42.000577] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=28] PNIO [ratelimit] time: 1755162102000575, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007830, add_bytes: 0 [2025-08-14 09:01:42.693470] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571634a0, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957163780, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:5.00s, expected_run_time:09:01:42.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.716999] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=1713803] PNIO [ratelimit] time: 1755162103716997, bytes: 0, bw: 0.000000 MB/s, add_ts: 1716422, add_bytes: 0 [2025-08-14 09:01:43.716998] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=1021964][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:42.693|207=1.02s|210=0us|222=2us|total=1.02s) [2025-08-14 09:01:42.734127] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=27] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:42.736950] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:43.717053] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=981440] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:43.604168] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:1, time_interval:3.00s, expected_run_time:09:01:43.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.717254] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=112912][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:43.604|207=113.08ms|210=0us|222=2us|total=113.10ms) [2025-08-14 09:01:43.721516] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:42.760976] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1421) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=9] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-08-14 09:01:42.734852] WDIAG stat_cleanup (debug.c:35) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] PNIO [delay_warn] cost too much time: 2898020us, procedure: pn_pkts_handle_func [2025-08-14 09:01:43.717061] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=978629][errcode=0] epoll handler too much time :980122 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:42.742334] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:43.733082] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=989248][errcode=0] epoll handler too much time :990755 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:42.742363] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=44] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:43.733125] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=989246] temporary file module use no blocks [2025-08-14 09:01:43.733135] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:43.733148] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:42.749529] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:01:41.761325] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:43.717067] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:41.822633] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=554083851, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:43.733870] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1908325] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:41.836437] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=19] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:43.733970] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1894642] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:41.881007] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:01:41.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.734086] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=1850257][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:41.881|207=1.85s|210=0us|222=2us|total=1.85s) [2025-08-14 09:01:42.737940] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:41.753530] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=12] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:43.738619] WDIAG ev_invoke_pending (ev.c:2136) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=1982058][errcode=0] epoll handler too much time :1985093 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:43.738635] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=8] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:43.740321] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b9690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:43.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:43.740353] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891968e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:43.737, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:43.740365] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e71b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891973210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:43.739, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:43.740376] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918119b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a1090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:43.737, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:43.744613] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:43.733343] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=16250] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:43.745590] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=12227] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:43.745604] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:43.745612] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:43.745619] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:43.745626] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:43.736145] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15979] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:42.754111] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get_max_applied_scn(scn={val:1755161749741193000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:50, palf_committed_end_lsn_:{lsn:1604090090}, palf_committed_end_scn_:{val:1755161749741193000, v:0}, last_check_scn_:{val:1755161749741193001, v:0}, max_applied_cb_scn_:{val:1755161749741193000, v:0}}) [2025-08-14 09:01:43.733319] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1968986] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:01:43.734440] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=994994][errcode=0] epoll handler too much time :996518 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:43.747096] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=991472] get_min_unreplayed_log_info(lsn={lsn:1604088821}, scn={val:1755100948296325001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873479, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:12492179, ret_code_:0}}, next_to_submit_lsn_:{lsn:1604088821}, next_to_submit_scn_:{val:1755100948296325001, v:0}, base_lsn_:{lsn:1409200128}, base_scn_:{val:1753840802540914000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:1604088821}, end_lsn:{lsn:1604088821}, read_buf:{buf_len_:3747840, buf_:0x7f98f6a05000}, block_size:67104768, log_storage_:0x7f990bfebbf0, log_storage_:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:42, read_io_size:147709952, read_disk_cost_ts:1061450, start_lsn:{lsn:1409200128}}}}}}) [2025-08-14 09:01:43.733162] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623004182014, flush_io_finished_ret_:0, flush_io_finished_round_:26251, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:43.747413] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:43.750815] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:43.750855] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:43.741174] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:43.751156] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72774b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ea810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:43.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:43.751176] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fbe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7280c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:43.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:43.752711] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:43.753881] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:43.747194] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=94] get wrs ts(ls_id={id:1}, delta=354005685, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:43.755185] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7972] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:43.755207] INFO [STORAGE] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:123) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] wead read timestamp is too old(timestamp={val:1755161749741193000, v:0}, gts_scn={val:1755161762721798002, v:0}, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1755161749741193000, v:0}}) [2025-08-14 09:01:43.748790] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=-4389] timer task use too much time(end=1755162103734122, start=1755162101881005, delta=1853117) [2025-08-14 09:01:43.747082] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10919][errcode=0] epoll handler too much time :10947 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:43.755783] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task use too much time(end=1755162103717267, start=1755162103604166, delta=113101) [2025-08-14 09:01:43.747162] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13821] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:01:43.755807] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=8630][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:43.745642] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:01:43.755837] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=10179] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:01:43.756875] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=359028084, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:43.756900] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:43.761949] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891968990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:43.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:43.761978] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891973090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:43.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:43.761990] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7280b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:43.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:43.772011] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919aef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b9090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:43.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:43.772050] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7270a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919a4190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:43.762, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:43.772063] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891971710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:43.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:43.772073] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a0f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:43.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:43.772083] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7ef890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:43.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:43.755255] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task use too much time(end=1755162103717012, start=1755162102693468, delta=1023544) [2025-08-14 09:01:43.776090] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=20800][errcode=0] task scheduled too delay(scheduled_delay=1220813, expected_run_ts_us_=1755162102555275, ObClockGenerator::getRealClock()=1755162103776089, ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:01:42.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.776130] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:01:42.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.755153] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task use too much time(end=1755162103716426, start=1755162101749534, delta=1966892) [2025-08-14 09:01:43.776311] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=21122][errcode=0] task scheduled too delay(scheduled_delay=2020822, expected_run_ts_us_=1755162101755487, ObClockGenerator::getRealClock()=1755162103776310, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:41.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.776346] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:41.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.776362] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1687046, expected_run_ts_us_=1755162102089316, ObClockGenerator::getRealClock()=1755162103776362, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:42.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.776373] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:42.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.776386] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1572088, expected_run_ts_us_=1755162102204297, ObClockGenerator::getRealClock()=1755162103776385, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:42.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.776402] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:42.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:41.776471] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:01:43.776433] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:43.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.776488] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161101770923, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755161572801635) [2025-08-14 09:01:43.777074] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=556037999, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:43.777088] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:43.780932] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:43.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.782084] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f734de30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891919810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:43.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:43.782116] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891972f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:43.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:43.782127] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196b090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:43.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:43.782137] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d59b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ed090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:43.773, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:43.782147] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7efa10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:43.777, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:43.792119] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72eba10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:43.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:43.792144] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f0190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:43.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:43.802196] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728a790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:43.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:43.802222] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7281110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:43.790, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:43.817322] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [723][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=23] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=16, total_destructed_detector=16, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f99333562b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-08-14 09:01:43.832387] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f1690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:43.827, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:43.833397] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:43.834009] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=6] tx gc loop thread is running(MTL_ID()=1) [2025-08-14 09:01:43.834020] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] try gc retain ctx [2025-08-14 09:01:43.834025] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=5] try to retry start_working [2025-08-14 09:01:43.835652] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=556096389, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:43.839043] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EE91-0-0] [lt=22] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-08-14 09:01:43.842424] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739af30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196bc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:43.837, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:43.852463] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739e510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:43.841, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:43.852488] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f0310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:43.849, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:43.852499] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7280f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:43.849, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:43.880993] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:43.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.902782] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728a610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:43.892, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:43.922845] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7ee090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:43.917, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:43.932916] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7281a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:43.922, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:43.981044] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:43.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.997449] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:43.989, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:44.017529] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:44.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:44.081122] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:43.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:44.098036] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:44.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:44.181182] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:44.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:44.218665] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:44.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:44.281215] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:44.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:44.381277] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:44.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:44.481351] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:44.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:44.493312] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fa190, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa470, time_wheel:0x7f99341cdbf0, thread_pool:0x7f99341cd3d0, is_running:1, time_interval:2.00s, expected_run_time:09:01:44.334, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:44.505137] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fe410, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe6f0, time_wheel:0x7f9922df9d40, thread_pool:0x7f9922df9520, is_running:1, time_interval:2.00s, expected_run_time:09:01:44.432, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:44.579665] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a4658, timer:0x7f9956bffee0, task:0x7f9969bb3038, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr12TabletGCTaskE", scheduled_time:1755161923366542, pushed_time:0, delay:20000}, running_task_token={this:0x7f994e1a4498, timer:0x7f9956bffee0, task:0x7f9969bb3010, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr11TableGCTaskE", scheduled_time:1755161923356444, pushed_time:1755161923356659, delay:20000}, total_delay=181213122, threashold=1000000, tenant_id=1) [2025-08-14 09:01:44.581413] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:44.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:44.681475] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:44.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:44.721315] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:44.721771] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=8] PNIO [ratelimit] time: 1755162104721769, bytes: 0, bw: 0.000000 MB/s, add_ts: 1004772, add_bytes: 0 [2025-08-14 09:01:42.717506] INFO [SHARE] end_run (ob_tenant_base.cpp:389) [15515][T1001_TimerWK12][T0][Y0-0000000000000000-0-0] [lt=26538] tenant thread end_run(ret=0, thread_count_=133, id_=1001, GET_GROUP_ID()=0) [2025-08-14 09:01:43.855376] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=6499][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:01:43.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:43.878975] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c0b90, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571c0e70, time_wheel:0x7f9959fef990, thread_pool:0x7f9959fef170, is_running:1, time_interval:3.00s, expected_run_time:09:01:43.764, task_priority:1, with_handle_protected:1}) [553]OB_MALLOC COST TOO MUCH TIME, cost_time=1969591, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=55, WASH_BLOCK_END=1969530, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=4], seq:[0, 1, 2, 3, 4] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=1620528, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=41, WASH_BLOCK_END=1620487], seq:[0, 1, 2] [2025-08-14 09:01:43.148838] EDIAG try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1620580, owner: ObLog, click_count: 5, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=1620546, ALLOC_END=23, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x26499505 0x97074a3 0x26498fa9 0x7f9973e081cf 0x7f9973a39dd3 [659]OB_MALLOC COST TOO MUCH TIME, cost_time=995108, tenant_id=500, label=OB_LOG_ELECTION_PREPARE_REQUEST, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=86, WASH_BLOCK_END=995016, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=4], seq:[0, 1, 2, 3, 4] [705]OB_MALLOC COST TOO MUCH TIME, cost_time=2942433, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=42, WASH_BLOCK_END=2942356, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=35], seq:[0, 1, 2, 3, 4] [388]OB_MALLOC COST TOO MUCH TIME, cost_time=3967075, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1003902, WASH_BLOCK_END=969284, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1016170], seq:[0, 1, 2, 3, 4] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=990247, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [833]OB_MALLOC COST TOO MUCH TIME, cost_time=2941532, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=42, WASH_BLOCK_END=2941490], seq:[0, 1, 2] [694]OB_MALLOC COST TOO MUCH TIME, cost_time=2847304, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=42, WASH_BLOCK_END=2847261], seq:[0, 1, 2] [404]OB_MALLOC COST TOO MUCH TIME, cost_time=2846411, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=2846371], seq:[0, 1, 2] [2025-08-14 09:01:42.879697] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2846454, owner: ObLog, click_count: 5, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=2846420, ALLOC_END=21, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [718]OB_MALLOC COST TOO MUCH TIME, cost_time=2908557, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=45, WASH_BLOCK_END=2908512], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=1976733, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=1976693], seq:[0, 1, 2] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=1972884, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=42, WASH_BLOCK_END=1972842], seq:[0, 1, 2] [400]OB_MALLOC COST TOO MUCH TIME, cost_time=982729, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=39, WASH_BLOCK_END=982672, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=18], seq:[0, 1, 2, 3, 4] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=1948493, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=32, WASH_BLOCK_END=1948461], seq:[0, 1, 2] [822]OB_MALLOC COST TOO MUCH TIME, cost_time=1947912, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=31, WASH_BLOCK_END=1947881], seq:[0, 1, 2] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=1944709, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=32, WASH_BLOCK_END=1944677], seq:[0, 1, 2] [2025-08-14 09:01:42.784344] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2975398, owner: ObLog, click_count: 5, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=2941554, ALLOC_END=33831, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [276]OB_MALLOC COST TOO MUCH TIME, cost_time=2678618, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=54, WASH_BLOCK_END=2678530, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=34], seq:[0, 1, 2, 3, 4] [2025-08-14 09:01:43.081669] EDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 2678684, owner: ObLog, click_count: 5, time dist:[FORMAT_END=22, ObMalloc#start=1, ObMalloc#end=2678630, ALLOC_END=31, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x9757ced 0x97569df 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [1031]OB_MALLOC COST TOO MUCH TIME, cost_time=2029473, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=39, WASH_BLOCK_END=2029434], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=2029377, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=25, WASH_BLOCK_END=2029352], seq:[0, 1, 2] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=2029344, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=43, WASH_BLOCK_END=2029301], seq:[0, 1, 2] [737]OB_MALLOC COST TOO MUCH TIME, cost_time=2029287, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=26, WASH_BLOCK_END=2029260], seq:[0, 1, 2] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=2029211, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=39, WASH_BLOCK_END=2029172], seq:[0, 1, 2] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=2029166, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=25, WASH_BLOCK_END=2029141], seq:[0, 1, 2] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=2028017, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=2027977], seq:[0, 1, 2] [317]OB_MALLOC COST TOO MUCH TIME, cost_time=2027660, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=41, WASH_BLOCK_END=2027619], seq:[0, 1, 2] [389]OB_MALLOC COST TOO MUCH TIME, cost_time=2027599, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=37, WASH_BLOCK_END=2027562], seq:[0, 1, 2] [381]OB_MALLOC COST TOO MUCH TIME, cost_time=2027557, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=49, WASH_BLOCK_END=2027508], seq:[0, 1, 2] [2025-08-14 09:01:44.735586] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=855311][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:43.878|207=856.61ms|210=1us|222=2us|total=856.63ms) [2025-08-14 09:01:44.735611] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4389] timer task use too much time(end=1755162104735610, start=1755162103878973, delta=856637) [2025-08-14 09:01:43.835669] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:44.736826] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8678] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:44.737622] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b9390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:44.728, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:44.737651] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f734c030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891807390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:44.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:44.742161] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9957] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:43.933460] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:44.676633] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:01:44.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:44.733292] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=982175] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:01:44.747273] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14021] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:44.747320] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=41] temporary file module use no blocks [2025-08-14 09:01:44.747678] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919aae10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:44.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:44.747701] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891971b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:44.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:44.747712] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e67b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891972c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:44.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:44.747723] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739e390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:44.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:44.749657] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=355007555, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [397]OB_MALLOC COST TOO MUCH TIME, cost_time=2027624, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=45, WASH_BLOCK_END=2027579], seq:[0, 1, 2] [2025-08-14 09:01:44.749676] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:44.750223] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:44.747327] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:44.750305] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=2970] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:44.750321] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623006186973, flush_io_finished_ret_:0, flush_io_finished_round_:26252, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:44.744442] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:44.747440] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:44.753650] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=11] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:44.754138] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:44.758172] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891968690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:44.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:44.758203] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891968510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:44.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:44.758220] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891972a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:44.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:44.758231] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ed390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:44.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:44.768245] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7280090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:44.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:44.750281] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:44.770997] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20682][errcode=0] epoll handler too much time :20722 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:44.749707] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get_max_applied_scn(scn={val:1755161744727824001, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:49, palf_committed_end_lsn_:{lsn:7323686009}, palf_committed_end_scn_:{val:1755161747742105001, v:0}, last_check_scn_:{val:1755161749741193000, v:0}, max_applied_cb_scn_:{val:1755161744727824001, v:0}}) [2025-08-14 09:01:44.771585] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21837] get_min_unreplayed_log_info(lsn={lsn:7323685240}, scn={val:1755100946796590002, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873485, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:8946764, ret_code_:0}}, next_to_submit_lsn_:{lsn:7323685240}, next_to_submit_scn_:{val:1755100946796590002, v:0}, base_lsn_:{lsn:7180210176}, base_scn_:{val:1753840803166611000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:7323685240}, end_lsn:{lsn:7323685240}, read_buf:{buf_len_:3747840, buf_:0x7f98f7a05000}, block_size:67104768, log_storage_:0x7f9901131bf0, log_storage_:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:28, read_io_size:98508800, read_disk_cost_ts:645339, start_lsn:{lsn:7180210176}}}}}}) [2025-08-14 09:01:42.799206] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=32] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:01:43.148838] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=11] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:01:44.772167] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=74] get wrs ts(ls_id={id:1001}, delta=360044102, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:44.772181] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:43.776068] WDIAG eloop_delay_warn (debug.h:93) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=996610][errcode=0] PNIO [delay_warn] eloop handle events delay high: 3896027, malloc=0/0 write=0/0 read=0/1 server_process=2898020/1 client_cb=0/0 [2025-08-14 09:01:44.774137] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=22] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [574]OB_MALLOC COST TOO MUCH TIME, cost_time=2027573, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=30, WASH_BLOCK_END=2027543], seq:[0, 1, 2] [2025-08-14 09:01:44.774166] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] temporary file module use no blocks [2025-08-14 09:01:44.774172] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:44.774187] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:44.774197] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=9] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623006213841, flush_io_finished_ret_:0, flush_io_finished_round_:26279, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:44.777258] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:44.777279] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=17] temporary file module use no blocks [2025-08-14 09:01:44.777288] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:44.777299] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:44.777311] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623006215960, flush_io_finished_ret_:0, flush_io_finished_round_:26286, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:44.782505] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:44.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:44.788431] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72707b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919a4010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:44.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:44.798476] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=54][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b4d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:44.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:44.808539] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739acb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196bb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:44.798, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:44.818568] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891973690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:44.806, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:44.818596] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891973510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:44.809, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:44.829920] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196b510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:44.815, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:44.850059] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891973390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:44.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:44.850086] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ed210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:44.832, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:44.850100] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72eb410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:44.841, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:44.860135] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:44.859, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:44.872025] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=1620580] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:01:44.882542] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:44.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:44.982595] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:44.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:45.082658] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:44.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:45.182720] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:45.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:45.282781] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:45.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:45.382820] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:45.293, task_priority:1, with_handle_protected:1}) [387]OB_MALLOC COST TOO MUCH TIME, cost_time=2027534, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=53, WASH_BLOCK_END=2027481], seq:[0, 1, 2] [2025-08-14 09:01:45.482870] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:45.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:45.582934] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:45.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:45.677165] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=69973][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:01:45.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:45.682997] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:45.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:45.720956] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:45.725054] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73516b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196f510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:45.716, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:45.727891] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=25] PNIO [ratelimit] time: 1755162105727887, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006118, add_bytes: 0 [2025-08-14 09:01:45.728788] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=898859] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161101770923, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755161572801635) [2025-08-14 09:01:45.729446] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] get wrs ts(ls_id={id:1}, delta=557989187, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:45.729471] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:42.878718] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb2658, timer:0x7f9934119ee0, task:0x7f9934405038, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr12TabletGCTaskE", scheduled_time:1755161981770099, pushed_time:0, delay:20000}, running_task_token={this:0x7f9931bb2578, timer:0x7f9934119ee0, task:0x7f9934405028, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr17RefreshConfigTaskE", scheduled_time:1755161881725525, pushed_time:1755162043719350, delay:10000000}, total_delay=121108617, threashold=1000000, tenant_id=1001) [2025-08-14 09:01:42.879697] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162101.789289 cost: 1.090406 loop:26450, slowcnt: 10 [2025-08-14 09:01:42.817359] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=25] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:01:45.735979] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b5990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:45.720, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:45.736011] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b5810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:45.727, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:45.736026] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ed690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:45.727, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:45.736038] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72851b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7289410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:45.721, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:44.773699] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=355031762, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:45.737520] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=962360] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:45.739165] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1001}, delta=361010753, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:45.739217] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=48] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:45.745130] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6721] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:45.746684] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739e210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:45.737, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:45.746714] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7281d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:45.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:45.747169] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3741] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:45.750031] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [395]OB_MALLOC COST TOO MUCH TIME, cost_time=2027560, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=50, WASH_BLOCK_END=2027510], seq:[0, 1, 2] [2025-08-14 09:01:45.750256] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=34] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:45.753173] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:45.753413] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=11] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [477]OB_MALLOC COST TOO MUCH TIME, cost_time=1985663, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=31, WASH_BLOCK_END=1985632], seq:[0, 1, 2] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=1985030, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=33, WASH_BLOCK_END=1984997], seq:[0, 1, 2] [851]OB_MALLOC COST TOO MUCH TIME, cost_time=1946047, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=46, WASH_BLOCK_END=1946001], seq:[0, 1, 2] [2025-08-14 09:01:43.816204] EDIAG [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [851][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1946110, owner: ObLog, click_count: 7, time dist:[FORMAT_END=21, ObMalloc#start=1, ObMalloc#end=1946054, ObMalloc#start=30, ObMalloc#end=3, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x973b640 0x973b132 0x973a32d 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:42.722690] EDIAG stat_cleanup (debug.c:35) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3038086, owner: ObLog, click_count: 7, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=1009810, ObMalloc#start=20, ObMalloc#end=2027673, ALLOC_END=570, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x27a8a131 0x97bd8be 0x97bac33 0x97ba441 0x27a8a6d6 0x97b7ba0 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [416]OB_MALLOC COST TOO MUCH TIME, cost_time=1006595, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1006588], seq:[0] [854]OB_MALLOC COST TOO MUCH TIME, cost_time=1994299, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1994290], seq:[0] [2025-08-14 09:01:45.756746] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891825690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:45.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:45.756776] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891807210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:45.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:45.756790] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891969590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:45.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:45.756802] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739e090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:45.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:42.784344] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=25] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:01:43.081669] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc43b8, timer:0x7f9970f40f80, task:0x564d6d07a448, task_type:"N9oceanbase5share17ObServerTraceTaskE", scheduled_time:1755161621733806, pushed_time:0, delay:8000000}, running_task_token={this:0x7f9970fc5fb8, timer:0x7f9970f40f80, task:0x564d727182f0, task_type:"N9oceanbase5share16ObAshRefreshTaskE", scheduled_time:1755161611717613, pushed_time:1755161614757742, delay:120000000}, total_delay=481347862, threashold=1000000, tenant_id=500) [2025-08-14 09:01:45.737035] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:45.764771] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27692][errcode=0] epoll handler too much time :27747 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:43.816204] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [851][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=24] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=16, total_destructed_detector=16, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f9922fee2b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-08-14 09:01:45.765280] WDIAG [DETECT] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [851][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=1946110][errcode=0] cost too much time:ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t), (this=|threshold=100.00ms|start at 09:01:45.765|total=1.95s) [2025-08-14 09:01:42.722690] WDIAG stat_cleanup (debug.c:35) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] PNIO [delay_warn] cost too much time: 2855020us, procedure: pn_pkts_handle_func [2025-08-14 09:01:45.770989] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d67b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891969290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:45.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:42.742668] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3995735, owner: ObLog, click_count: 6, time dist:[FORMAT_END=8, ObMalloc#start=0, ObMalloc#end=990252, ObMalloc#start=20, ObMalloc#end=2027576, FREE_END=977878], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:41.765139] EDIAG [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4971949, owner: ObLog, click_count: 6, time dist:[FORMAT_END=13, ObMalloc#start=1, ObMalloc#end=1964046, ObMalloc#start=26, ObMalloc#end=2029305, FREE_END=978557], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1bf2ec62 0x986ebd7 0x986bee6 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:41.772953] EDIAG do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4964173, owner: ObLog, click_count: 6, time dist:[FORMAT_END=15, ObMalloc#start=1, ObMalloc#end=1956376, ObMalloc#start=19, ObMalloc#end=2029218, FREE_END=978544], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x202c530b 0x202c6888 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:41.753646] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=40][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4984093, owner: ObLog, click_count: 6, time dist:[FORMAT_END=7, ObMalloc#start=0, ObMalloc#end=1977763, ObMalloc#start=21, ObMalloc#end=2027544, FREE_END=978758], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [472]OB_MALLOC COST TOO MUCH TIME, cost_time=1015673, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=32, WASH_BLOCK_END=1015603, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=38], seq:[0, 1, 2, 3, 4] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=1015982, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=18, WASH_BLOCK_END=1015964], seq:[0, 1, 2] [400]OB_MALLOC COST TOO MUCH TIME, cost_time=1015909, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=25, WASH_BLOCK_END=1015884], seq:[0, 1, 2] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=1015879, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=17, WASH_BLOCK_END=1015862], seq:[0, 1, 2] [822]OB_MALLOC COST TOO MUCH TIME, cost_time=1015891, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=18, WASH_BLOCK_END=1015873], seq:[0, 1, 2] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=1015903, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=17, WASH_BLOCK_END=1015885], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=874616, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=29, WASH_BLOCK_END=874587], seq:[0, 1, 2] [513]OB_MALLOC COST TOO MUCH TIME, cost_time=864285, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=26, WASH_BLOCK_END=864259], seq:[0, 1, 2] [2025-08-14 09:01:42.732909] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4009756, owner: ObLog, click_count: 6, time dist:[FORMAT_END=16, ObMalloc#start=3, ObMalloc#end=999713, ObMalloc#start=21, ObMalloc#end=2027607, FREE_END=982396], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:42.734011] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4008756, owner: ObLog, click_count: 6, time dist:[FORMAT_END=13, ObMalloc#start=1, ObMalloc#end=998786, ObMalloc#start=20, ObMalloc#end=2027582, FREE_END=982354], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:41.776103] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [477][T1_StandbySchem][T1][YB42AC4D0007-00063C495B6A02F9-0-0] [lt=24][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4965278, owner: ObLog, click_count: 6, time dist:[FORMAT_END=26, ObMalloc#start=0, ObMalloc#end=1954504, ObMalloc#start=42800, ObMalloc#end=1985680, FREE_END=982268], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:42.733426] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4009442, owner: ObLog, click_count: 6, time dist:[FORMAT_END=7, ObMalloc#start=0, ObMalloc#end=999276, ObMalloc#start=30, ObMalloc#end=2027636, FREE_END=982493], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:43.783401] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2961410, owner: ObLog, click_count: 6, time dist:[FORMAT_END=10, ObMalloc#start=0, ObMalloc#end=1944717, ObMalloc#start=20, ObMalloc#end=1015911, FREE_END=752], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:43.891980] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2853193, owner: ObLog, click_count: 8, time dist:[FORMAT_END=853442, ObMalloc#start=2, ObMalloc#end=982741, ObMalloc#start=19, ObMalloc#end=1015916, ObMalloc#start=983, ObMalloc#end=28, FREE_END=62], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:43.779540] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2965511, owner: ObLog, click_count: 8, time dist:[FORMAT_END=22, ObMalloc#start=1, ObMalloc#end=1948501, ObMalloc#start=19, ObMalloc#end=1015886, ObMalloc#start=777, ObMalloc#end=51, FREE_END=254], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:43.750968] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2994841, owner: ObLog, click_count: 8, time dist:[FORMAT_END=20, ObMalloc#start=1, ObMalloc#end=1976744, ObMalloc#start=21, ObMalloc#end=1015697, ObMalloc#start=2280, ObMalloc#end=30, FREE_END=47], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [857]OB_MALLOC COST TOO MUCH TIME, cost_time=1016883, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [854]OB_MALLOC COST TOO MUCH TIME, cost_time=2989347, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2956710], seq:[0, 1] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=1041390, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1006616], seq:[0, 1] [683]OB_MALLOC COST TOO MUCH TIME, cost_time=276554, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=41, WASH_BLOCK_END=276513], seq:[0, 1, 2] [797]OB_MALLOC COST TOO MUCH TIME, cost_time=260672, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=36, WASH_BLOCK_END=260634], seq:[0, 1, 2] [2025-08-14 09:01:44.958335] EDIAG [COMMON] generate_mod_stat_task (memory_dump.cpp:213) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1811147, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1810299, ObMalloc#start=1, ObMalloc#end=29, ObMalloc#start=20, ObMalloc#end=604, FREE_END=194], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf98b221 0xf98aea2 0x96b1d30 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:43.780149] EDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2965039, , BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0xfb622cd 0xfb619cb 0xfb892d3 0x994150a 0x993eecd 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [528]OB_MALLOC COST TOO MUCH TIME, cost_time=2942990, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=2942981], seq:[0] [2025-08-14 09:01:42.763312] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4949284, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=6, ALLOC_BLOCK#end=3, FORMAT_END=4949272, ALLOC_END=2, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:44.770942] EDIAG [COORDINATOR] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [657][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2944859, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2944852, ObMalloc#start=1, ObMalloc#end=4, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ea7c4 0x9818f9c 0x1231eaee 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:41.755634] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5963233, owner: ObLog, click_count: 5, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=3967085, ALLOC_END=1996136, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:42.738928] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [776][T1002_StandbySc][T1002][YB42AC4D0007-00063C495DFA02EB-0-0] [lt=16][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4981486, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=9, ALLOC_BLOCK#end=14866, FORMAT_END=4958795, ALLOC_END=7816, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:41.778757] EDIAG [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1935238, owner: ObLog, click_count: 5, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=1935196, ALLOC_END=28, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1f832f91 0x97c36fe 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:41.776471] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=20][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1996934, owner: ObLog, click_count: 5, time dist:[FORMAT_END=8, ObMalloc#start=0, ObMalloc#end=1954176, ALLOC_END=42750, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [528]OB_MALLOC COST TOO MUCH TIME, cost_time=2964113, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2943017], seq:[0, 1] [2025-08-14 09:01:43.716587] EDIAG [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4018533, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1054396, ObMalloc#start=1, ObMalloc#end=2964131, ALLOC_END=4, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x12377491 0x1236e8e9 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [717]OB_MALLOC COST TOO MUCH TIME, cost_time=1964094, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1964086], seq:[0] [2025-08-14 09:01:45.783058] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:45.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:45.792514] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891807090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:45.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:45.792543] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739aa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196b990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:45.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:45.802591] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ab710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:45.791, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:45.802622] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ecf10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:45.791, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:45.802649] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7280810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:45.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:45.802660] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7288810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:45.799, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:45.812651] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891972d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:45.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:45.812677] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7351e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196f990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:45.803, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:45.812688] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a22b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ed510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:45.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:45.812698] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7288690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:45.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:45.832771] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ebe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:45.826, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:45.832796] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7350530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196ea90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:45.824, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:45.872977] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fbbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7280e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:45.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:45.883000] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72860b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7288c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:45.873, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:45.883140] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:45.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:45.983203] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:45.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:46.083267] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:45.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:46.183311] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:46.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:46.283376] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:46.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:46.336477] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7faa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7281890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:46.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:46.336512] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72842b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7289b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:46.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:46.383401] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:46.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:46.483456] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:46.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:46.583517] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:46.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:46.627976] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7eef90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:46.614, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:46.628006] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eaf30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7ef110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:46.614, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:46.657307] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:1, time_interval:3.00s, expected_run_time:09:01:46.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:46.677759] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:01:46.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:46.683578] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:46.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:46.720261] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:45.742194] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:46.722945] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=979267][errcode=0] epoll handler too much time :980762 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:45.746847] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=13432] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:01:45.750917] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:46.723639] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=971245] temporary file module use no blocks [2025-08-14 09:01:46.723652] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:46.723667] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:41.763024] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:01:45.764424] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=2678684][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc56f8, timer:0x7f9970f40f80, task:0x564d6ef58bc8, task_type:"N9oceanbase7storage24ObPrintTenantMemoryUsageE", scheduled_time:1755161623782823, pushed_time:0, delay:10000000}, running_task_token={this:0x7f9970fc5fb8, timer:0x7f9970f40f80, task:0x564d727182f0, task_type:"N9oceanbase5share16ObAshRefreshTaskE", scheduled_time:1755161611717613, pushed_time:1755161614757742, delay:120000000}, total_delay=481981600, threashold=1000000, tenant_id=500) [2025-08-14 09:01:46.725089] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=959212][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc4818, timer:0x7f9970f40f80, task:0x564d6eb457d0, task_type:"N9oceanbase3sql23ObDASTaskResultGCRunnerE", scheduled_time:1755161623784040, pushed_time:0, delay:10000000}, running_task_token={this:0x7f9970fc5fb8, timer:0x7f9970f40f80, task:0x564d727182f0, task_type:"N9oceanbase5share16ObAshRefreshTaskE", scheduled_time:1755161611717613, pushed_time:1755161614757742, delay:120000000}, total_delay=482941048, threashold=1000000, tenant_id=500) [2025-08-14 09:01:41.755969] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:41.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:45.765538] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162105728810}, version_delta=558027021) [2025-08-14 09:01:46.733855] WDIAG pktc_handle_req_queue (pktc_post.h:138) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=994636][errcode=0] PNIO [delay_warn] delay high: 4134028 [2025-08-14 09:01:46.735329] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=17] PNIO [ratelimit] time: 1755162106735326, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007439, add_bytes: 0 [2025-08-14 09:01:46.723682] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623007190120, flush_io_finished_ret_:0, flush_io_finished_round_:26253, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:46.744513] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3995735][errcode=0] epoll handler too much time :4001822 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:46.744561] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:46.744582] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=26] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:46.745355] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4984093][errcode=0] epoll handler too much time :4991716 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:46.745438] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=73] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:46.745459] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:45.883031] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a6958, timer:0x7f9966fddee0, task:0x7f99697d33a0, task_type:"N9oceanbase5table20ObTableClientInfoMgr27ObTableClientInfoRetireTaskE", scheduled_time:1755161984788358, pushed_time:0, delay:60000000}, running_task_token={this:0x7f994e1a65d8, timer:0x7f9966fddee0, task:0x7f994f1bbcc0, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161694824965, pushed_time:1755161925772608, delay:5000000}, total_delay=121094671, threashold=1000000, tenant_id=1) [2025-08-14 09:01:46.741038] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:46.736753] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:46.749431] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12659][errcode=0] epoll handler too much time :12694 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:46.752578] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=36] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:46.753011] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:46.753112] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891968810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:46.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:46.753150] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891970810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:46.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:46.753161] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919827b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891970690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:46.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:46.754408] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=9] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:43.763830] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [854][T1002_ObXAHbWor][T1002][Y0-0000000000000000-0-0] [lt=6921] XA scheduler heartbeat task statistics(avg_time=2) [2025-08-14 09:01:46.757738] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119812272, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-08-14 09:01:46.757785] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] get wrs ts(ls_id={id:1}, delta=357016230, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:46.757812] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:46.758440] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25] get wrs ts(ls_id={id:1001}, delta=362029599, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:46.758453] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:46.764057] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739f890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:46.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:46.751855] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=2281][errcode=-4389] timer task use too much time(end=1755162106751853, start=1755162103779538, delta=2972315) [2025-08-14 09:01:46.771105] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=19216][errcode=0] task scheduled too delay(scheduled_delay=2978754, expected_run_ts_us_=1755162103792349, ObClockGenerator::getRealClock()=1755162106771103, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:43.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:46.771151] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:43.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:46.771224] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:46.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:46.752591] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=2212][errcode=-4389] timer task use too much time(end=1755162106752590, start=1755162103750967, delta=3001623) [2025-08-14 09:01:46.774118] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72742b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b4910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:46.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:46.774155] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919840b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891971590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:46.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:46.774167] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180ef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739f710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:46.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:46.774178] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180ecb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739f590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:46.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:46.774188] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7288b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:46.767, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:46.774206] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a0790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:46.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:46.778370] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=23] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:46.778392] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=16] temporary file module use no blocks [2025-08-14 09:01:46.778400] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:46.778410] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:46.778421] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623008218065, flush_io_finished_ret_:0, flush_io_finished_round_:26288, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:46.783644] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:46.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:46.784165] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919925b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196d890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:46.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:46.784186] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a1b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:46.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:46.784196] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7289a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:46.777, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:46.794266] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728ca30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7291590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:46.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:46.804331] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739ee10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:46.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:46.814367] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ebd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:46.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:46.814405] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739ec90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:46.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:46.824443] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72874b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7288090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:46.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:46.834504] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7283090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:46.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:46.834529] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739dc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728a490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:46.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:46.836847] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c0f70, caller:mds_tenant_service.cpp:mtl_start:117, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_145, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571c1250, time_wheel:0x7f9959fef990, thread_pool:0x7f9959fef170, is_running:1, time_interval:15.00s, expected_run_time:09:01:46.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:46.836872] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c0b90, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571c0e70, time_wheel:0x7f9959fef990, thread_pool:0x7f9959fef170, is_running:1, time_interval:3.00s, expected_run_time:09:01:46.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:46.883708] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:46.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:45.930604] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=2846454][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162101.789289 cost: 4.141314 loop:26450, slowcnt: 12 [2025-08-14 09:01:46.904801] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a59b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7282790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:46.895, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:46.948818] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=864346][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a74b8, timer:0x7f9966fddee0, task:0x7f9969106030, task_type:"N9oceanbase8observer20ObTableQueryASyncMgrE", scheduled_time:1755161985758107, pushed_time:0, delay:0}, running_task_token={this:0x7f994e1a65d8, timer:0x7f9966fddee0, task:0x7f994f1bbcc0, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161694824965, pushed_time:1755161925772608, delay:5000000}, total_delay=121190709, threashold=1000000, tenant_id=1) [2025-08-14 09:01:47.678350] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571634a0, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957163780, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:5.00s, expected_run_time:09:01:47.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:42.763312] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:47.720125] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4949284][errcode=0] epoll handler too much time :4956823 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:47.720155] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:47.720167] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:44.770942] WDIAG [COORDINATOR] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [657][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=0] cost too much time:ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_(), (this=|threshold=1.00s|start at 09:01:47.720|total=4.94s) [2025-08-14 09:01:47.721406] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:46.801292] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:46.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:47.725897] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=923202][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:46.801|207=924.60ms|210=0us|222=2us|total=924.64ms) [2025-08-14 09:01:47.725932] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=-4389] timer task use too much time(end=1755162107725931, start=1755162106801290, delta=924641) [2025-08-14 09:01:47.725948] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=715915, expected_run_ts_us_=1755162107010031, ObClockGenerator::getRealClock()=1755162107725947, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:47.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:46.871539] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=18854][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:46.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:46.886845] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=2965039] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:01:46.949594] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=2853193][errcode=0] EASY SLOW: thread: 7f9953e7d300, lastrun: 1755162103.747188 cost: 3.202404 loop:26475, slowcnt: 3 [2025-08-14 09:01:47.048916] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a6798, timer:0x7f9966fddee0, task:0x7f99697d29f8, task_type:"N9oceanbase5table21ObTableGroupCommitMgr19ObTableGroupOpsTaskE", scheduled_time:1755161925783083, pushed_time:0, delay:1000000}, running_task_token={this:0x7f994e1a65d8, timer:0x7f9966fddee0, task:0x7f994f1bbcc0, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161694824965, pushed_time:1755161925772608, delay:5000000}, total_delay=181265832, threashold=1000000, tenant_id=1) [2025-08-14 09:01:47.678406] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=54][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:01:47.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:47.721860] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=4971949] tx gc loop thread is running(MTL_ID()=1001) [2025-08-14 09:01:47.726661] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=4792] try gc retain ctx [2025-08-14 09:01:47.726669] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=8] try to retry start_working [2025-08-14 09:01:47.726674] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:125) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=4] try to adjust log cb pool [2025-08-14 09:01:47.726692] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=4] get_max_applied_scn(scn={val:1755161719775179002, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:45, palf_committed_end_lsn_:{lsn:8461098908}, palf_committed_end_scn_:{val:1755161719775179002, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161719775179002, v:0}}) [2025-08-14 09:01:41.755634] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:47.736356] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=23] PNIO [ratelimit] time: 1755162107736354, bytes: 0, bw: 0.000000 MB/s, add_ts: 1001028, add_bytes: 0 [2025-08-14 09:01:47.736432] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:47.736787] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891971a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:47.726, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:47.736822] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739eb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:47.721, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:47.726363] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:233) [529][T1_RFLWorker][T1][YB42AC4D0007-00063C495B199C16-0-0] [lt=23789] ObRemoteFetchWorker is running(thread_index=0) [2025-08-14 09:01:47.726399] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=975257] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:01:47.726463] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:288) [860][T1002_ArcFetche][T1002][YB42AC4D0007-00063C495E399C16-0-0] [lt=3559] ObArchiveFetcher is running(thread_index=0) [705]OB_MALLOC COST TOO MUCH TIME, cost_time=1006744, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1006721], seq:[0] [705]OB_MALLOC COST TOO MUCH TIME, cost_time=2023040, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=18, WASH_BLOCK_END=1016258, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1006759], seq:[0, 1, 2, 3, 4] [2025-08-14 09:01:42.781523] EDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4965764, owner: ObLog, click_count: 7, time dist:[FORMAT_END=216, ObMalloc#start=1, ObMalloc#end=2942458, ObMalloc#start=28, ObMalloc#end=2023050, ALLOC_END=9, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0xfc0e73d 0x993f560 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:43.747196] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12722][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4008029, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=2, ALLOC_BLOCK#end=3, FORMAT_END=4008022, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:44.752062] EDIAG [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3004860, owner: ObLog, click_count: 3, time dist:[FORMAT_END=3004859, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x18bdb084 0x18bda6b1 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:45.765396] EDIAG eloop_delay_warn (debug.h:93) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=3038086][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1993125, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=3, FORMAT_END=1993119, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x27ad6769 0x27a8a7e8 0x97b7bb0 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:41.763024] EDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] LOGGER COST TOO MUCH TIME, cost: 4953919, owner: ObLog, click_count: 7, time dist:[FORMAT_END=17, ObMalloc#start=0, ObMalloc#end=1966059, ObMalloc#start=21, ObMalloc#end=2029351, ALLOC_END=958471, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x13b06ffb 0x958a39e 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:46.744602] EDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1008) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1017192, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1017184, ALLOC_END=7, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x27a5a6c0 0x94da840 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:44.248813] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3509747, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=7, ALLOC_BLOCK#end=2, FORMAT_END=3509738, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:47.726319] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=853484][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:46.871|207=854.78ms|210=1us|222=1us|total=854.80ms) [2025-08-14 09:01:47.740408] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=14065][errcode=-4389] timer task use too much time(end=1755162107740407, start=1755162106871537, delta=868870) [2025-08-14 09:01:47.721935] WDIAG pktc_handle_req_queue (pktc_post.h:138) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] PNIO [delay_warn] delay high: 6600046 [2025-08-14 09:01:43.716587] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=32] finish to resize log external storage handler(current_ts=1755162102740952, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-08-14 09:01:47.727921] WDIAG ev_invoke_pending (ev.c:2136) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5963233][errcode=0] epoll handler too much time :5972296 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:47.741251] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13287] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:47.741265] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:47.741301] WDIAG [COORDINATOR] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [657][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=2944859][errcode=0] cost too much time:ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_failure(), (this=|threshold=1.00s|start at 09:01:47.741|total=4.96s) [2025-08-14 09:01:47.725967] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:47.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:47.726719] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=18] get_min_unreplayed_log_info(lsn={lsn:8461098908}, scn={val:1755100947600130001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161920778323, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:23084275, ret_code_:0}}, next_to_submit_lsn_:{lsn:8461098908}, next_to_submit_scn_:{val:1755100947600130001, v:0}, base_lsn_:{lsn:7515734016}, base_scn_:{val:1753840867171901045, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6e05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:8461098908}, end_lsn:{lsn:8461098908}, read_buf:{buf_len_:3747840, buf_:0x7f98f6e05000}, block_size:67104768, log_storage_:0x7f99297a3bf0, log_storage_:{log_tail:{lsn:8461098908}, readable_log_tail:{lsn:8461098908}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8455200768}, min_scn:{val:1755160384851871002, v:0}, curr_block_id:126, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:215, min_block_id:93, max_block_id:127, curr_writable_block_id:126}, logical_block_size_:67104768, curr_block_writable_size_:61206628, block_header_serialize_buf_:0x7f99297a48b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:198, read_io_size:699822080, read_disk_cost_ts:4178035, start_lsn:{lsn:7515734016}}}}}}) [2025-08-14 09:01:47.741710] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=160] update_check_range(min_check_version=2540, max_check_version=2540, global_version=2636) [2025-08-14 09:01:47.744973] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=376] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:47.745525] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:47.747326] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8169] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:47.748734] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2770] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:47.748989] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=4008756] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:47.752319] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:47.752435] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a0610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:47.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:47.753118] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=840] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:47.753167] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1126] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:47.754389] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=769] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:42.781523] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:12632581, max_lsn:8461098908, max_scn:{val:1755161719775179002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:8214679957}, reuse_lsn:{lsn:8461098908}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:8419155868}}, last_submit_lsn:{lsn:8461098786}, last_submit_end_lsn:{lsn:8461098908}, last_submit_log_id:12632581, last_submit_log_pid:45, max_flushed_lsn:{lsn:8461098786}, max_flushed_end_lsn:{lsn:8461098908}, max_flushed_log_pid:45, committed_end_lsn:{lsn:8461098908}, last_slide_log_id:12632581, last_slide_scn:{val:1755161719775179002, v:0}, last_slide_lsn:{lsn:8461098786}, last_slide_end_lsn:{lsn:8461098908}, last_slide_log_pid:45, last_slide_log_accum_checksum:1091335677, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f992979c530}, [StateMgr]={this:0x7f99297a09b0, self:"172.77.0.7:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:46}, leader:"0.0.0.0:0", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:8461098908}, scan_disk_log_finished:true, last_check_start_id:12632582, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161899748977, palf_role_change_cb:0x7f99297a7470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:45, prev:{version:1, config:{config_version:{proposal_id:43, config_seq:41}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:45, config_seq:42}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:41, prev_lsn:{lsn:8461098218}, prev_mode_pid:45}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:41, prev_lsn:{lsn:8461098218}, prev_end_lsn:{lsn:8461098377}, prev_mode_pid:45}, reconfig_barrier:{prev_log_proposal_id:41, prev_lsn:{lsn:8461098218}, prev_end_lsn:{lsn:8461098377}, prev_mode_pid:45}, persistent_config_version:{proposal_id:45, config_seq:42}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755162099867155, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755162099867155, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f992979dc70}, [ModeMgr]={palf_id:1, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:45, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:46, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:46, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:1753719139394335001, v:0}, min_block_id:93, min_block_min_scn:{val:1753712284262753001, v:0}, base_lsn_for_block_gc:{lsn:8186781696}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:46}, log_config_meta:{version:3, proposal_id:45, prev:{version:1, config:{config_version:{proposal_id:43, config_seq:41}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:45, config_seq:42}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:41, prev_lsn:{lsn:8461098218}, prev_mode_pid:45}, log_snapshot_meta:{version:2, base_lsn:{lsn:8186781696}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:46, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:655360}, readable_log_tail:{lsn:655360}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8522305536}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/meta", dir_fd:214, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66449408, block_header_serialize_buf_:0x7f99297a2bb8, flashback_version:0}, log_storage:{log_tail:{lsn:8461098908}, readable_log_tail:{lsn:8461098908}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8455200768}, min_scn:{val:1755160384851871002, v:0}, curr_block_id:126, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:215, min_block_id:93, max_block_id:127, curr_writable_block_id:126}, logical_block_size_:67104768, curr_block_writable_size_:61206628, block_header_serialize_buf_:0x7f99297a48b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1755161900739661, this:0x7f99297a10f0}, [Reconfirm]={palf_id:1, self:"172.77.0.7:2882", state:"FETCH_MAX_LOG_LSN", new_proposal_id:46, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"172.77.0.7:2882", majority_max_accept_pid:45, majority_max_lsn:{lsn:8461098908}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:1755161954782466, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f99297a0b30}) [2025-08-14 09:01:43.747196] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12722] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:47.761319] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4008029][errcode=0] epoll handler too much time :4014130 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:47.761338] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:47.761351] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:42.799206] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1969625, owner: ObLog, click_count: 5, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=1969609, ALLOC_END=3, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:42.817359] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2908952, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=6, ALLOC_BLOCK#end=13, FORMAT_END=5, ObMalloc#start=1, ObMalloc#end=2908899, ALLOC_END=27, APPEND_END=1], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:43.755091] EDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 4003827, owner: ObLog, click_count: 5, time dist:[FORMAT_END=4003815, ObMalloc#start=1, ObMalloc#end=10, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1b53d414 0x97ad9aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:44.921754] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2839152, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2839152, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:42.878718] EDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 2847372, owner: ObLog, click_count: 5, time dist:[FORMAT_END=36, ObMalloc#start=1, ObMalloc#end=2847314, ALLOC_END=21, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x9757ced 0x97569df 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [717]OB_MALLOC COST TOO MUCH TIME, cost_time=1990193, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1964104], seq:[0, 1] [2025-08-14 09:01:45.774289] EDIAG [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1990239, owner: ObLog, click_count: 5, time dist:[FORMAT_END=26, ObMalloc#start=2, ObMalloc#end=1990207, ALLOC_END=3, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1d189030 0x1d18285e 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:43.754926] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4007923, owner: ObLog, click_count: 6, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=1972907, ObMalloc#start=21, ObMalloc#end=1015998, FREE_END=1018986], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 08:58:31.762770] EDIAG [SHARE] pre_run (ob_tenant_base.cpp:375) [15523][][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 67875325, owner: ObLog, click_count: 7, time dist:[FORMAT_END=1853, ObMalloc#start=1, ObMalloc#end=69, ObMalloc#start=67873394, ObMalloc#end=3, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x20332dc6 0x203328a6 0x2668bb79 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:41.755969] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4961872, owner: ObLog, click_count: 9, time dist:[ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=2, FORMAT_END=15, ObMalloc#start=0, ObMalloc#end=1973018, ObMalloc#start=19, ObMalloc#end=2029386, ALLOC_END=959428, APPEND_END=1], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:44.735398] EDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=878685][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3031920, owner: ObLog, click_count: 3, time dist:[FORMAT_END=3031920, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ed35c 0x97aead0 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:41.774506] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4963250, owner: ObLog, click_count: 6, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=1954920, ObMalloc#start=29, ObMalloc#end=2029173, FREE_END=979118], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:44.752062] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=16] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:01:45.765396] WDIAG eloop_delay_warn (debug.h:93) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=3038086][errcode=0] PNIO [delay_warn] eloop handle events delay high: 5898041, malloc=0/0 write=0/0 read=0/1 server_process=2855020/1 client_cb=0/0 [2025-08-14 09:01:47.761565] WDIAG pktc_handle_req_queue (pktc_post.h:138) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=1993125][errcode=0] PNIO [delay_warn] delay high: 7040049 [2025-08-14 09:01:40.753706] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, param_config_:{memory_limit:536870912, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=865568, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:01:47.726478] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=14][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:01:46.744602] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1008) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x7f98f72b5030, session_id:3221590961, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:10, sql_initiative_shutdown:false, reader:{fd:278}, err:5, last_decode_time:1755162075745307, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:01:47.721987] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=23][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:01:44.248813] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:44.239, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:47.763893] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=3509747][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:44.248|207=3.52s|210=0us|222=1us|total=3.52s) [2025-08-14 09:01:47.763906] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task use too much time(end=1755162107763906, start=1755162104248811, delta=3515095) [2025-08-14 09:01:47.763917] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=0] task scheduled too delay(scheduled_delay=3508428, expected_run_ts_us_=1755162104255487, ObClockGenerator::getRealClock()=1755162107763916, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:44.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:47.763933] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:44.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:47.763948] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=3174632, expected_run_ts_us_=1755162104589316, ObClockGenerator::getRealClock()=1755162107763948, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:44.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:47.763959] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:44.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:47.763980] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=0] task scheduled too delay(scheduled_delay=3059682, expected_run_ts_us_=1755162104704297, ObClockGenerator::getRealClock()=1755162107763980, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:44.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:47.763990] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:44.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:47.764018] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] task scheduled too delay(scheduled_delay=3037424, expected_run_ts_us_=1755162104726594, ObClockGenerator::getRealClock()=1755162107764018, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd110, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd3f0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:10.00s, expected_run_time:09:01:44.726, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:47.764029] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd110, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd3f0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:10.00s, expected_run_time:09:01:44.726, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:47.764145] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=348536682, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=251602, succ_cluster_heartbeat_count=1472, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161006924036004, v:0}, local_cluster_delta=1099834444, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:01:43.755091] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task use too much time(end=1755162103716542, start=1755162101754005, delta=1962537) [2025-08-14 09:01:47.765003] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=4003827][errcode=0] task scheduled too delay(scheduled_delay=5781471, expected_run_ts_us_=1755162101983531, ObClockGenerator::getRealClock()=1755162107765003, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:41.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:47.765025] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:41.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:47.765081] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=4020447, expected_run_ts_us_=1755162103744633, ObClockGenerator::getRealClock()=1755162107765081, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:43.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:47.765096] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:43.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:44.921754] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] EASY SLOW: thread: 7f9954779300, lastrun: 1755162102.763301 cost: 2.158451 loop:26467, slowcnt: 21 [2025-08-14 09:01:47.765379] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [15442][T1001_L0_G1][T1001][YB42AC4D0007-00063C496CD99C88-0-0] [lt=28] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.77.0.7:2882", child={server:"172.77.0.7:2882", region:"deault_region", register_time_us:1755162099867155, keepalive_ts:-1}, is_to_leader=true) [2025-08-14 09:01:47.754622] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:47.765864] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11225][errcode=0] epoll handler too much time :11250 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:47.765923] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=2847372][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb3fb8, timer:0x7f99333cb180, task:0x7f99333cb220, task_type:"N9oceanbase8observer18ObTableLoadService21ObClientTaskPurgeTaskE", scheduled_time:1755161984781007, pushed_time:0, delay:1000000}, running_task_token={this:0x7f9931bb3ed8, timer:0x7f99333cb180, task:0x7f99333cb208, task_type:"N9oceanbase8observer18ObTableLoadService25ObClientTaskAutoAbortTaskE", scheduled_time:1755161959782824, pushed_time:1755162045729074, delay:1000000}, total_delay=122984915, threashold=1000000, tenant_id=1001) [2025-08-14 09:01:47.765970] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=47][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb2498, timer:0x7f9934119ee0, task:0x7f9934405010, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr11TableGCTaskE", scheduled_time:1755162043738943, pushed_time:0, delay:20000}, running_task_token={this:0x7f9931bb2578, timer:0x7f9934119ee0, task:0x7f9934405028, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr17RefreshConfigTaskE", scheduled_time:1755161881725525, pushed_time:1755162043719350, delay:10000000}, total_delay=64027027, threashold=1000000, tenant_id=1001) [2025-08-14 09:01:47.765980] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb3b58, timer:0x7f99333cb180, task:0x7f99333cb1a8, task_type:"N9oceanbase8observer18ObTableLoadService17ObCheckTenantTaskE", scheduled_time:1755162046728894, pushed_time:0, delay:1000000}, running_task_token={this:0x7f9931bb3ed8, timer:0x7f99333cb180, task:0x7f99333cb208, task_type:"N9oceanbase8observer18ObTableLoadService25ObClientTaskAutoAbortTaskE", scheduled_time:1755161959782824, pushed_time:1755162045729074, delay:1000000}, total_delay=61037086, threashold=1000000, tenant_id=1001) [2025-08-14 09:01:47.766795] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=1996934] gc stale ls task succ [2025-08-14 09:01:45.774289] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:47.767563] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=1990239] temporary file module use no blocks [2025-08-14 09:01:47.767570] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:47.767581] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:47.767594] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623007213995, flush_io_finished_ret_:0, flush_io_finished_round_:26280, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:47.767663] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=68] get wrs ts(ls_id={id:1}, delta=358012059, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:47.767679] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:47.768937] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4007923][errcode=0] epoll handler too much time :4014020 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:47.768954] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:47.768965] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:47.770081] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:47.770109] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=11] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:47.771791] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=4961872][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:41.755|207=6.02s|210=0us|222=2us|total=6.02s) [2025-08-14 09:01:47.771812] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task use too much time(end=1755162107771811, start=1755162101755968, delta=6015843) [2025-08-14 09:01:44.735398] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=878685][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:43.855|207=880.02ms|210=0us|222=2us|total=3.92s) [2025-08-14 09:01:47.771921] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=3031920][errcode=-4389] timer task use too much time(end=1755162107771921, start=1755162103855374, delta=3916547) [2025-08-14 09:01:47.771930] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=5][errcode=0] task scheduled too delay(scheduled_delay=2978018, expected_run_ts_us_=1755162104793911, ObClockGenerator::getRealClock()=1755162107771929, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd070, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fd350, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:5.00s, expected_run_time:09:01:44.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:47.771946] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd070, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fd350, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:5.00s, expected_run_time:09:01:44.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:47.771957] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=2978020, expected_run_ts_us_=1755162104793936, ObClockGenerator::getRealClock()=1755162107771956, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:01:44.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:47.771966] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:01:44.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:47.740621] WDIAG eloop_delay_warn (debug.h:93) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=18640][errcode=0] PNIO [delay_warn] eloop handle events delay high: 2969021, malloc=0/0 write=0/0 read=0/1 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:47.772088] WDIAG delay_warn (debug.h:103) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=31383][errcode=0] PNIO [delay_warn] pktc_flush_cb delay high: 3000021, start_us=1755162104770751 [2025-08-14 09:01:47.744353] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=20600] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:47.772208] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27809] temporary file module use no blocks [2025-08-14 09:01:47.772217] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:42.238903] EDIAG [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4498668, owner: ObLog, click_count: 6, time dist:[FORMAT_END=22, ObMalloc#start=1, ObMalloc#end=1493066, ObMalloc#start=23, ObMalloc#end=2027565, FREE_END=977991], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b537806 0x1b4f816d 0x1b4fbf53 0x1b5014e0 0x1b5009d1 0x1b50387c 0x1b8905de 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:47.774013] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ab590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:47.750, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:47.774055] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891969e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:47.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:47.774073] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d80b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196a190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:47.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:47.774089] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196a010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:47.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:47.774104] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891992330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196d710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:47.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:47.774119] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739f110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:47.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:47.774143] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891971410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:47.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:47.774159] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f0f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:47.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:46.983769] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:46.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:47.775442] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=790474][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:46.983|207=791.67ms|210=0us|222=2us|total=791.69ms) [2025-08-14 09:01:47.725881] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [880][T1002_ArcSender][T1002][YB42AC4D0007-00063C495EB99C16-0-0] [lt=31] ObArchiveSender is running(thread_index=5) [2025-08-14 09:01:47.775463] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task use too much time(end=1755162107775462, start=1755162106983767, delta=791695) [2025-08-14 09:01:47.775920] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1001}, delta=363047679, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:47.775955] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=31] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:47.772230] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:47.776763] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=4526] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623009184083, flush_io_finished_ret_:0, flush_io_finished_round_:26254, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:47.777356] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=959834] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=558988315, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=253002, succ_cluster_heartbeat_count=2176, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161000606550000, v:0}, local_cluster_delta=1106120281, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:01:47.777457] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=44] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161101770923, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755161572801635) [2025-08-14 09:01:47.777484] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get_max_applied_scn(scn={val:1755161547738516001, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:50, palf_committed_end_lsn_:{lsn:5282737655}, palf_committed_end_scn_:{val:1755161547738516001, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161547738516001, v:0}}) [2025-08-14 09:01:47.777506] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] get_min_unreplayed_log_info(lsn={lsn:5282737655}, scn={val:1755100947825033001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161624848121, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:13063726, ret_code_:0}}, next_to_submit_lsn_:{lsn:5282737655}, next_to_submit_scn_:{val:1755100947825033001, v:0}, base_lsn_:{lsn:4831543296}, base_scn_:{val:1755063445733337001, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7605000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:5282737655}, end_lsn:{lsn:5282737655}, read_buf:{buf_len_:3747840, buf_:0x7f98f7605000}, block_size:67104768, log_storage_:0x7f993e529bf0, log_storage_:{log_tail:{lsn:5282737655}, readable_log_tail:{lsn:5282737655}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:5234171904}, min_scn:{val:1755152244802535004, v:0}, curr_block_id:78, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:206, min_block_id:42, max_block_id:79, curr_writable_block_id:78}, logical_block_size_:67104768, curr_block_writable_size_:18539017, block_header_serialize_buf_:0x7f993e52a8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:85, read_io_size:301367296, read_disk_cost_ts:2364604, start_lsn:{lsn:4831543296}}}}}}) [2025-08-14 09:01:47.777735] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] get_max_applied_scn(scn={val:1755161749741193000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:50, palf_committed_end_lsn_:{lsn:1604090090}, palf_committed_end_scn_:{val:1755161749741193000, v:0}, last_check_scn_:{val:1755161749741193001, v:0}, max_applied_cb_scn_:{val:1755161749741193000, v:0}}) [2025-08-14 09:01:47.777766] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25] get wrs ts(ls_id={id:1}, delta=358034310, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:47.777805] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=36] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:47.778121] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=86] get wrs ts(ls_id={id:1}, delta=560036987, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:47.778146] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:47.781068] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=560036987, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:47.781091] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:47.781270] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=23] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:47.781310] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=28] temporary file module use no blocks [2025-08-14 09:01:47.781326] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=15] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:47.781337] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:47.781354] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=16] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623009218213, flush_io_finished_ret_:0, flush_io_finished_round_:26289, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:47.781418] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=15333][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:47.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:47.781628] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1001}, delta=363053348, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:47.781649] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:46.748227] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [663][T1001_StandbySc][T1001][YB42AC4D0007-00063C495C7A02EB-0-0] [lt=27][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1031949, , BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:41.789297] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4947889, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:47.781801] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=4947889][errcode=0] epoll handler too much time :5992511 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:47.781827] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:47.781849] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:41.779604] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=16][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4965883, , BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:47.781924] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=4965883] gc stale ls task succ [2025-08-14 09:01:46.771493] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20617][errcode=0] epoll handler too much time :20655 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:46.771493] EDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20617][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1008980, , BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:47.782032] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1008980] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:47.782052] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:40.753706] EDIAG [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 6997520, owner: ObLog, click_count: 5, time dist:[FORMAT_END=4954338, ObMalloc#start=2, ObMalloc#end=1041404, ALLOC_END=1001775, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0xfad91f4 0x99f635e 0xfad88de 0xfad867a 0x1f9ebaba 0x97c8ff1 0x1f981a55 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:47.784076] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b9c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:47.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:47.784116] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72751b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b5210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:47.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:47.784133] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d31b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72eb890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:47.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:47.784158] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73959b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f1090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:47.773, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:47.794147] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7283210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:47.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:47.794181] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72856b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7289290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:47.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:47.794197] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7efb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:47.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:47.801543] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:47.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:47.804211] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919920b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196d590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:47.790, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:47.840458] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:47.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:47.854534] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72eb710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:47.840, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:47.854573] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d54b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ecd90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:47.846, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:47.854592] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ecc10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:47.848, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:47.866056] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb4958, timer:0x7f9966ee3f40, task:0x7f99333d35a8, task_type:"N9oceanbase3omt11ObTenantSrs27TenantSrsUpdatePeriodicTaskE", scheduled_time:1755161742719922, pushed_time:0, delay:0}, running_task_token={this:0x7f9931bb4178, timer:0x7f9966ee3f40, task:0x7f99333d2fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161723481487, pushed_time:1755161737766048, delay:900000000}, total_delay=365146133, threashold=1000000, tenant_id=1001) [2025-08-14 09:01:47.866087] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb4338, timer:0x7f9966ee3f40, task:0x7f99333da1b0, task_type:"N9oceanbase5table20ObTableObjectPoolMgr26ObTableSessEliminationTaskE", scheduled_time:1755161742732119, pushed_time:0, delay:5000000}, running_task_token={this:0x7f9931bb4178, timer:0x7f9966ee3f40, task:0x7f99333d2fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161723481487, pushed_time:1755161737766048, delay:900000000}, total_delay=365133968, threashold=1000000, tenant_id=1001) [2025-08-14 09:01:47.866098] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb4418, timer:0x7f9966ee3f40, task:0x7f99333da1c8, task_type:"N9oceanbase5table20ObTableObjectPoolMgr27ObTableSessSysVarUpdateTaskE", scheduled_time:1755161742735029, pushed_time:0, delay:5000000}, running_task_token={this:0x7f9931bb4178, timer:0x7f9966ee3f40, task:0x7f99333d2fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161723481487, pushed_time:1755161737766048, delay:900000000}, total_delay=365131069, threashold=1000000, tenant_id=1001) [2025-08-14 09:01:47.866113] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb44f8, timer:0x7f9966ee3f40, task:0x7f99333da1e0, task_type:"N9oceanbase5table20ObTableObjectPoolMgr28ObTableReqResEliminationTaskE", scheduled_time:1755161742735074, pushed_time:0, delay:5000000}, running_task_token={this:0x7f9931bb4178, timer:0x7f9966ee3f40, task:0x7f99333d2fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161723481487, pushed_time:1755161737766048, delay:900000000}, total_delay=365131039, threashold=1000000, tenant_id=1001) [2025-08-14 09:01:47.871945] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:01:47.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:47.875512] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:47.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:47.877786] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] get wrs ts(ls_id={id:1}, delta=358135555, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:47.877812] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:47.878424] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] get wrs ts(ls_id={id:1001}, delta=363149982, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:47.878444] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:47.878871] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=560139290, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:47.878897] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:47.940480] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:47.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:47.951011] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=139] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:01:47.975573] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:47.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:47.978828] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=560239464, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:47.978859] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:47.995471] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:47.989, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:47.996580] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:47.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.015563] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:48.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.022884] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:48.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.040521] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:47.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.075602] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:47.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.096064] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:48.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.140585] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:48.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.175663] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:48.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.216689] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:48.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.222357] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [571][T1_ArcSender][T1][YB42AC4D0007-00063C495BA99C16-0-0] [lt=16][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:01:48.240647] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:48.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.246868] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:48.239, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.247868] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:48.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.266411] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb2e38, timer:0x7f9966a57e50, task:0x7f993335ec40, task_type:"N9oceanbase7obmysql20ObConstructQueueTaskE", scheduled_time:1755162047260468, pushed_time:0, delay:500000}, running_task_token={this:0x7f9931bb2d58, timer:0x7f9966a57e50, task:0x7f993335ebf0, task_type:"N9oceanbase7obmysql15ObEliminateTaskE", scheduled_time:1755161984280870, pushed_time:1755162046760808, delay:500000}, total_delay=61005942, threashold=1000000, tenant_id=1001) [2025-08-14 09:01:48.267006] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:48.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.274485] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:48.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.275737] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:48.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.304636] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:48.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.340681] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:48.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.375806] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:48.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.440741] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:48.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.474091] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=296][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fa190, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa470, time_wheel:0x7f99341cdbf0, thread_pool:0x7f99341cd3d0, is_running:1, time_interval:2.00s, expected_run_time:09:01:48.334, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.475869] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:48.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.498062] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:48.489, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.518171] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:48.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.525926] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:48.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.540801] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:48.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.574227] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=325][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fe410, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe6f0, time_wheel:0x7f9922df9d40, thread_pool:0x7f9922df9520, is_running:1, time_interval:2.00s, expected_run_time:09:01:48.432, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.575950] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:48.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.598616] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:48.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.640833] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:48.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.675995] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:48.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.679225] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=48005][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:01:48.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.719416] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:48.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.720280] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:48.721185] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:48.736467] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:48.739524] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc190, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc470, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:10.00s, expected_run_time:09:01:48.720, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.739944] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ea390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:48.722, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:48.740251] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=13521] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:01:48.740888] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:48.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.742255] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:48.742256] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:48.742383] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=17] PNIO [ratelimit] time: 1755162108742380, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006026, add_bytes: 0 [2025-08-14 09:01:47.971939] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:47.855, task_priority:1, with_handle_protected:1}) [567]OB_MALLOC COST TOO MUCH TIME, cost_time=668257, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:01:45.730313] EDIAG [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3014782, owner: ObLog, click_count: 3, time dist:[FORMAT_END=3014777, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1bdd73e2 0x1bdd6e36 0x986bc6f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:47.741357] EDIAG [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=14610][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1007567, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1007555, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1bdd73e2 0x1bdd6e36 0x986bc6f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:41.755944] EDIAG [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4981186, owner: ObLog, click_count: 10, time dist:[FORMAT_END=14, ObMalloc#start=0, ObMalloc#end=1972897, ObMalloc#start=37, ObMalloc#end=2029484, FREE_END=978754, ALLOC_BLOCK#start=980660, ALLOC_BLOCK#end=92, ALLOC_BLOCK#start=10195, ALLOC_BLOCK#end=93], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x18257d0c 0x1823b885 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:48.257924] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:48.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.743966] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=485305][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:48.257|207=486.04ms|210=0us|222=1us|total=486.06ms) [2025-08-14 09:01:48.743985] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4389] timer task use too much time(end=1755162108743984, start=1755162108257922, delta=486062) [2025-08-14 09:01:47.805676] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=1969625] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:01:47.785757] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=2975398] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:01:48.744009] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=260476, expected_run_ts_us_=1755162108483531, ObClockGenerator::getRealClock()=1755162108744008, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:48.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.744024] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:48.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.744040] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8030, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8310, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:10.00s, expected_run_time:09:01:48.730, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.726706] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=4964173] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:01:48.745205] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:47.815835] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=2908952] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:01:48.746360] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:48.743856] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=770748][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:47.971|207=771.92ms|210=0us|222=2us|total=774.52ms) [2025-08-14 09:01:48.746464] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=2603][errcode=-4389] timer task use too much time(end=1755162108746464, start=1755162107971937, delta=774527) [2025-08-14 09:01:48.746688] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [1000][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F01-0-0] [lt=10][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1002, svr="172.77.0.7:2882", version={val:1755161758221798000, v:0}, valid_part_count=0, total_part_count=2, generate_timestamp=1755162099836883) [2025-08-14 09:01:48.747378] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=4953919][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:01:48.747420] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:01:48.747448] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [657][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:0, time_interval:100.00ms, expected_run_time:09:01:48.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.747471] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [657][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:01:48.747|total=5.97s) [2025-08-14 09:01:48.747538] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:48.748192] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=561005266, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:48.748215] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:48.748802] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29] get wrs ts(ls_id={id:1}, delta=561005266, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:48.748825] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:48.748965] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=4498668][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162102238897], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:48.748992] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162102238897], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:48.749015] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162102238897]}) [2025-08-14 09:01:48.749445] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=13] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:45.730313] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=5] finish update min start scn(max_decided_scn={val:1755161547738516001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1755161493768447002, v:0}, keep_alive_scn_:{val:1755161493768447002, v:0}, update_ts_:1755162104771926}) [2025-08-14 09:01:47.741357] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=14610] finish update min start scn(max_decided_scn={val:1755161719775179002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1755161287778691002, v:0}, keep_alive_scn_:{val:1755161287778691002, v:0}, update_ts_:1755162107737696}) [2025-08-14 09:01:48.752128] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=31] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:48.752169] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:47.772129] WDIAG [RPC] create (ob_poc_rpc_server.cpp:140) [316][pnio1][T0][YB42AC4D0007-00063C49612B8F08-0-0] [lt=13][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1815, fly_ts=7134723, send_timestamp=1755162100637404, peer="172.77.0.7:49788", sz=265) [2025-08-14 09:01:48.752655] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:48.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.753505] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ab410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:48.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:48.753541] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891826cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891918190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:48.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:48.753556] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b5390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:48.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:48.753571] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919845b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891971890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:48.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:48.747436] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:00:41.841, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=08:59:41.841], idx=471, next_allow_print_ts="09:01:11.841") [2025-08-14 09:01:48.753633] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6179][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:01:48.753650] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:01:48.753670] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:01:48.753687] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:01:48.753702] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:01:48.753717] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:01:48.757795] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:48.757848] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:48.757921] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:45.764714] EDIAG [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=990885][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2988745, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2988737, ObMalloc#start=0, ObMalloc#end=2, ALLOC_END=0, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1fce4e91 0x1fce455d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:45.764714] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=990885] blacklist_loop exec finished(cost_time=26, is_enabled=true, send_cnt=0) [2025-08-14 09:01:48.759942] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:48.760225] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=7] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:38.779968] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:662) [740][T1001_Occam][T1001][YB42AC4D0007-00063C4966399E1E-0-0] [lt=12][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f98ada75ea0, tx_id:{txid:0}, state:1, addr:"172.77.0.7:2882", tenant_id:1001, session_id:3221590960, assoc_session_id:3221590960, client_sid:3221590960, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1755162094747985, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180067074, seq_base:1755162093745433, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}) [2025-08-14 09:01:48.760463] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:48.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.762433] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:48.762712] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:48.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:44.746700] EDIAG [RS] get_valid_sts_after (ob_tenant_info_loader.cpp:454) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=812007][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4012030, owner: ObLog, click_count: 5, time dist:[FORMAT_END=4012022, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1001d942 0x996d843 0x996d15a 0x996cfd8 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:47.726576] EDIAG [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1038475, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1038471, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1af3d850 0x1af3c294 0x20071394 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:47.727939] EDIAG [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1038745, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1038739, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1af3d850 0x1af3c294 0x20071394 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:47.726590] EDIAG [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1040140, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1040136, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1af3d850 0x1af3c294 0x20071394 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:48.763742] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ea210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:48.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:48.763782] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ea090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:48.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:48.763798] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196a910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:48.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:48.763819] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728a910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:48.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:48.763832] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eacb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7ef290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:48.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:48.753730] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:01:48.764069] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10320][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:00:52.998, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:22.998], idx=771, next_allow_print_ts="09:01:22.998") [2025-08-14 09:01:48.764086] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:01:48.764109] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=08:59:57.942, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:27.942], idx=868, next_allow_print_ts="09:00:27.942") [2025-08-14 09:01:48.764289] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:01:48.764304] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:01:48.764328] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:01:48.749034] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=56, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:01:48.764470] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15410] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162102238897]}) [2025-08-14 09:01:48.764809] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EE93-0-0] [lt=12][errcode=0] cost too much time:mds_tenant_service.cpp:int oceanbase::storage::mds::ObTenantMdsTimer::try_recycle_mds_table_(oceanbase::storage::ObTablet &, const share::SCN &), (this=|threshold=5.00ms|start at 09:01:48.745|375=0us|total=19.48ms) [2025-08-14 09:01:48.764838] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EE93-0-0] [lt=20][errcode=0] cost too much time:mds_tenant_service.cpp:int oceanbase::storage::mds::ObTenantMdsTimer::process_with_tablet_(oceanbase::storage::ObTablet &), (this=|threshold=10.00ms|start at 09:01:48.745|315=0us|317=17us|319=19.49ms|total=19.52ms) [2025-08-14 09:01:48.765640] WDIAG stat_cleanup (debug.c:35) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=978598][errcode=0] PNIO [delay_warn] cost too much time: 994007us, procedure: pn_pkts_handle_func [2025-08-14 09:01:48.765683] WDIAG delay_warn (debug.h:103) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] PNIO [delay_warn] pktc_flush_cb delay high: 994007, start_us=1755162107770772 [2025-08-14 09:01:48.765695] WDIAG eloop_delay_warn (debug.h:93) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] PNIO [delay_warn] eloop handle events delay high: 994007, malloc=0/0 write=0/1 read=0/2 server_process=994007/1 client_cb=0/0 [2025-08-14 09:01:48.765722] WDIAG [RPC] create (ob_poc_rpc_server.cpp:140) [316][pnio1][T0][YB42AC4D0007-00063C49607B920C-0-0] [lt=9][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1815, fly_ts=7644128, send_timestamp=1755162101121593, peer="172.77.0.7:49788", sz=265) [2025-08-14 09:01:48.766570] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:355) [835][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=25] finish check and clear obj lock(ls->get_ls_id()={id:1001}) [2025-08-14 09:01:47.726576] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] [COMPACTION DAG STATUS] (start_time=1755162095731474, end_time=1755162107720127, dag_status={}) [2025-08-14 09:01:48.767726] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=18] gc stale ls task succ [2025-08-14 09:01:48.767756] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:48.767777] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=17] temporary file module use no blocks [2025-08-14 09:01:48.767786] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:48.767800] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:48.767814] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623010207389, flush_io_finished_ret_:0, flush_io_finished_round_:26281, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:47.727939] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] [COMPACTION DAG STATUS] (start_time=1755162093754721, end_time=1755162107720133, dag_status={}) [2025-08-14 09:01:47.726590] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=15] [COMPACTION DAG STATUS] (start_time=1755162096731586, end_time=1755162107720054, dag_status={}) [2025-08-14 09:01:48.770761] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:48.770874] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:48.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.774994] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891968090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:48.767, task_priority:1, with_handle_protected:0}) [857]OB_MALLOC COST TOO MUCH TIME, cost_time=797381, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:01:48.776918] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:48.776950] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=21] temporary file module use no blocks [2025-08-14 09:01:48.776966] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=15] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:48.776979] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:48.776992] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623010216561, flush_io_finished_ret_:0, flush_io_finished_round_:26255, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:48.777123] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get_min_unreplayed_log_info(lsn={lsn:1604088821}, scn={val:1755100948296325001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873479, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:12492179, ret_code_:0}}, next_to_submit_lsn_:{lsn:1604088821}, next_to_submit_scn_:{val:1755100948296325001, v:0}, base_lsn_:{lsn:1409200128}, base_scn_:{val:1753840802540914000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:1604088821}, end_lsn:{lsn:1604088821}, read_buf:{buf_len_:3747840, buf_:0x7f98f6a05000}, block_size:67104768, log_storage_:0x7f990bfebbf0, log_storage_:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:42, read_io_size:147709952, read_disk_cost_ts:1061450, start_lsn:{lsn:1409200128}}}}}}) [2025-08-14 09:01:48.777209] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=1935238] current monitor number(seq_=-1) [2025-08-14 09:01:48.777228] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=102] get wrs ts(ls_id={id:1}, delta=359021563, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:48.777253] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:48.777276] INFO [STORAGE] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:123) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] wead read timestamp is too old(timestamp={val:1755161749741193000, v:0}, gts_scn={val:1755161762721798002, v:0}, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1755161749741193000, v:0}}) [2025-08-14 09:01:48.777812] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=1017192] kill and revert session(conn.sessid_=3221590961, proxy_sessid=0, ret=0) [2025-08-14 09:01:48.766595] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [835][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99319fe030, caller:ob_table_lock_service.cpp:start:276, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE6$_1273, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe310, time_wheel:0x7f99535f3990, thread_pool:0x7f99535f3170, is_running:0, time_interval:10.00s, expected_run_time:09:01:56.426, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.777866] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1001}, delta=364034932, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:48.777866] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [835][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=11251][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:01:48.777|total=2.22s) [2025-08-14 09:01:48.777881] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:48.777900] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4981186] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3790) [2025-08-14 09:01:48.777913] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=359021563, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:48.777916] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:01:48.777925] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:48.745371] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [719][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571f9d70, caller:ob_table_lock_service.cpp:start:276, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE6$_1273, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa090, time_wheel:0x7f99375f7d10, thread_pool:0x7f99375f74f0, is_running:0, time_interval:10.00s, expected_run_time:09:01:52.332, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.779025] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [719][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=33601][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:01:48.779|total=6.30s) [2025-08-14 09:01:48.779394] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=4963250] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:48.779666] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] get wrs ts(ls_id={id:1001}, delta=364034932, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:48.779696] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:48.779740] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] get wrs ts(ls_id={id:1}, delta=359021563, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:48.779757] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:48.781275] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1001}, delta=364034932, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:48.781311] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=34] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:48.782048] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=12] gc stale ls task succ [2025-08-14 09:01:48.782514] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:48.782932] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:48.785073] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ad690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:48.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:48.785123] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196a490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:48.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:48.785136] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196a310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:48.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:48.785173] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7282610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:48.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:48.795122] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72716b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919a4910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:48.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:48.795167] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a0310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:48.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:48.795180] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918100b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a0190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:48.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:48.795191] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f1990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:48.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:48.801219] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:48.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.805161] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891825510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:48.793, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:48.805196] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180fe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a0010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:48.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:48.811281] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40b7f0, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40bad0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:10.00s, expected_run_time:09:01:48.802, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.815225] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a1990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:48.805, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:48.825280] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73927b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:48.812, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:48.825329] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7399090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:48.815, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:48.825340] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7398f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:48.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:48.835339] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73954b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f1210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:48.827, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:48.840948] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:48.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.846529] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:48.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.848946] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=561108722, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:48.848987] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:48.867774] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb3d18, timer:0x7f99333cb180, task:0x7f99333cb1d8, task_type:"N9oceanbase8observer18ObTableLoadService8ObGCTaskE", scheduled_time:1755161987758103, pushed_time:0, delay:30000000}, running_task_token={this:0x7f9931bb3ed8, timer:0x7f99333cb180, task:0x7f99333cb208, task_type:"N9oceanbase8observer18ObTableLoadService25ObClientTaskAutoAbortTaskE", scheduled_time:1755161959782824, pushed_time:1755162045729074, delay:1000000}, total_delay=121109670, threashold=1000000, tenant_id=1001) [2025-08-14 09:01:48.878799] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:48.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.879500] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:01:48.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.879832] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=359138544, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:48.879864] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=30] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:48.880489] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1001}, delta=364151913, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:48.880510] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:48.885661] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891970510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:48.873, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:48.941009] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:48.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.946591] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:48.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.948999] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27] get wrs ts(ls_id={id:1}, delta=561209703, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:48.949024] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:48.960270] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=23] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:01:48.978874] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:48.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.979911] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=359237760, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:48.979931] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:48.980553] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1001}, delta=364252207, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:48.980574] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:48.992436] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:48.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.996324] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:48.989, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.016458] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:49.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.022583] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:49.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.041045] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:48.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.046608] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:48.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.049053] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=561308996, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:49.049081] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:49.078936] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:48.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.079984] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=359338508, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:49.080020] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=34] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:49.080662] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1001}, delta=364351877, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:49.080684] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:49.096980] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:49.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.141109] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:49.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.146666] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:49.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.149115] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=561409991, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:49.149153] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:49.178998] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:49.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.180048] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=359437853, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:49.180072] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:49.180699] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1001}, delta=364452447, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:49.180720] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:49.217746] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:49.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.241171] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:49.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.243915] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:49.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.246732] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:49.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.247939] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:49.239, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.248535] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161101770923, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755161572801635) [2025-08-14 09:01:49.249143] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=34] get wrs ts(ls_id={id:1}, delta=561510502, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:49.249188] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=41] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:49.253939] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:49.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.257850] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7ee390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:49.248, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:49.264499] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:01:49.264541] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:01:49.264565] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:01:49.264578] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:00:41.841, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=08:59:41.841], idx=471, next_allow_print_ts="09:01:11.841") [2025-08-14 09:01:49.264605] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:01:49.264620] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:01:49.264638] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:01:49.264665] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:01:49.264678] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:01:49.264695] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:01:49.264707] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:00:52.998, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:22.998], idx=771, next_allow_print_ts="09:01:22.998") [2025-08-14 09:01:49.264739] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=08:59:57.942, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:27.942], idx=868, next_allow_print_ts="09:00:27.942") [2025-08-14 09:01:49.264912] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:01:49.264927] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:01:49.264940] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:01:49.268033] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:49.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.274084] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:49.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.279060] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:49.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.280120] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=359538385, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:49.280143] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:49.280782] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28] get wrs ts(ls_id={id:1001}, delta=364551754, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:49.280799] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:49.304273] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:49.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.327396] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=676495][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a4e38, timer:0x7f9967161f10, task:0x7f994b3f4c40, task_type:"N9oceanbase7obmysql20ObConstructQueueTaskE", scheduled_time:1755162048238296, pushed_time:0, delay:500000}, running_task_token={this:0x7f994e1a4d58, timer:0x7f9967161f10, task:0x7f994b3f4bf0, task_type:"N9oceanbase7obmysql15ObEliminateTaskE", scheduled_time:1755162042245325, pushed_time:1755162047738501, delay:500000}, total_delay=61089098, threashold=1000000, tenant_id=1) [2025-08-14 09:01:49.341213] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:49.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.346801] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:49.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.349180] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] get wrs ts(ls_id={id:1}, delta=561609937, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:49.349212] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:49.379121] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:49.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.380109] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=359638899, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:49.380130] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:49.380730] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25] get wrs ts(ls_id={id:1001}, delta=364652268, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:49.380747] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:49.441242] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=52][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:49.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.446876] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:49.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.449298] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=561709127, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:49.449333] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=32] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:49.479202] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:49.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.480192] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=359738896, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:49.480213] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:49.480802] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1001}, delta=364752265, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:49.480816] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:49.495480] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:49.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.499318] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:49.489, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.519420] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:49.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.525589] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:49.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.541257] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:49.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.546914] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=63][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:49.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.549365] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=561810332, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:49.549388] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:49.579278] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=50][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:49.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.580275] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=359838234, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:49.580297] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:49.580888] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1001}, delta=364851603, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:49.580907] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:49.599932] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:49.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.641293] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:49.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.646946] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:49.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.649413] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=561910071, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:49.649437] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:49.679318] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:49.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.679770] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:01:49.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.680004] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:1, time_interval:3.00s, expected_run_time:09:01:49.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.680271] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=359938982, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:49.680290] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:49.680860] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1001}, delta=364952351, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:49.680877] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:49.710858] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:49.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:45.728539] EDIAG [ELECT] print_debug_ts_if_reach_warn_threshold (election_message.cpp:63) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=969129][errcode=-4016] LOGGER COST TOO MUCH TIME, cost: 3983432, owner: ObLog, click_count: 5, time dist:[FORMAT_END=3983421, ObMalloc#start=2, ObMalloc#end=4, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0xf9dd1cf 0x9952a1e 0xfcf4c7f 0xfcf0cf1 0xfc721d0 0xfd2bbd0 0xf9d16a8 0xf9d4150 0xfa09ca2 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:41.791034] EDIAG [RS] skip_check_member_list_change_ (ob_ls_recovery_stat_handler.cpp:109) [777][T1002_TeRec][T1002][YB42AC4D0007-00063C495DDA000E-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1982341, owner: ObLog, click_count: 7, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=1939778, ALLOC_END=42552, APPEND_END=1, ALLOC_BLOCK#start=6, ALLOC_BLOCK#end=25], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x1067fb2c 0x1067ecc2 0x1067e03d 0x1002f2a3 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [997]OB_MALLOC COST TOO MUCH TIME, cost_time=974058, tenant_id=1002, label=SqlExecutor, ctx_id=0, prio=0, size=245760, [2025-08-14 09:01:48.768553] INFO [COMMON] is_local_clog_not_isolated (ob_io_define.cpp:1278) [511][T1_TimerWK2_SSTableGC][T1][YB42AC4D0007-00063C495D6A02D9-0-0] [lt=28] clog_not_isolated(clog_not_isolated=false, clog_io_isolation_mode=1, group_key={group_id_:0, mode_:2}, func_type=0) [2025-08-14 09:01:49.720712] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EE93-0-0] [lt=11][errcode=0] cost too much time:mds_tenant_service.cpp:int oceanbase::storage::mds::ObTenantMdsTimer::get_tablet_oldest_scn_(oceanbase::storage::ObTablet &, share::SCN &), (this=|threshold=5.00ms|start at 09:01:48.766|344=0us|total=954.50ms) [2025-08-14 09:01:49.720739] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:49.720740] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EE93-0-0] [lt=20][errcode=0] cost too much time:mds_tenant_service.cpp:int oceanbase::storage::mds::ObTenantMdsTimer::process_with_tablet_(oceanbase::storage::ObTablet &), (this=|threshold=10.00ms|start at 09:01:48.766|315=0us|317=954.50ms|319=4us|total=954.52ms) [2025-08-14 09:01:49.721839] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:49.723745] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=13] [sql nio session](*s={this:0x7f98f733f030, session_id:3221589417, trace_id:YB42AC4D0007-00063C496359B268-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:252}, err:0, last_decode_time:1755161813752329, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1778, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:01:49.723791] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=46] [sql nio session](*s={this:0x7f98f733e030, session_id:3221589980, trace_id:YB42AC4D0007-00063C496359B261-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:251}, err:0, last_decode_time:1755161748778070, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1681, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:01:49.723806] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=14] [sql nio session](*s={this:0x7f98f7332030, session_id:3221590132, trace_id:YB42AC4D0007-00063C496359B265-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:266}, err:0, last_decode_time:1755161768783802, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:2125, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:01:49.724214] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EE93-0-0] [lt=10] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-08-14 09:01:49.725307] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EE93-0-0] [lt=1081][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571fc8b0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcb90, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:0, time_interval:3.00s, expected_run_time:09:01:49.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.725334] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EE93-0-0] [lt=25][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:01:49.725|total=2.89s) [2025-08-14 09:01:49.736499] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:49.738335] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:49.738368] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:227) [1031][PxTargetMgr0][T1001][Y0-0000000000000000-0-0] [lt=25] refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3791) [2025-08-14 09:01:49.738377] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:01:49.738600] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=6] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:49.738621] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=21] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3792) [2025-08-14 09:01:49.738632] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:01:49.740328] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=21] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:01:49.741367] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:49.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.742215] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:49.742258] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:49.742499] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7270030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919a3b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:49.735, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:49.742531] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ecb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ac310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:49.739, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:49.742542] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7353730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7396990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:49.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:49.742553] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7efd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:49.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:49.743158] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:49.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.745162] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=18378] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:01:49.745196] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:49.746379] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:49.746382] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=41] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:49.746804] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:49.747015] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:49.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.748538] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:49.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.749462] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:49.749541] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=13] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:49.752144] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:49.752179] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:49.752572] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ea30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ac190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:49.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:49.758609] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:49.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.764899] EDIAG [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=4018533][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1004453, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1004448, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x123708d6 0x1236ebc0 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [321]OB_MALLOC COST TOO MUCH TIME, cost_time=989512, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=989503], seq:[0] [2025-08-14 09:01:49.762561] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:49.762633] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919afbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b9810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:49.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:49.762655] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919854b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891972190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:49.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:49.762664] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891972010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:49.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:49.762683] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891971e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:49.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:49.762692] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d45b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ec490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:49.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:49.763263] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:49.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.765122] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:01:49.765159] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:01:49.765176] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:01:49.765186] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:00:41.841, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=08:59:41.841], idx=471, next_allow_print_ts="09:01:11.841") [2025-08-14 09:01:49.765208] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:01:49.765217] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:01:49.765242] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:01:49.765253] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:01:49.765268] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:01:49.765280] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:01:49.765288] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:00:52.998, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:22.998], idx=771, next_allow_print_ts="09:01:22.998") [2025-08-14 09:01:49.765311] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=08:59:57.942, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:27.942], idx=868, next_allow_print_ts="09:00:27.942") [2025-08-14 09:01:49.765479] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:01:49.765492] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:01:49.765504] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:01:49.768329] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=15] gc stale ls task succ [2025-08-14 09:01:49.768360] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:49.768381] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=15] temporary file module use no blocks [2025-08-14 09:01:49.768388] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:49.768399] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:49.768411] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623011207609, flush_io_finished_ret_:0, flush_io_finished_round_:26282, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:49.770022] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [556][T1_OBJLockGC][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fafd0, caller:ob_table_lock_service.cpp:start:276, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE6$_1273, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fb2b0, time_wheel:0x7f994b19db50, thread_pool:0x7f994b19d330, is_running:1, time_interval:10.00s, expected_run_time:09:01:49.696, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.764899] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=4018533] snapshot_traffic_info once [2025-08-14 09:01:49.771447] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:49.772732] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72747b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b4c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:49.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:49.772759] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7280510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:49.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:49.778951] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:49.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.779405] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:49.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.782116] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=14] gc stale ls task succ [2025-08-14 09:01:49.782309] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:49.782324] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] temporary file module use no blocks [2025-08-14 09:01:49.782329] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=5] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:49.782339] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:49.782350] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623011221217, flush_io_finished_ret_:0, flush_io_finished_round_:26291, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:49.782788] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891825390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:49.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:49.782809] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ad510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:49.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:49.782820] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891969d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:49.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:49.782830] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:49.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:49.783990] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:49.784248] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [412][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=1040894] [sql nio session](*s={this:0x7f98f733a030, session_id:3221590255, trace_id:YB42AC4D0007-00063C49651A3232-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:257}, err:0, last_decode_time:1755161952736711, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1716, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:01:49.792858] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b5c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:49.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:49.792891] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ec310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:49.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:49.809090] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:49.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.812981] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919836b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891970f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:49.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:49.828322] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=33][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a5ed8, timer:0x7f994b3feee0, task:0x7f994b3fef68, task_type:"N9oceanbase8observer18ObTableLoadService25ObClientTaskAutoAbortTaskE", scheduled_time:1755162048762959, pushed_time:0, delay:1000000}, running_task_token={this:0x7f994e1a5c38, timer:0x7f994b3feee0, task:0x7f994b3fef20, task_type:"N9oceanbase8observer18ObTableLoadService15ObHeartBeatTaskE", scheduled_time:1755162023721549, pushed_time:1755162047763044, delay:10000000}, total_delay=61065362, threashold=1000000, tenant_id=1) [2025-08-14 09:01:49.841431] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:49.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.843163] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891970e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:49.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:49.843186] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72eb590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:49.831, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:49.847079] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:49.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.866723] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=2839152][errcode=0] EASY SLOW: thread: 7f99544fb300, lastrun: 1755162109.750577 cost: 0.116143 loop:26475, slowcnt: 1 [2025-08-14 09:01:49.873334] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7280390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:49.862, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:49.879456] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:49.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.880144] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd070, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fd350, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:5.00s, expected_run_time:09:01:49.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.880173] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:01:49.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.941493] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:49.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.947146] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:49.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.979500] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:49.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.994516] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:49.989, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.014641] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:50.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.020262] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:50.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.041554] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:49.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.047219] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:49.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.079530] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:49.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.095159] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:50.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.141595] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:50.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.147298] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:50.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.180011] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:50.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.215823] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:50.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.241641] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:50.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.245927] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:50.239, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.247324] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:50.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.265676] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:01:50.265708] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:01:50.265724] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:01:50.265734] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:00:41.841, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=08:59:41.841], idx=471, next_allow_print_ts="09:01:11.841") [2025-08-14 09:01:50.265748] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:01:50.265757] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:01:50.265772] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:01:50.265782] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:01:50.265794] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:01:50.265804] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:01:50.265813] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:00:52.998, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:22.998], idx=771, next_allow_print_ts="09:01:22.998") [2025-08-14 09:01:50.265838] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=08:59:57.942, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:27.942], idx=868, next_allow_print_ts="09:00:27.942") [2025-08-14 09:01:50.266011] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:01:50.266023] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:01:50.266014] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:50.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.266033] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:01:50.271511] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:50.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.280076] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:50.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.301638] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:50.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.341709] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:50.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.347387] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:50.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.380097] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:50.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.441757] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:50.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.447451] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:50.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.480165] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:50.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.497405] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:50.489, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.522869] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:50.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.541802] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:50.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.547486] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:50.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.641866] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:50.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.647537] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:50.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:49.778274] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=31] current monitor number(seq_=-1) [2025-08-14 09:01:50.738682] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119812272, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-08-14 09:01:50.738727] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] get wrs ts(ls_id={id:1}, delta=360997303, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:50.738742] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:50.738767] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get_max_applied_scn(scn={val:1755161744727824001, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:49, palf_committed_end_lsn_:{lsn:7323686009}, palf_committed_end_scn_:{val:1755161747742105001, v:0}, last_check_scn_:{val:1755161749741193000, v:0}, max_applied_cb_scn_:{val:1755161744727824001, v:0}}) [2025-08-14 09:01:50.738794] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] get_min_unreplayed_log_info(lsn={lsn:7323685240}, scn={val:1755100946796590002, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873485, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:8946764, ret_code_:0}}, next_to_submit_lsn_:{lsn:7323685240}, next_to_submit_scn_:{val:1755100946796590002, v:0}, base_lsn_:{lsn:7180210176}, base_scn_:{val:1753840803166611000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:7323685240}, end_lsn:{lsn:7323685240}, read_buf:{buf_len_:3747840, buf_:0x7f98f7a05000}, block_size:67104768, log_storage_:0x7f9901131bf0, log_storage_:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:28, read_io_size:98508800, read_disk_cost_ts:645339, start_lsn:{lsn:7180210176}}}}}}) [2025-08-14 09:01:49.868472] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=47][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb2f18, timer:0x7f9966ee3f40, task:0x7f993335be50, task_type:"N9oceanbase3sql3dtl23ObDTLIntermResultGCTaskE", scheduled_time:1755161747765953, pushed_time:0, delay:10000000}, running_task_token={this:0x7f9931bb4178, timer:0x7f9966ee3f40, task:0x7f99333d2fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161723481487, pushed_time:1755161737766048, delay:900000000}, total_delay=362102518, threashold=1000000, tenant_id=1001) [2025-08-14 09:01:49.880058] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a4658, timer:0x7f9966a63f10, task:0x7f9922e05038, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr12TabletGCTaskE", scheduled_time:1755162048800633, pushed_time:0, delay:20000}, running_task_token={this:0x7f991b7a4498, timer:0x7f9966a63f10, task:0x7f9922e05010, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr11TableGCTaskE", scheduled_time:1755162048800605, pushed_time:1755162048800872, delay:20000}, total_delay=61079424, threashold=1000000, tenant_id=1002) [2025-08-14 09:01:50.517522] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:50.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.738987] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=221127][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:50.517|207=221.46ms|210=1us|222=1us|total=221.49ms) [2025-08-14 09:01:50.739009] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4389] timer task use too much time(end=1755162110739008, start=1755162110517520, delta=221488) [2025-08-14 09:01:50.739021] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=149703, expected_run_ts_us_=1755162110589316, ObClockGenerator::getRealClock()=1755162110739020, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:50.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.739037] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:50.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.680341] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:01:50.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.739494] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=75] get wrs ts(ls_id={id:1001}, delta=366010672, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:50.739510] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:43.766325] EDIAG [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=15994][errcode=0] LOGGER COST TOO MUCH TIME, cost: 6967138, owner: ObLog, click_count: 5, time dist:[FORMAT_END=6967131, ObMalloc#start=2, ObMalloc#end=4, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x989c577 0x989bc96 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [663]OB_MALLOC COST TOO MUCH TIME, cost_time=965380, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=47, WASH_BLOCK_END=965333], seq:[0, 1, 2] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=964498, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=964458], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=1001440, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=989534], seq:[0, 1] [2025-08-14 09:01:49.750140] EDIAG eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1001484, owner: ObLog, click_count: 5, time dist:[FORMAT_END=26, ObMalloc#start=1, ObMalloc#end=1001456, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x27a8a8c8 0x97b7e1a 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [393]OB_MALLOC COST TOO MUCH TIME, cost_time=998219, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=52, WASH_BLOCK_END=998166], seq:[0, 1, 2] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=997768, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=50, WASH_BLOCK_END=997718], seq:[0, 1, 2] [303]OB_MALLOC COST TOO MUCH TIME, cost_time=942298, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=47, WASH_BLOCK_END=942251], seq:[0, 1, 2] [683]OB_MALLOC COST TOO MUCH TIME, cost_time=277697, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=61, WASH_BLOCK_END=277636], seq:[0, 1, 2] [797]OB_MALLOC COST TOO MUCH TIME, cost_time=177853, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=51, WASH_BLOCK_END=177802], seq:[0, 1, 2] [2025-08-14 09:01:49.839536] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c0b90, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571c0e70, time_wheel:0x7f9959fef990, thread_pool:0x7f9959fef170, is_running:1, time_interval:3.00s, expected_run_time:09:01:49.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.739550] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=898650][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:49.839|207=900.01ms|210=0us|222=1us|total=900.03ms) [2025-08-14 09:01:50.580253] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:50.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.719566] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:50.721503] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:50.736363] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:50.730034] WDIAG [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:304) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=2988745][errcode=-4389] blacklist_loop cost too much time(cost_time=971987) [2025-08-14 09:01:50.742637] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:50.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.742731] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:50.739610] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18078][errcode=0] epoll handler too much time :18117 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:43.766325] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=15994] [thread pool stat](pool_stat={pool:0x564d6e8922c0, last_check_time:1755162103766323, last_idle_time:102477764955, last_thread_count:12}, *pool={name:TimerWK, this:0x564d6e8922c0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:10, threads_idle_time:102477764955, tenant_id:500}, qsize=10, current_idle_time=102477764955) [2025-08-14 09:01:50.739597] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19998][errcode=0] epoll handler too much time :20041 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:50.739581] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=159085][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:50.580|207=159.33ms|210=0us|222=1us|total=159.34ms) [2025-08-14 09:01:50.744528] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=4937][errcode=-4389] timer task use too much time(end=1755162110744527, start=1755162110580251, delta=164276) [2025-08-14 09:01:50.739568] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4389] timer task use too much time(end=1755162110739567, start=1755162109839534, delta=900033) [2025-08-14 09:01:50.739533] INFO [STORAGE.TRANS] amend (ob_tenant_weak_read_server_version_mgr.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [WRS] tenant has no valid partition on current server, generate weak read snapshot version based on current timestamp(tenant_id=1002, epoch_tstamp_=1755162110738704, total_part_count_=2, valid_inner_part_count_=0, valid_user_part_count_=0, version_={val:1755161758221798000, v:0}, base_version_when_no_valid_partition={val:1755161758221798000, v:0}) [2025-08-14 09:01:50.744597] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=5038] get wrs ts(ls_id={id:1}, delta=361000529, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:50.744609] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:50.745185] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=366013898, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:50.745194] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:50.745332] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:50.745353] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=15] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:01:50.746398] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:50.747452] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:50.739050] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:50.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.748288] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:50.749074] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8442][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:50.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.740565] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:01:50.750312] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:50.750331] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196a790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:50.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:50.740363] INFO [COMMON] loop_running_dag_net_list (ob_tenant_dag_scheduler.cpp:3845) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1038475] loop running dag net list(ret=0, running_dag_net_list_size=0, slow_dag_net_cnt=0, finished_dag_net_list_size=0) [2025-08-14 09:01:50.750388] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=9] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:48.754199] EDIAG [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=3014782][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2002045, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2002044, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b51df0f 0x986bcde 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:49.754221] EDIAG [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1008608, owner: ObLog, click_count: 5, time dist:[FORMAT_END=7, ObMalloc#start=0, ObMalloc#end=997778, ALLOC_END=10823, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a1f867 0x27a135c3 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:49.753741] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1009202, owner: ObLog, click_count: 5, time dist:[FORMAT_END=7, ObMalloc#start=1, ObMalloc#end=998232, ALLOC_END=10962, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:48.751182] EDIAG [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=1007567][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2010443, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2010442, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b51df0f 0x986bcde 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:50.742709] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:49.998405] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:49.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:48.746719] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [1000][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F01-0-0] [lt=24][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1002, req={req_server:"172.77.0.7:2882", version:{val:1755161758221798000, v:0}, valid_part_count:0, total_part_count:2, generate_timestamp:1755162099836883}, twrs={inited:true, tenant_id:1002, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162109680260}, server_version_for_stat:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162109680260}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:01:50.751915] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=24] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:50.751931] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] temporary file module use no blocks [2025-08-14 09:01:50.751937] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=5] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:50.738878] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=4965764] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161956777957, last_flushed_end_lsn:{lsn:8461098908}}]) [2025-08-14 09:01:50.752009] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:50.752030] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:50.750359] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918114b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a0d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:50.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:50.752459] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=2093][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728aa90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:50.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:50.750341] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [316][pnio1][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1002, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162099836894", warnings:[]}, tenant_id_=1002, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:01:50.752480] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f0490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:50.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:50.752491] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f0610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:50.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:50.752500] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7281290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:50.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:50.750381] INFO [COMMON] loop_finished_dag_net_list (ob_tenant_dag_scheduler.cpp:3868) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10001] loop finsihed dag net list(ret=0, finished_dag_net_list_size=0) [2025-08-14 09:01:50.753075] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=563013840, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:50.753090] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:49.750140] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=20] PNIO [ratelimit] time: 1755162109750137, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007757, add_bytes: 0 [2025-08-14 09:01:50.753178] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=1001484] PNIO [ratelimit] time: 1755162110753177, bytes: 0, bw: 0.000000 MB/s, add_ts: 1003040, add_bytes: 0 [2025-08-14 09:01:50.751947] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:50.753261] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=1310] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623011216786, flush_io_finished_ret_:0, flush_io_finished_round_:26256, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:50.752509] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891825810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:50.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:50.753544] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1029][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891971290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:50.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:50.753571] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a0c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:50.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:50.740401] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [723][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=18] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=16, total_destructed_detector=16, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f99333562b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-08-14 09:01:50.753283] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [564][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=10049] XA scheduler heartbeat task statistics(avg_time=2581) [2025-08-14 09:01:50.754107] WDIAG [DETECT] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [723][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=13682][errcode=0] cost too much time:ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t), (this=|threshold=100.00ms|start at 09:01:50.754|total=1.90s) [2025-08-14 09:01:50.759124] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EE95-0-0] [lt=10] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-08-14 09:01:50.759148] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EE95-0-0] [lt=13][errcode=-4016] task executed too delay(delay_time=920855, *p_task_={this:0x7f99571fc8b0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcb90, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:0, time_interval:3.00s, expected_run_time:09:01:52.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.753676] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=13075] current log_task status(palf_id=1, self="172.77.0.7:2882", log_id=12632582, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161956777957, last_flushed_end_lsn:{lsn:8461098908}}]) [2025-08-14 09:01:48.754199] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=3014782] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136826952869001, v:0}, clog_base_lsn:{lsn:5099962368}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1755071646732636000, v:0}, all_id_meta:{id_meta:[{limited_id:1755161555795565000, latest_log_ts:{val:1755161545796642001, v:0}}, {limited_id:22000001, latest_log_ts:{val:1755135645787806003, v:0}}, {limited_id:60000001, latest_log_ts:{val:1755135646400653001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:50, palf_env_:0x7f9950bf4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-08-14 09:01:50.753200] WDIAG [RPC] handle_resp (ob_poc_rpc_proxy.cpp:192) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] async_cb handler cost too much time(total_time=2977540, decode_time=-1755162107774306, process_time=0, session_destroy_time=1755162110751846, ret=0, pcode=5395) [2025-08-14 09:01:50.759565] WDIAG stat_cleanup (debug.c:35) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=6354][errcode=0] PNIO [delay_warn] cost too much time: 2985020us, procedure: pn_pktc_resp_cb [2025-08-14 09:01:50.759594] WDIAG eloop_delay_warn (debug.h:93) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] PNIO [delay_warn] eloop handle events delay high: 2998020, malloc=0/0 write=0/0 read=0/1 server_process=0/0 client_cb=2985020/1 [2025-08-14 09:01:50.761762] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=3004860] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:01:50.753629] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2826) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6997520] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:371, [delay/us]: prepare:0, schedule:0, submit:600000000, device:959233, total:959241) [2025-08-14 09:01:50.762697] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fc830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891825210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:50.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:50.762728] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7282910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:50.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:49.754221] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=25] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:50.764383] WDIAG ev_invoke_pending (ev.c:2136) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=1008608][errcode=0] epoll handler too much time :1010164 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:50.764407] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=16] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:49.753741] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:50.764486] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1009202][errcode=0] epoll handler too much time :1010754 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:50.762742] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a54b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7282a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:50.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:50.764501] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:50.764523] WDIAG pkts_handle_req_queue (pkts_post.h:51) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] PNIO [delay_warn] delay high: 1036007 [2025-08-14 09:01:50.575833] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fe410, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe6f0, time_wheel:0x7f9922df9d40, thread_pool:0x7f9922df9520, is_running:1, time_interval:2.00s, expected_run_time:09:01:50.432, task_priority:1, with_handle_protected:1}) [385]OB_MALLOC COST TOO MUCH TIME, cost_time=1014482, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=50, WASH_BLOCK_END=1001311, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=12762], seq:[0, 1, 2, 3, 4] [2025-08-14 09:01:49.750587] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1014509, owner: ObLog, click_count: 5, time dist:[FORMAT_END=8, ObMalloc#start=0, ObMalloc#end=1014498, ALLOC_END=2, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:50.764561] WDIAG delay_warn (debug.h:103) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] PNIO [delay_warn] pktc_flush_cb delay high: 3003020, start_us=1755162107760772 [2025-08-14 09:01:50.764553] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=188431][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:50.575|207=188.72ms|210=0us|222=2us|total=188.74ms) [2025-08-14 09:01:50.764571] WDIAG pktc_handle_req_queue (pktc_post.h:138) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] PNIO [delay_warn] delay high: 2987020 [2025-08-14 09:01:50.764571] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] timer task use too much time(end=1755162110764571, start=1755162110575831, delta=188740) [2025-08-14 09:01:50.764578] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=6967138] [thread pool stat](pool_stat={pool:0x7f99667cbcf0, last_check_time:1755162110764577, last_idle_time:34719886791, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x7f99667cbcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:34719886791, tenant_id:500}, qsize=-1, current_idle_time=34719886791) [2025-08-14 09:01:50.764598] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=19] [thread pool stat](pool_stat={pool:0x7f99667ccaf0, last_check_time:1755162110764597, last_idle_time:26494382053, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x7f99667ccaf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:26494382053, tenant_id:500}, qsize=-1, current_idle_time=26494382053) [2025-08-14 09:01:50.764596] WDIAG [RPC] create (ob_poc_rpc_server.cpp:140) [317][pnio1][T0][YB42AC4D0007-00063C49607B9209-0-0] [lt=5][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1815, fly_ts=10043218, send_timestamp=1755162100721376, peer="172.77.0.7:42280", sz=265) [2025-08-14 09:01:50.764609] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] [thread pool stat](pool_stat={pool:0x7f9969de56b0, last_check_time:1755162110764608, last_idle_time:102938681541, last_thread_count:17}, *pool={name:TimerWK, this:0x7f9969de56b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:16, threads_idle_time:102938681541, tenant_id:1}, qsize=9, current_idle_time=102938681541) [2025-08-14 09:01:50.475826] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fa190, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa470, time_wheel:0x7f99341cdbf0, thread_pool:0x7f99341cd3d0, is_running:1, time_interval:2.00s, expected_run_time:09:01:50.334, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.764618] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] [thread pool stat](pool_stat={pool:0x7f99356b0e70, last_check_time:1755162110764618, last_idle_time:102366343170, last_thread_count:15}, *pool={name:TimerWK, this:0x7f99356b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:12, threads_idle_time:102366343170, tenant_id:1001}, qsize=12, current_idle_time=102366343170) [2025-08-14 09:01:50.764627] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=7] [thread pool stat](pool_stat={pool:0x7f99230b0e70, last_check_time:1755162110764626, last_idle_time:102534969158, last_thread_count:17}, *pool={name:TimerWK, this:0x7f99230b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:16, threads_idle_time:102534969158, tenant_id:1002}, qsize=13, current_idle_time=102534969158) [2025-08-14 09:01:50.764632] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=288357][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:50.475|207=288.80ms|210=0us|222=11us|total=288.82ms) [2025-08-14 09:01:50.764643] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task use too much time(end=1755162110764642, start=1755162110475824, delta=288818) [2025-08-14 09:01:49.809890] INFO handle (memory_dump.cpp:671) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=14] statistics: tenant_cnt: 6, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 168 271 0 1 2 110 1 0 1 3 7 0 0 1 4 1 1 0 1 5 86 6 0 1 6 13 0 0 1 7 22 2 0 1 8 53 1 0 1 12 17 3 0 1 13 10 1 0 1 14 1 1 0 1 15 1 1 0 1 16 4 6 0 1 18 4 3 0 1 20 2 1 0 1 22 1 1 0 1 24 1 0 0 1 25 6 0 0 500 0 563 275 0 500 7 188 6 0 500 8 41 1 0 500 9 2 1 0 500 10 9 2 0 500 16 50 4 0 500 17 52 7 0 500 22 38 64 0 500 23 27 10 0 508 0 2 7 0 508 8 5 1 0 509 0 2 7 0 509 8 5 1 0 1001 0 147 251 0 1001 2 26 1 0 1001 3 5 0 0 1001 4 1 1 0 1001 5 85 6 0 1001 6 11 0 0 1001 7 30 2 0 1001 8 36 1 0 1001 12 17 3 0 1001 13 3 1 0 1001 14 1 1 0 1001 16 6 3 0 1001 18 4 3 0 1001 20 4 1 0 1001 25 8 0 0 1002 0 262 278 0 1002 2 198 1 0 1002 3 4 0 0 1002 4 1 1 0 1002 5 101 6 0 1002 6 14 0 0 1002 7 6 2 0 1002 8 73 1 0 1002 12 18 3 0 1002 13 7 1 0 1002 14 1 2 0 1002 16 8 8 0 1002 18 9 3 0 1002 20 2 1 0 1002 21 1 0 0 1002 24 1 0 0 1002 25 7 0 0 cost_time: 2085293 [2025-08-14 09:01:48.751182] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=1007567] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136829134139001, v:0}, clog_base_lsn:{lsn:8186781696}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1753840867372367034, v:0}, all_id_meta:{id_meta:[{limited_id:1755161729775179000, latest_log_ts:{val:1755161719775179001, v:0}}, {limited_id:18000001, latest_log_ts:{val:1755157440846556001, v:0}}, {limited_id:60000001, latest_log_ts:{val:1755135657266496001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:46, palf_env_:0x7f99321fa030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-08-14 09:01:50.738696] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=24] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:4096/1:417980 disk:4096/1:417980 t1m2:4096/1:417980 t1m2g0:4096/1:417980 |q0: 0 |q1: 0 [2025-08-14 09:01:50.753112] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162110752506}, version_delta=563014596) [2025-08-14 09:01:50.765913] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12221] get wrs ts(ls_id={id:1}, delta=563024968, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:50.765928] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:50.759179] WDIAG [OCCAM] dump_statistics (ob_vtable_event_recycle_buffer.h:144) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EE96-0-0] [lt=29][errcode=-4006] not init(ret=-4006) [2025-08-14 09:01:49.750587] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:50.766652] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1014509][errcode=0] epoll handler too much time :1016073 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:50.766667] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:50.766688] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [623][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9209-0-0] [lt=25][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162100721364) [2025-08-14 09:01:50.766709] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [623][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9209-0-0] [lt=15][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162100721364}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162110765354}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162110765354}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:01:50.768717] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=1038745] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:50.768740] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=22] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:50.768746] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:50.768752] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:50.768758] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:50.768791] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=32] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:50.768796] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:50.768803] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:50.768807] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=4] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:50.768816] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:01:50.768824] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:01:50.770140] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:01:50.770169] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:01:50.770189] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:01:50.770199] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:00:41.841, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=08:59:41.841], idx=471, next_allow_print_ts="09:01:11.841") [2025-08-14 09:01:50.770210] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:01:50.770226] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:01:50.770245] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:01:50.770451] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=198][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:01:50.770464] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:01:50.770478] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:01:50.770490] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:01:50.770500] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:00:52.998, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:22.998], idx=771, next_allow_print_ts="09:01:22.998") [2025-08-14 09:01:50.770518] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=08:59:57.942, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:27.942], idx=868, next_allow_print_ts="09:00:27.942") [2025-08-14 09:01:50.770679] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:01:50.770931] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=246][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:01:50.770942] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:01:50.747566] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:50.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.771054] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=10988] blacklist_loop exec finished(cost_time=17, is_enabled=true, send_cnt=0) [2025-08-14 09:01:50.771087] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:50.771203] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [950][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F08-0-0] [lt=7][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1002, svr="172.77.0.7:2882", version={val:1755161758221798000, v:0}, valid_part_count=0, total_part_count=2, generate_timestamp=1755162100637393) [2025-08-14 09:01:50.771393] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9211-0-0] [lt=6388][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162101621919) [2025-08-14 09:01:50.771406] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9211-0-0] [lt=13][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162101621919}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162110765354}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162110765354}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:01:50.771459] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [622][T1_L0_G0][T1][YB42AC4D0007-00063C49607B920C-0-0] [lt=33][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162101121583) [2025-08-14 09:01:50.771470] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [622][T1_L0_G0][T1][YB42AC4D0007-00063C49607B920C-0-0] [lt=11][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162101121583}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162110765354}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162110765354}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:01:50.764557] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8636] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:01:50.771223] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [950][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F08-0-0] [lt=14][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1002, req={req_server:"172.77.0.7:2882", version:{val:1755161758221798000, v:0}, valid_part_count:0, total_part_count:2, generate_timestamp:1755162100637393}, twrs={inited:true, tenant_id:1002, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162110744581}, server_version_for_stat:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162110744581}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:01:50.773801] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1}, delta=361022291, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:50.773825] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:50.775193] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=366035660, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:50.781779] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:50.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.781883] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=14] gc stale ls task succ [2025-08-14 09:01:50.782036] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:50.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.782065] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1755][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72edc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:50.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:50.782078] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72edb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:50.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:50.782089] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7398490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:50.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:50.782100] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7398310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:50.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:50.782110] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7289110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:50.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:50.782121] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ac910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:50.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:50.782131] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728ac10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:50.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:50.782178] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=14] gc stale ls task succ [2025-08-14 09:01:50.787284] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:50.787643] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [979][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F0D-0-0] [lt=19][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1002, svr="172.77.0.7:2882", version={val:1755161758221798000, v:0}, valid_part_count=0, total_part_count=2, generate_timestamp=1755162101141487) [2025-08-14 09:01:50.787667] INFO pktc_resp_cb_on_msg (pktc_resp.h:72) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=13] PNIO resp cb not found: packet_id=559659 [2025-08-14 09:01:50.787674] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [979][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F0D-0-0] [lt=29][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1002, req={req_server:"172.77.0.7:2882", version:{val:1755161758221798000, v:0}, valid_part_count:0, total_part_count:2, generate_timestamp:1755162101141487}, twrs={inited:true, tenant_id:1002, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162110744581}, server_version_for_stat:{version:{val:18446744073709551615, v:3}, total_part_count:1, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162110773786}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:01:50.787711] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [317][pnio1][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162100721371", warnings:[]}, tenant_id_=1, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:01:50.787747] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9212-0-0] [lt=22][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162101721967) [2025-08-14 09:01:50.787766] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9212-0-0] [lt=18][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162101721967}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162110765354}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162110765354}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:01:50.788434] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [851][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=16] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=16, total_destructed_detector=16, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f9922fee2b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-08-14 09:01:50.789265] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=563024968, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:50.789290] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 08:59:02.843452] EDIAG [SQL] do_open_plan (ob_result_set.cpp:575) [1061][T1002_MergeSche][T1001][YB42AC4D0007-00063C4962499CBA-0-0] [lt=12][errcode=-4283] LOGGER COST TOO MUCH TIME, cost: 167676529, owner: ObLog, click_count: 3, time dist:[FORMAT_END=167676527, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x14944284 0x93375f3 0x932e8ab 0x96ac524 0x96a672d 0x96a5033 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x1fd07c25 0x11254ef7 0x1125a47b 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:47.726651] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1531) [693][T1001_TimerWK3_FreInfoReload][T1001][YB42AC4D0007-00063C495DC9FC50-0-0] [lt=21] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f98b4f62a80, tx_id:{txid:0}, state:1, addr:"172.77.0.7:2882", tenant_id:1001, session_id:3221590777, assoc_session_id:3221590777, client_sid:3221590777, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1755161719764278, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180067074, seq_base:1755161719762243, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], conflict_txs:[]}, rollback=true, need_rollback=true, session={this:0x7f98394041f8, id:3221590777, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1001, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f98b4f62a80}, exec_ctx.get_errcode()=-4038) [2025-08-14 09:01:50.753718] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [742][T1001_DBMSSched][T1001][Y0-0000000000000000-0-0] [lt=5][errcode=0] long time to hold one guard(schema_mgr=0x7f98ad7818f0, tenant_id=1, version=1753073979411448, cur_timestamp=1755161864773076, ref_timestamp=1755161801748404, lbt()="0x26a1806d 0x974f3e5 0x974e159 0x974e084 0x948f9df 0x9bf0be9 0x12e4ea0d 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3") [2025-08-14 09:01:50.763333] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:50.790896] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27521][errcode=0] epoll handler too much time :27572 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:50.751658] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=752111][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:49.998|207=753.25ms|210=1us|222=1us|total=792.54ms) [2025-08-14 09:01:50.790947] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=39228][errcode=-4389] timer task use too much time(end=1755162110790946, start=1755162109998403, delta=792543) [2025-08-14 09:01:50.790961] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=546327, expected_run_ts_us_=1755162110244633, ObClockGenerator::getRealClock()=1755162110790960, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:50.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.790982] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:50.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.754215] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE2-0-0] [lt=989350][errcode=0] long time to hold one guard(schema_mgr=0x7f98a2767660, tenant_id=1002, version=1755156894100160, cur_timestamp=1755162015746259, ref_timestamp=1755161800760341, lbt()="0x26a1806d 0x974f3e5 0x974e159 0x974e084 0x948fab3 0x97257c0 0x97256f4 0x9725691 0x1e6838ef 0x1e683258 0x1056234f 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3") [2025-08-14 09:01:49.782419] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [663][T1001_StandbySc][T1001][YB42AC4D0007-00063C495C7A02ED-0-0] [lt=20][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1008018, owner: ObLog, click_count: 7, time dist:[FORMAT_END=14, ObMalloc#start=0, ObMalloc#end=965396, ObMalloc#start=15015, ObMalloc#end=27585, ALLOC_END=5, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:49.783336] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1007130, owner: ObLog, click_count: 6, time dist:[FORMAT_END=7, ObMalloc#start=1, ObMalloc#end=964507, ObMalloc#start=15121, ObMalloc#end=27459, FREE_END=35], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:50.790350] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=953331] [MEMORY] tenant: 1, limit: 2147483648 hold: 1006882816 cache_hold: 56623104 cache_used: 56623104 cache_item_count: 27 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 444993536 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 230686720 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 1163264 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 483328 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 53714944 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 1458176 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 14204928 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 111149056 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35651584 limit= 429496720 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 8003584 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= PS_CACHE_CTX_ID hold_bytes= 65536 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 6438912 limit= 9223372036854775807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 3481600 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3964928 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 212992 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 884736 limit= 9223372036854775807 [2025-08-14 09:01:50.792013] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=1007130][errcode=0] epoll handler too much time :1008685 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:50.792033] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [736]OB_MALLOC COST TOO MUCH TIME, cost_time=1009326, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=50, WASH_BLOCK_END=965452, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=15926], seq:[0, 1, 2, 3, 4] [2025-08-14 09:01:49.782276] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:49.782276] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1009371, owner: ObLog, click_count: 5, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=1009349, ALLOC_END=3, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:50.794915] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [317][pnio1][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1002, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162101141493", warnings:[]}, tenant_id_=1002, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:01:50.794946] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [317][pnio1][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162101721973", warnings:[]}, tenant_id_=1, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:01:50.795023] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [623][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9215-0-0] [lt=26][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162107777401) [2025-08-14 09:01:50.795033] INFO pktc_resp_cb_on_msg (pktc_resp.h:72) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=9] PNIO resp cb not found: packet_id=559667 [2025-08-14 09:01:50.795050] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [623][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9215-0-0] [lt=21][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162107777401}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162110788691}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162110788691}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:01:50.795089] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [623][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9219-0-0] [lt=27][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162108948391) [2025-08-14 09:01:50.795095] INFO pktc_resp_cb_on_msg (pktc_resp.h:72) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=37] PNIO resp cb not found: packet_id=559675 [2025-08-14 09:01:50.795103] INFO pktc_resp_cb_on_msg (pktc_resp.h:72) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO resp cb not found: packet_id=559682 [2025-08-14 09:01:50.795101] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [623][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9219-0-0] [lt=11][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162108948391}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162110788691}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162110788691}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:01:50.795133] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [623][T1_L0_G0][T1][YB42AC4D0007-00063C49607B921D-0-0] [lt=25][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162109448653) [2025-08-14 09:01:50.795144] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [623][T1_L0_G0][T1][YB42AC4D0007-00063C49607B921D-0-0] [lt=11][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162109448653}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162110788691}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162110788691}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:01:50.795165] INFO [SHARE] end_run (ob_tenant_base.cpp:389) [15523][T1002_TimerWK16][T0][Y0-0000000000000000-0-0] [lt=67875325] tenant thread end_run(ret=0, thread_count_=282, id_=1002, GET_GROUP_ID()=0) [2025-08-14 09:01:50.795838] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=563024968, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:50.795869] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:50.796118] INFO [MDS] dump_map_holding_item (mds_tenant_service.cpp:462) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EE96-0-0] [lt=6822] finish scan map holding items(scan_cnt=0) [2025-08-14 09:01:50.796494] WDIAG [RPC] handle_resp (ob_poc_rpc_proxy.cpp:192) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=2140][errcode=0] async_cb handler cost too much time(total_time=1965983, decode_time=-1755162108830510, process_time=0, session_destroy_time=1755162110796493, ret=0, pcode=1815) [2025-08-14 09:01:50.796517] WDIAG stat_cleanup (debug.c:35) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] PNIO [delay_warn] cost too much time: 1966014us, procedure: pn_pktc_resp_cb [2025-08-14 09:01:50.796526] WDIAG eloop_delay_warn (debug.h:93) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] PNIO [delay_warn] eloop handle events delay high: 1966014, malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=1966014/1 [2025-08-14 09:01:50.796552] WDIAG pktc_handle_req_queue (pktc_post.h:138) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] PNIO [delay_warn] delay high: 1917014 [2025-08-14 09:01:50.796589] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [316][pnio1][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1002, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162100637400", warnings:[]}, tenant_id_=1002, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:01:50.796612] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [316][pnio1][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162101121589", warnings:[]}, tenant_id_=1, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:01:50.796630] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [316][pnio1][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162101621924", warnings:[]}, tenant_id_=1, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:01:50.796679] WDIAG [RPC] create (ob_poc_rpc_server.cpp:140) [316][pnio1][T0][YB42AC4D0007-00063C49612B8F16-0-0] [lt=11][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1815, fly_ts=1916866, send_timestamp=1755162108879812, peer="172.77.0.7:49788", sz=265) [2025-08-14 09:01:50.796737] INFO pktc_resp_cb_on_msg (pktc_resp.h:72) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=15] PNIO resp cb not found: packet_id=559681 [2025-08-14 09:01:50.796749] INFO pktc_resp_cb_on_msg (pktc_resp.h:72) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=12] PNIO resp cb not found: packet_id=559674 [2025-08-14 09:01:50.796753] INFO pktc_resp_cb_on_msg (pktc_resp.h:72) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO resp cb not found: packet_id=559666 [2025-08-14 09:01:50.796785] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B921C-0-0] [lt=25][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162109348572) [2025-08-14 09:01:50.796810] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B921C-0-0] [lt=19][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162109348572}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162110795217}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162110795217}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:01:50.739769] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=4012030] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:01:50.796891] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=57034] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:01:50.775210] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:50.796950] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4015] clock out of order(cur_ts=1755162110790314, cur_ts_=1755162110763484, delta=26830) [2025-08-14 09:01:50.797408] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [963][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F16-0-0] [lt=9][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1002, svr="172.77.0.7:2882", version={val:1755161758221798000, v:0}, valid_part_count=0, total_part_count=2, generate_timestamp=1755162108879793) [2025-08-14 09:01:50.797429] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [963][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F16-0-0] [lt=20][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1002, req={req_server:"172.77.0.7:2882", version:{val:1755161758221798000, v:0}, valid_part_count:0, total_part_count:2, generate_timestamp:1755162108879793}, twrs={inited:true, tenant_id:1002, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162110744581}, server_version_for_stat:{version:{val:18446744073709551615, v:3}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162110773786}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:01:50.797698] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=563024968, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:50.797719] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:50.798038] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21699] get wrs ts(ls_id={id:1}, delta=361022291, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:50.798061] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:50.798624] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25] get wrs ts(ls_id={id:1001}, delta=366035660, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:50.798641] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:50.801926] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:50.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.812719] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b5690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:50.802, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:50.812754] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891970990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:50.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:50.812766] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728f9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7292010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:50.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:50.822779] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72eae10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:50.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:50.828959] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc5538, timer:0x564d6efbde60, task:0x564d6efbde38, task_type:"N9oceanbase12blocksstable14ObBlockManager19InspectBadBlockTaskE", scheduled_time:1755162049755883, pushed_time:0, delay:1000000}, running_task_token={this:0x7f9970fc5618, timer:0x564d6efbde60, task:0x564d6efbde20, task_type:"N9oceanbase12blocksstable14ObBlockManager13MarkBlockTaskE", scheduled_time:1755162036737744, pushed_time:1755162048755971, delay:30000000}, total_delay=61073075, threashold=1000000, tenant_id=500) [2025-08-14 09:01:50.832834] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739f290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:50.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:50.842697] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:50.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.844612] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:50.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.873009] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7289e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:50.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:50.873043] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7288f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:50.866, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:50.880738] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:01:50.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.897501] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=563158346, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:50.897527] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:50.898100] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=361156721, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:50.898122] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:50.898762] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26] get wrs ts(ls_id={id:1001}, delta=366170090, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:50.898784] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:50.929012] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a6a38, timer:0x7f9966fddee0, task:0x7f994f1b35a8, task_type:"N9oceanbase3omt11ObTenantSrs27TenantSrsUpdatePeriodicTaskE", scheduled_time:1755161929777851, pushed_time:0, delay:0}, running_task_token={this:0x7f994e1a65d8, timer:0x7f9966fddee0, task:0x7f994f1bbcc0, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161694824965, pushed_time:1755161925772608, delay:5000000}, total_delay=181151160, threashold=1000000, tenant_id=1) [2025-08-14 09:01:50.942757] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:50.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.944636] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:50.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.947647] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=23464][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:50.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.964871] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=25960] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:01:50.991987] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:50.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.993792] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:50.989, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:50.997563] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=563258738, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:50.997586] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:50.998149] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=361256061, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:50.998181] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=29] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:50.998824] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27] get wrs ts(ls_id={id:1001}, delta=366270502, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:50.998842] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:51.013927] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:51.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.023328] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:51.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.042818] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:50.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.044673] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:50.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.047707] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:50.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.094445] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:51.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.097627] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=563359092, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:51.097652] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:51.098226] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=361356415, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:51.098262] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=32] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:51.098881] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=31] get wrs ts(ls_id={id:1001}, delta=366369784, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:51.098897] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:51.142878] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:51.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.144727] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:51.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.147780] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:51.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.197745] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=563458506, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:51.197770] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:51.198276] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=361456888, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:51.198317] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=38] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:51.198948] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] get wrs ts(ls_id={id:1001}, delta=366470257, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:51.198978] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:51.215646] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:51.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.223718] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:51.223748] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=28] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3793) [2025-08-14 09:01:51.223757] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:01:51.223771] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:51.223788] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=16] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3794) [2025-08-14 09:01:51.242950] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:51.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.243324] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:51.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.244741] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:51.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.245822] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:51.239, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.247844] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:51.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.253364] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:51.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.265901] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:51.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.271110] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:01:51.271148] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:01:51.271177] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:01:51.271190] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:00:41.841, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=08:59:41.841], idx=471, next_allow_print_ts="09:01:11.841") [2025-08-14 09:01:51.271210] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:01:51.271227] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:01:51.274665] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:51.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.298282] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=361554394, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:51.298308] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:51.298895] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1001}, delta=366570593, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:51.298909] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:51.304851] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:51.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.335574] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7281b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.343011] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:51.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.344770] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:51.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.443075] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:51.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.444787] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:51.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.494634] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:51.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.497195] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:51.489, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.517284] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:51.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.526121] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:51.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.543137] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:51.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.544860] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:51.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.597745] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:51.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.643202] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:51.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.644932] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:51.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.683800] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=58632][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:01:51.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.718353] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:51.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.720017] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4407] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:51.721852] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4384] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:51.726977] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=21] [sql nio session](*s={this:0x7f98f72b3030, session_id:3221562808, trace_id:YB42AC4D0007-00063C4965DA1F2E-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:267}, err:0, last_decode_time:1755161748766816, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1890, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:01:51.727044] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=67] [sql nio session](*s={this:0x7f98f72ab030, session_id:3221589435, trace_id:YB42AC4D0007-00063C4965DA1F35-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:246}, err:0, last_decode_time:1755161762721798, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:2456, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:01:51.727063] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=19] [sql nio session](*s={this:0x7f98f72a9030, session_id:3221590528, trace_id:YB42AC4D0007-00063C4965DA1F34-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:250}, err:0, last_decode_time:1755161762715415, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:205, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:01:51.727079] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=16] [sql nio session](*s={this:0x7f98f72b5030, session_id:3221590961, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:10, sql_initiative_shutdown:false, reader:{fd:278}, err:5, last_decode_time:1755162075745307, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:01:51.223792] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:01:50.790648] WDIAG [SERVER] scheduler (ob_dbms_sched_job_master.cpp:170) [742][T1001_DBMSSched][T1001][Y0-0000000000000000-0-0] [lt=36869][errcode=0] fail to check tenant(tmp_ret=-4012) [2025-08-14 09:01:51.729455] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=504586] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:51.729511] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=55] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3795) [2025-08-14 09:01:51.729521] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:01:51.271254] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:01:51.729594] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=457598][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:01:51.729621] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:01:49.731, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:01:42.751], idx=659, next_allow_print_ts="09:02:19.731") [2025-08-14 09:01:51.729635] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:01:51.729651] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:01:51.729664] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:01:51.729677] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:00:52.998, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:22.998], idx=771, next_allow_print_ts="09:01:22.998") [2025-08-14 09:01:51.729705] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=08:59:57.942, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:27.942], idx=868, next_allow_print_ts="09:00:27.942") [2025-08-14 09:01:51.730043] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:01:51.730071] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:01:51.730086] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:01:51.347907] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:51.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.730230] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=381741][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:51.347|207=382.32ms|210=1us|222=1us|total=382.35ms) [2025-08-14 09:01:51.730258] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4389] timer task use too much time(end=1755162111730258, start=1755162111347906, delta=382352) [2025-08-14 09:01:51.730410] WDIAG [COORDINATOR] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [657][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=0] cost too much time:ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_(), (this=|threshold=1.00s|start at 09:01:51.730|total=1.98s) [2025-08-14 09:01:51.730545] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [317][pnio1][T1][YB42AC4D0007-00063C49607B9215-0-0] [lt=6][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1, delta=78967023, last_succ_cluster_heartbeat_tstamp_=1755162032763521) [2025-08-14 09:01:51.730562] WDIAG [RPC] handle_resp (ob_poc_rpc_proxy.cpp:192) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] async_cb handler cost too much time(total_time=935437, decode_time=935419, process_time=15, session_destroy_time=3, ret=0, pcode=1815) [2025-08-14 09:01:51.730577] WDIAG stat_cleanup (debug.c:35) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] PNIO [delay_warn] cost too much time: 936006us, procedure: pn_pktc_resp_cb [2025-08-14 09:01:51.730588] WDIAG eloop_delay_warn (debug.h:93) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] PNIO [delay_warn] eloop handle events delay high: 936006, malloc=0/0 write=0/1 read=0/2 server_process=0/0 client_cb=936006/1 [2025-08-14 09:01:51.730607] WDIAG pktc_handle_req_queue (pktc_post.h:138) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] PNIO [delay_warn] delay high: 833006 [2025-08-14 09:01:51.730644] WDIAG [RPC] handle_resp (ob_poc_rpc_proxy.cpp:192) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=3][errcode=0] async_cb handler cost too much time(total_time=933711, decode_time=933708, process_time=2, session_destroy_time=1, ret=0, pcode=1815) [2025-08-14 09:01:51.730656] WDIAG stat_cleanup (debug.c:35) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] PNIO [delay_warn] cost too much time: 934006us, procedure: pn_pktc_resp_cb [2025-08-14 09:01:51.730677] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:50.791864] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9101] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:01:51.730833] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=937544] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:01:51.737290] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4655] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:51.730664] WDIAG eloop_delay_warn (debug.h:93) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] PNIO [delay_warn] eloop handle events delay high: 934006, malloc=0/0 write=0/0 read=0/1 server_process=0/0 client_cb=934006/1 [2025-08-14 09:01:51.737414] WDIAG pktc_handle_req_queue (pktc_post.h:138) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=6731][errcode=0] PNIO [delay_warn] delay high: 841006 [2025-08-14 09:01:51.737433] WDIAG pkts_handle_req_queue (pkts_post.h:51) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] PNIO [delay_warn] delay high: 940006 [2025-08-14 09:01:51.737490] WDIAG [RPC] create (ob_poc_rpc_server.cpp:140) [316][pnio1][T0][YB42AC4D0007-00063C49607B9221-0-0] [lt=8][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1815, fly_ts=840584, send_timestamp=1755162110896905, peer="172.77.0.7:49788", sz=265) [2025-08-14 09:01:51.737636] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9221-0-0] [lt=26][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162110896890) [2025-08-14 09:01:51.737692] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [316][pnio1][T1002][YB42AC4D0007-00063C49612B8F16-0-0] [lt=15][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1002, delta=104000368, last_succ_cluster_heartbeat_tstamp_=1755162007737324) [2025-08-14 09:01:51.737664] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9221-0-0] [lt=26][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162110896890}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162111197102}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162111197102}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:01:51.730621] WDIAG pkts_handle_req_queue (pkts_post.h:51) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] PNIO [delay_warn] delay high: 936006 [2025-08-14 09:01:51.730716] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:51.737856] WDIAG [RPC] create (ob_poc_rpc_server.cpp:140) [317][pnio1][T0][YB42AC4D0007-00063C49612B8F1E-0-0] [lt=7135][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1815, fly_ts=839785, send_timestamp=1755162110898071, peer="172.77.0.7:42280", sz=265) [2025-08-14 09:01:51.737907] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [979][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F1E-0-0] [lt=29][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1002, svr="172.77.0.7:2882", version={val:1755161758221798000, v:0}, valid_part_count=0, total_part_count=2, generate_timestamp=1755162110898058) [2025-08-14 09:01:51.737925] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [979][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F1E-0-0] [lt=17][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1002, req={req_server:"172.77.0.7:2882", version:{val:1755161758221798000, v:0}, valid_part_count:0, total_part_count:2, generate_timestamp:1755162110898058}, twrs={inited:true, tenant_id:1002, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162111298266}, server_version_for_stat:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162111298266}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:01:51.738197] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72756b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b5510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.738227] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7283b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.724, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.738256] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a36b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7283c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.739789] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=838230] LogLoopThread round_cost_time(us)(round_cost_time=7957257) [2025-08-14 09:01:51.740611] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=958886] current monitor number(seq_=-1) [2025-08-14 09:01:51.741555] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8908] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:51.742272] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:51.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.743244] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:51.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.744993] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:51.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.746435] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:51.746509] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:51.746525] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=8] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:01:51.746647] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:51.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.730425] WDIAG [COORDINATOR] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [657][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=0] cost too much time:ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_failure(), (this=|threshold=1.00s|start at 09:01:51.737|total=2.96s) [2025-08-14 09:01:51.747285] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:51.748263] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919911b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196cc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.738, task_priority:1, with_handle_protected:0}) [857]OB_MALLOC COST TOO MUCH TIME, cost_time=350765, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=453889, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:01:51.748289] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d40b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ec190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.748300] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728b390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.748309] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728b510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.748317] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.748327] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.749365] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=5581] LogLoopThread round_cost_time(us)(round_cost_time=8956465) [2025-08-14 09:01:51.749371] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=8641] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:01:51.749571] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:51.749634] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119812272, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-08-14 09:01:51.749668] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=362008303, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:51.749681] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:51.750279] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1001}, delta=367021672, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:51.750293] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:51.750322] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=362008303, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:51.750331] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:51.750370] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:51.750484] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=8] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:51.750859] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=367022730, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:51.750869] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:51.750892] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=362009361, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:51.750900] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:51.751696] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8] get wrs ts(ls_id={id:1001}, delta=367022730, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:51.751713] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:51.752592] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:51.752696] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [622][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9225-0-0] [lt=16][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162111297150) [2025-08-14 09:01:51.752881] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [979][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F22-0-0] [lt=26][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1002, svr="172.77.0.7:2882", version={val:1755161758221798000, v:0}, valid_part_count=0, total_part_count=2, generate_timestamp=1755162111398289) [2025-08-14 09:01:51.752904] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [979][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F22-0-0] [lt=17][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1002, req={req_server:"172.77.0.7:2882", version:{val:1755161758221798000, v:0}, valid_part_count:0, total_part_count:2, generate_timestamp:1755162111398289}, twrs={inited:true, tenant_id:1002, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162111750885}, server_version_for_stat:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162111750885}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:01:51.753371] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:51.753389] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] temporary file module use no blocks [2025-08-14 09:01:51.753396] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:51.753409] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:51.753449] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623013193058, flush_io_finished_ret_:0, flush_io_finished_round_:26257, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:51.753503] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:51.753511] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=362009361, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:51.753526] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:51.755183] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:51.757693] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=8] PNIO [ratelimit] time: 1755162111757691, bytes: 0, bw: 0.000000 MB/s, add_ts: 1004514, add_bytes: 0 [2025-08-14 09:01:51.752478] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SERVER] net_in=0kB/s, net_out=0kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=1280000kB/s [2025-08-14 09:01:51.758887] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1001}, delta=367030005, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:51.758911] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:51.759615] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b9210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.744, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.759648] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919aaf90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.759660] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919aba10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.759678] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ad390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.759745] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=65][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196cb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.759760] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196b810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:50.740880] EDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4038] LOGGER COST TOO MUCH TIME, cost: 1018432, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1018428, ALLOC_END=0, APPEND_END=4, ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=2], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1ba227a3 0x1b89078a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:51.752716] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [622][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9225-0-0] [lt=20][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162111297150}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162111197102}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162111197102}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:01:50.740880] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:01:51.752575] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:51.762264] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:51.762982] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:51.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.746573] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:51.769809] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ad210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.751, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.770114] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:51.770266] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] get_fetch_log_dst failed or invalid(ret=0, palf_id=1, self="172.77.0.7:2882", fetch_log_dst="172.77.0.7:2882") [2025-08-14 09:01:50.772897] INFO [COMMON] handle (ob_io_struct.cpp:3057) [515][T1_DiskCB0][T1][YB42AC4D0007-00063C495D6A02D9-0-0] [lt=52100] callback runner call handle(get_queue_count()=0, req={is_inited_:true, tenant_id_:1, control_block_:0x7f99662ae300, ref_cnt_:1, raw_buf_:0x7f98af0263e0, fd_:{first_id:1152921504606978093, second_id:6330, third_id:0, fd_id:-1, slot_version:-1, device_handle:0x7f9969de7130}, is_object_device_req():false, trace_id_:YB42AC4D0007-00063C495D6A02D9-0-0, retry_count_:0, tenant_io_mgr_:{ptr:0x7f994b36a030}, storage_accesser:{ptr:null}, io_result_:{is_inited_:true, is_finished_:false, is_canceled_:false, has_estimated_:false, complete_size_:4096, offset_:1802570, size_:921, timeout_us_:10000000, result_ref_cnt_:3, out_ref_cnt_:2, flag_:{mode:"READ", group_id_:0, func_type_:0, wait_event_id_:3, is_sync_:false, is_unlimited_:false, is_detect_:false, is_write_through_:false, is_sealed_:true, is_time_detect_:false, need_close_dev_and_fd_:false, reserved_:0}, ret_code_:{io_ret_:0, fs_errno_:0}, tenant_id_:1, tenant_io_mgr_:{ptr:0x7f994b36a030}, user_data_buf_:null, buf_:null, io_callback_:0x7f9858859400, time_log:{begin_ts:1755162108745057, enqueue_used:975225, dequeue_used:427, submit_used:1, return_used:46449, callback_enqueue_used:986995, callback_dequeue_used:10387, callback_finish_used:-1, end_used:-1}}, part_id:-1}) [2025-08-14 09:01:51.766303] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19698][errcode=0] epoll handler too much time :19735 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:51.752620] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2140] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:51.774495] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=21840] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:51.774511] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:51.774521] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:51.774530] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:51.774539] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:51.774549] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:51.774560] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:51.774569] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:51.774580] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:01:51.774587] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:01:51.747289] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [657][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=16836][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:0, time_interval:100.00ms, expected_run_time:09:01:51.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.776066] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [657][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=28730][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:01:51.776|total=3.00s) [2025-08-14 09:01:51.752398] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=564012038, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:51.776109] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23671] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:51.776173] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EE97-0-0] [lt=22][errcode=0] cost too much time:mds_tenant_service.cpp:int oceanbase::storage::mds::ObTenantMdsTimer::get_tablet_oldest_scn_(oceanbase::storage::ObTablet &, share::SCN &), (this=|threshold=5.00ms|start at 09:01:51.763|344=12.93ms|total=12.95ms) [2025-08-14 09:01:51.776201] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EE97-0-0] [lt=22][errcode=0] cost too much time:mds_tenant_service.cpp:int oceanbase::storage::mds::ObTenantMdsTimer::process_with_tablet_(oceanbase::storage::ObTablet &), (this=|threshold=10.00ms|start at 09:01:51.763|315=0us|317=12.96ms|319=3us|total=12.98ms) [2025-08-14 09:01:51.778491] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:51.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.779875] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891825b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.781953] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=9] gc stale ls task succ [2025-08-14 09:01:51.781954] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=23] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:51.781984] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=25] temporary file module use no blocks [2025-08-14 09:01:51.781991] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:51.782009] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=17] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:51.782027] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=18] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623013221676, flush_io_finished_ret_:0, flush_io_finished_round_:26284, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:51.782222] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=12] gc stale ls task succ [2025-08-14 09:01:51.782463] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=21] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:51.782484] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=19] temporary file module use no blocks [2025-08-14 09:01:51.782489] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=4] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:51.782497] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=7] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:51.782522] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=25] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623013222205, flush_io_finished_ret_:0, flush_io_finished_round_:26293, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:51.782666] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:51.783033] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:51.789936] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919acf10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.789971] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f0790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.793280] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=1009371] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:51.799997] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919abe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.800029] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b5090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.800041] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199fe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919acd90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.800056] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e80b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891973b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.800090] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a31b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ede10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.808689] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:51.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.810068] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ab890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.806, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.810094] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196a610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.810106] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7280210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.810121] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7288510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.800, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.810140] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728e330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7290690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.805, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.810158] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f1b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.803, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.830036] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc5458, timer:0x7f9970ecbf90, task:0x564d6cd25880, task_type:"N9oceanbase10rootserver18ObSSNTAllocServiceE", scheduled_time:1755162050765815, pushed_time:0, delay:3000000}, running_task_token={this:0x7f9970fc5378, timer:0x7f9970ecbf90, task:0x564d6cd25100, task_type:"N9oceanbase10rootserver23ObIngressBWAllocServiceE", scheduled_time:1755162024729962, pushed_time:1755162047772379, delay:3000000}, total_delay=61064220, threashold=1000000, tenant_id=500) [2025-08-14 09:01:51.830205] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739e990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.824, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.830344] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:51.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.830751] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7110][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:51.840255] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891969b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.832, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.840724] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=857612][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a5998, timer:0x7f9935f8df20, task:0x7f99200f5a78, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr15MediumCheckTaskE", scheduled_time:1755161721762022, pushed_time:0, delay:20000000}, running_task_token={this:0x7f991b7a58b8, timer:0x7f9935f8df20, task:0x7f99200f5a38, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr14MediumLoopTaskE", scheduled_time:1755161718796975, pushed_time:1755161718805054, delay:120000000}, total_delay=390078701, threashold=1000000, tenant_id=1002) [2025-08-14 09:01:51.840753] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6418, timer:0x7f994b1dfb70, task:0x7f9920110068, task_type:"N9oceanbase5share22ObIndexUsageReportTaskE", scheduled_time:1755161727099859, pushed_time:0, delay:900000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=384740894, threashold=1000000, tenant_id=1002) [2025-08-14 09:01:51.840764] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6bf8, timer:0x7f994b1dfb70, task:0x7f99201e29f8, task_type:"N9oceanbase5table21ObTableGroupCommitMgr19ObTableGroupOpsTaskE", scheduled_time:1755161732717636, pushed_time:0, delay:1000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=379123128, threashold=1000000, tenant_id=1002) [2025-08-14 09:01:51.840778] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6878, timer:0x7f994b1dfb70, task:0x7f9920110178, task_type:"N9oceanbase5share27ObIndexUsageRefreshConfTaskE", scheduled_time:1755161733727027, pushed_time:0, delay:2000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=378113751, threashold=1000000, tenant_id=1002) [2025-08-14 09:01:51.840795] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6958, timer:0x7f994b1dfb70, task:0x7f9920110810, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161734741162, pushed_time:0, delay:5000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=377099633, threashold=1000000, tenant_id=1002) [2025-08-14 09:01:51.840808] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6a38, timer:0x7f994b1dfb70, task:0x7f9920110af8, task_type:"N9oceanbase5share23ObStorageIOUsageRepoter23TenantIOUsageReportTaskE", scheduled_time:1755161734803774, pushed_time:0, delay:5000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=377037034, threashold=1000000, tenant_id=1002) [2025-08-14 09:01:51.840825] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6e98, timer:0x7f994b1dfb70, task:0x7f99201075a8, task_type:"N9oceanbase3omt11ObTenantSrs27TenantSrsUpdatePeriodicTaskE", scheduled_time:1755161734810627, pushed_time:0, delay:0}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=377030198, threashold=1000000, tenant_id=1002) [2025-08-14 09:01:51.840839] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a64f8, timer:0x7f994b1dfb70, task:0x7f992010e1b0, task_type:"N9oceanbase5table20ObTableObjectPoolMgr26ObTableSessEliminationTaskE", scheduled_time:1755161735735271, pushed_time:0, delay:5000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=376105568, threashold=1000000, tenant_id=1002) [2025-08-14 09:01:51.840856] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a65d8, timer:0x7f994b1dfb70, task:0x7f992010e1c8, task_type:"N9oceanbase5table20ObTableObjectPoolMgr27ObTableSessSysVarUpdateTaskE", scheduled_time:1755161735797729, pushed_time:0, delay:5000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=376043127, threashold=1000000, tenant_id=1002) [2025-08-14 09:01:51.840870] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a66b8, timer:0x7f994b1dfb70, task:0x7f992010e1e0, task_type:"N9oceanbase5table20ObTableObjectPoolMgr28ObTableReqResEliminationTaskE", scheduled_time:1755161735811808, pushed_time:0, delay:5000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=376029062, threashold=1000000, tenant_id=1002) [2025-08-14 09:01:51.840897] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a4ff8, timer:0x7f994b1dfb70, task:0x7f9922ff3e50, task_type:"N9oceanbase3sql3dtl23ObDTLIntermResultGCTaskE", scheduled_time:1755161742809616, pushed_time:0, delay:10000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=369031281, threashold=1000000, tenant_id=1002) [2025-08-14 09:01:51.840911] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a5b58, timer:0x7f99535c3f30, task:0x7f9922ffee90, task_type:"N9oceanbase7storage21ObTenantFreezeInfoMgr24UpdateLSResvSnapshotTaskE", scheduled_time:1755161744768148, pushed_time:0, delay:10000000}, running_task_token={this:0x7f991b7a5a78, timer:0x7f99535c3f30, task:0x7f9922ffee78, task_type:"N9oceanbase7storage21ObTenantFreezeInfoMgr10ReloadTaskE", scheduled_time:1755161737748006, pushed_time:1755161737749908, delay:3000000}, total_delay=367072762, threashold=1000000, tenant_id=1002) [2025-08-14 09:01:51.840935] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a7838, timer:0x7f994b1dfb70, task:0x7f9920110820, task_type:"N9oceanbase10rootserver16ObMViewMdsOpTaskE", scheduled_time:1755161749755625, pushed_time:0, delay:5000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=362085310, threashold=1000000, tenant_id=1002) [2025-08-14 09:01:51.840950] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a74b8, timer:0x7f994b1dfb70, task:0x7f9920110730, task_type:"N9oceanbase10rootserver23ObMViewPushSnapshotTaskE", scheduled_time:1755161804754766, pushed_time:0, delay:60000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=307086184, threashold=1000000, tenant_id=1002) [2025-08-14 09:01:51.840966] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a7218, timer:0x7f994b1dfb70, task:0x7f99201107f8, task_type:"N9oceanbase10rootserver24ObMViewCleanSnapshotTaskE", scheduled_time:1755161804755624, pushed_time:0, delay:60000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=307085342, threashold=1000000, tenant_id=1002) [2025-08-14 09:01:51.840980] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6098, timer:0x7f99200ff180, task:0x7f99200ff208, task_type:"N9oceanbase8observer18ObTableLoadService25ObClientTaskAutoAbortTaskE", scheduled_time:1755162050765334, pushed_time:0, delay:1000000}, running_task_token={this:0x7f991b7a5fb8, timer:0x7f99200ff180, task:0x7f99200ff1f0, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162050755138, pushed_time:1755162084760013, delay:1000000}, total_delay=61075646, threashold=1000000, tenant_id=1002) [2025-08-14 09:01:51.843308] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:51.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.870487] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ade10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.864, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.870531] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891921990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.870548] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919adc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.870566] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919adb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.873434] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] EASY SLOW: thread: 7f99545f7300, lastrun: 1755162111.747175 cost: 0.126257 loop:26482, slowcnt: 1 [2025-08-14 09:01:51.880562] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72865b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7288990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.878, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.881347] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:01:51.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.900645] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728de30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7290b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.899, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.910671] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7ee210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:51.902, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:51.930410] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:51.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.930764] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:51.943383] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:51.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.965449] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=26] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:01:51.994267] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=55][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:51.989, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.014392] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:52.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.029998] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:52.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.030475] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:51.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.030810] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:52.043441] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:51.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.045162] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:51.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.094847] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:52.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.122566] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7289d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.111, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.130538] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:52.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.130870] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:52.143474] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:52.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.145201] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:52.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.215468] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:52.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.229608] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:52.229634] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=25] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3796) [2025-08-14 09:01:52.229644] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:01:52.229659] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:52.229668] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3797) [2025-08-14 09:01:52.229676] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:01:52.229860] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:52.230317] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:01:52.230346] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:01:52.230363] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:01:52.230379] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:00:41.841, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=08:59:41.841], idx=471, next_allow_print_ts="09:01:11.841") [2025-08-14 09:01:52.230392] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:01:52.230411] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:01:52.230424] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:01:52.230443] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:01:52.230455] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:01:49.731, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:01:42.751], idx=659, next_allow_print_ts="09:02:19.731") [2025-08-14 09:01:52.230468] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:01:52.230484] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:01:52.230497] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:01:52.230509] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:00:52.998, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:22.998], idx=771, next_allow_print_ts="09:01:22.998") [2025-08-14 09:01:52.230528] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=08:59:57.942, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:27.942], idx=868, next_allow_print_ts="09:00:27.942") [2025-08-14 09:01:52.230604] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:52.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.230703] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:01:52.230719] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:01:52.230732] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:01:52.230896] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:52.243517] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:52.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.245256] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:52.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.245652] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:52.239, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.265714] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:52.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.272247] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:52.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.302411] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:52.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.330672] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:52.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.330975] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:52.343581] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:52.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.345312] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:52.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.441372] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a4e38, timer:0x7f99535f3df0, task:0x7f9922ff6bf0, task_type:"N9oceanbase7obmysql15ObEliminateTaskE", scheduled_time:1755161931284195, pushed_time:0, delay:500000}, running_task_token={this:0x7f991b7a4f18, timer:0x7f99535f3df0, task:0x7f9922ff6c40, task_type:"N9oceanbase7obmysql20ObConstructQueueTaskE", scheduled_time:1755161931282757, pushed_time:1755161931285404, delay:500000}, total_delay=181157176, threashold=1000000, tenant_id=1002) [2025-08-14 09:01:52.443622] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:52.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.445373] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:52.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.465810] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fa190, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa470, time_wheel:0x7f99341cdbf0, thread_pool:0x7f99341cd3d0, is_running:1, time_interval:2.00s, expected_run_time:09:01:52.334, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.497038] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:52.489, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.517198] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:52.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.523664] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:52.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.543669] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:52.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.545428] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:52.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.565404] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fe410, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe6f0, time_wheel:0x7f9922df9d40, thread_pool:0x7f9922df9520, is_running:1, time_interval:2.00s, expected_run_time:09:01:52.432, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.597686] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:52.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.643730] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:52.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.645488] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:52.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.684363] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571634a0, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957163780, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:5.00s, expected_run_time:09:01:52.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.684400] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:01:52.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.718412] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:52.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.719906] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:52.721872] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:51.759302] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:51.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.730477] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=969705][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:51.759|207=971.17ms|210=0us|222=1us|total=971.19ms) [2025-08-14 09:01:52.730498] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task use too much time(end=1755162112730497, start=1755162111759300, delta=971197) [2025-08-14 09:01:52.730516] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=746983, expected_run_ts_us_=1755162111983531, ObClockGenerator::getRealClock()=1755162112730515, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:51.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.730533] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:51.983, task_priority:1, with_handle_protected:1}) [651]OB_MALLOC COST TOO MUCH TIME, cost_time=951742, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:01:51.778442] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EE97-0-0] [lt=17] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-08-14 09:01:52.731660] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EE97-0-0] [lt=951771][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571fc8b0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcb90, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:0, time_interval:3.00s, expected_run_time:09:01:52.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.731684] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EE97-0-0] [lt=23][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:01:52.731|total=1.97s) [387]OB_MALLOC COST TOO MUCH TIME, cost_time=983781, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=5443, WASH_BLOCK_END=978335, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1, 2, 3, 4] [2025-08-14 09:01:52.731717] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=1549] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 444993536 used= 389560096 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 70 washed_blocks= 287 washed_size= 70221824 [MEMORY] request_cached_chunk_cnt= 4 [MEMORY] hold= 389560096 used= 379314179 count= 32870 avg_used= 11539 mod=SUMMARY [2025-08-14 09:01:52.731757] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=18] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 230686720 used= 228884480 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 block_cnt= 110 chunk_cnt= 110 mod=Memstore [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:01:52.731771] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= EXECUTE_CTX_ID hold= 1163264 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 15 washed_size= 13516800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:52.731786] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:01:52.731801] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 53714944 used= 47487424 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 84 washed_blocks= 334 washed_size= 126640128 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 47487424 used= 43098388 count= 3994 avg_used= 10790 mod=SUMMARY [2025-08-14 09:01:52.731814] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= WORK_AREA hold= 1458176 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 13 washed_blocks= 66 washed_size= 25804800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:52.731829] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 14204928 used= 10970224 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 277 washed_size= 31932416 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10970224 used= 6078924 count= 18573 avg_used= 327 mod=SUMMARY [2025-08-14 09:01:52.731845] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 111149056 used= 108896256 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 block_cnt= 211 chunk_cnt= 53 mod=CoStack [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 mod=SUMMARY [2025-08-14 09:01:52.731865] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 56623104 used= 56623104 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 56623104 used= 0 count= 27 avg_used= 0 mod=SUMMARY [2025-08-14 09:01:52.731881] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 34365376 limit= 429496720 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 34365376 used= 31086608 count= 475 avg_used= 65445 mod=SUMMARY [2025-08-14 09:01:52.731896] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 8003584 used= 4947392 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 10 washed_blocks= 23 washed_size= 12967936 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4947392 used= 4793344 count= 604 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:52.731910] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:52.731924] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 65536 used= 31024 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 2031616 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 31024 used= 27058 count= 20 avg_used= 1352 mod=SUMMARY [2025-08-14 09:01:52.731937] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= RPC_CTX_ID hold= 6438912 used= 1835008 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1835008 used= 1202176 count= 80 avg_used= 15027 mod=SUMMARY [2025-08-14 09:01:52.731953] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 3481600 used= 2391232 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 10 washed_size= 4907008 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2391232 used= 2367040 count= 36 avg_used= 65751 mod=SUMMARY [2025-08-14 09:01:52.731970] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 3964928 used= 3080192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 3 washed_size= 229376 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3080192 used= 2983936 count= 376 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:52.731986] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= SCHEMA_SERVICE hold= 212992 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1884160 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:52.732001] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:52.732015] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= MERGE_NORMAL_CTX_ID hold= 884736 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 6 washed_blocks= 9 washed_size= 11698176 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:52.684432] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:1, time_interval:3.00s, expected_run_time:09:01:52.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:51.760886] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1018432] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:01:52.732164] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=0] get_fetch_log_dst failed or invalid(ret=0, palf_id=1, self="172.77.0.7:2882", fetch_log_dst="172.77.0.7:2882") [2025-08-14 09:01:51.776145] INFO [PALF] handle_register_parent_resp (log_config_mgr.cpp:2766) [15442][T1001_L0_G1][T1001][YB42AC4D0007-00063C496CD99C88-0-0] [lt=22] handle_register_parent_resp finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.77.0.7:2882", server={server:"172.77.0.7:2882", region:"default_region", register_time_us:1755162099867155, keepalive_ts:-1}, candidate_list={learner_num:0, learner_array:[]}, reg_ret=3) [2025-08-14 09:01:51.747187] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:52.732489] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=983812][errcode=0] epoll handler too much time :985312 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:52.229872] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=11] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3798) [2025-08-14 09:01:52.732881] WDIAG [PALF] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [15442][T1001_L0_G1][T1001][YB42AC4D0007-00063C496CD99C88-0-0] [lt=954755][errcode=0] cost too much time:log_rpc_processor.h:int oceanbase::palf::LogRegisterParentRespP::process_impl_() [FLAG = false], (this=|threshold=50.00ms|start at 09:01:52.732|total=1.97s) [2025-08-14 09:01:52.733043] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 500, limit: 9223372036854775807 hold: 698023936 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 332902400 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 39223296 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 85983232 limit= 9223372036854775807 [MEMORY] ctx_id= LIBEASY hold_bytes= 3145728 limit= 9223372036854775807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 18874368 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 2752512 limit= 9223372036854775807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 37232640 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 78925824 limit= 9223372036854775807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 98983936 limit= 9223372036854775807 [2025-08-14 09:01:52.733076] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 332902400 used= 301206128 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 501 washed_blocks= 2233 washed_size= 927375360 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 301206128 used= 293802378 count= 12658 avg_used= 23210 mod=SUMMARY [2025-08-14 09:01:52.733101] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 500 ctx_id= GLIBC hold= 39223296 used= 21964400 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 181 washed_blocks= 813 washed_size= 357138432 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 21964400 used= 10365221 count= 131705 avg_used= 78 mod=SUMMARY [2025-08-14 09:01:52.733114] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 500 ctx_id= CO_STACK hold= 85983232 used= 83091456 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 83091456 used= 82936896 count= 161 avg_used= 515136 block_cnt= 161 chunk_cnt= 41 mod=CoStack [MEMORY] hold= 83091456 used= 82936896 count= 161 avg_used= 515136 mod=SUMMARY [2025-08-14 09:01:52.733124] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 500 ctx_id= LIBEASY hold= 3145728 used= 3070816 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1048576 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3070816 used= 2859784 count= 223 avg_used= 12824 mod=SUMMARY [2025-08-14 09:01:52.733135] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 500 ctx_id= LOGGER_CTX_ID hold= 20971520 used= 16646144 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 16646144 used= 16637952 count= 8 avg_used= 2079744 mod=SUMMARY [2025-08-14 09:01:52.733156] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 500 ctx_id= RPC_CTX_ID hold= 2752512 used= 1286144 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 50 washed_blocks= 63 washed_size= 102105088 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1286144 used= 849280 count= 53 avg_used= 16024 mod=SUMMARY [2025-08-14 09:01:52.733167] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 500 ctx_id= PKT_NIO hold= 37232640 used= 35792384 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 42 washed_blocks= 63 washed_size= 82591744 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 35792384 used= 34871392 count= 116 avg_used= 300615 mod=SUMMARY [2025-08-14 09:01:52.733200] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=15] [MEMORY] tenant_id= 500 ctx_id= SCHEMA_SERVICE hold= 78925824 used= 68904576 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 22 washed_size= 5767168 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 68904576 used= 67642943 count= 4856 avg_used= 13929 mod=SUMMARY [2025-08-14 09:01:52.733212] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 500 ctx_id= UNEXPECTED_IN_500 hold= 98983936 used= 97126736 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 97126736 used= 86745504 count= 3006 avg_used= 28857 mod=SUMMARY [2025-08-14 09:01:52.733224] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant: 508, limit: 1073741824 hold: 15613952 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 5128192 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10485760 limit= 9223372036854775807 [2025-08-14 09:01:52.733242] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 508 ctx_id= DEFAULT_CTX_ID hold= 5128192 used= 3765328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3765328 used= 3723912 count= 84 avg_used= 44332 mod=SUMMARY [2025-08-14 09:01:52.733263] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 508 ctx_id= CO_STACK hold= 10485760 used= 10321920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10321920 used= 10302720 count= 20 avg_used= 515136 mod=SUMMARY [2025-08-14 09:01:52.733306] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant: 509, limit: 1073741824 hold: 15613952 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 5128192 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10485760 limit= 9223372036854775807 [2025-08-14 09:01:52.733313] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 509 ctx_id= DEFAULT_CTX_ID hold= 5128192 used= 3765328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3765328 used= 3723912 count= 84 avg_used= 44332 mod=SUMMARY [2025-08-14 09:01:52.733334] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 509 ctx_id= CO_STACK hold= 10485760 used= 10321920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10321920 used= 10302720 count= 20 avg_used= 515136 mod=SUMMARY [2025-08-14 09:01:52.733380] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant: 1001, limit: 1073741824 hold: 606650368 cache_hold: 44040192 cache_used: 44040192 cache_item_count: 21 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 302309376 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 54525952 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 3080192 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 483328 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 45514752 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 5521408 limit= 157286400 [MEMORY] ctx_id= GLIBC hold_bytes= 15499264 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 73424896 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35651584 limit= 214748360 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 3252224 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 12582912 limit= 1073741800 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 5570560 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3866624 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 1179648 limit= 9223372036854775807 [2025-08-14 09:01:52.733401] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= DEFAULT_CTX_ID hold= 302309376 used= 268966352 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 81 washed_blocks= 406 washed_size= 113410048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 268966352 used= 262851302 count= 3785 avg_used= 69445 mod=SUMMARY [2025-08-14 09:01:52.733410] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1001 ctx_id= MEMSTORE_CTX_ID hold= 54525952 used= 54099968 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 54099968 used= 54075216 count= 26 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:01:52.733416] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= EXECUTE_CTX_ID hold= 3080192 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 8 washed_size= 7405568 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:52.733423] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1001 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:01:52.733431] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1001 ctx_id= PLAN_CACHE_CTX_ID hold= 45514752 used= 36449984 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 77 washed_blocks= 201 washed_size= 132743168 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 36449984 used= 32916258 count= 3312 avg_used= 9938 mod=SUMMARY [2025-08-14 09:01:52.733437] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1001 ctx_id= WORK_AREA hold= 5521408 used= 0 limit= 157286400 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 9 washed_blocks= 40 washed_size= 17547264 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:52.733445] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= GLIBC hold= 15499264 used= 10889728 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 25 washed_blocks= 164 washed_size= 47415296 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10889728 used= 6020141 count= 18313 avg_used= 328 mod=SUMMARY [2025-08-14 09:01:52.733453] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1001 ctx_id= CO_STACK hold= 73424896 used= 71221248 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 32 washed_blocks= 34 washed_size= 2072576 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 71221248 used= 71088768 count= 138 avg_used= 515136 block_cnt= 138 chunk_cnt= 36 mod=CoStack [MEMORY] hold= 71221248 used= 71088768 count= 138 avg_used= 515136 mod=SUMMARY [2025-08-14 09:01:52.733464] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1001 ctx_id= KVSTORE_CACHE_ID hold= 44040192 used= 44040192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 44040192 used= 0 count= 21 avg_used= 0 mod=SUMMARY [2025-08-14 09:01:52.733471] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1001 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 33906624 limit= 214748360 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33906624 used= 30629648 count= 468 avg_used= 65447 mod=SUMMARY [2025-08-14 09:01:52.733478] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1001 ctx_id= TX_CALLBACK_CTX_ID hold= 3252224 used= 2342592 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 3 washed_blocks= 8 washed_size= 3039232 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2342592 used= 2269696 count= 286 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:52.733485] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1001 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:52.733495] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= RPC_CTX_ID hold= 12582912 used= 401408 limit= 1073741800 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 401408 used= 264064 count= 17 avg_used= 15533 mod=SUMMARY [2025-08-14 09:01:52.733504] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1001 ctx_id= TX_DATA_TABLE hold= 5570560 used= 2342784 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 8 washed_size= 2818048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2342784 used= 2319424 count= 30 avg_used= 77314 mod=SUMMARY [2025-08-14 09:01:52.733512] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1001 ctx_id= MDS_DATA_ID hold= 3866624 used= 2785280 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 5 washed_size= 4521984 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2785280 used= 2698240 count= 340 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:52.733526] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= MERGE_NORMAL_CTX_ID hold= 1179648 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 8 washed_blocks= 24 washed_size= 15597568 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:52.733541] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant: 1002, limit: 8589934592 hold: 1943941120 cache_hold: 48234496 cache_used: 48234496 cache_item_count: 23 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 1007513600 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 415236096 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 1114112 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 950272 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 211812352 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 1441792 limit= 429496725 [MEMORY] ctx_id= GLIBC hold_bytes= 12582912 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 149913600 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 37748736 limit= 1717986900 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 7766016 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 5980160 limit= 8589934500 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 5718016 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3047424 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 1032192 limit= 9223372036854775807 [2025-08-14 09:01:52.733569] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1002 ctx_id= DEFAULT_CTX_ID hold= 1007513600 used= 916421104 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 13 [MEMORY] hold= 185860096 used= 185534336 count= 114 avg_used= 1627494 block_cnt= 114 chunk_cnt= 104 mod=MysqlRequesReco [MEMORY] hold= 167792640 used= 167772192 count= 1 avg_used= 167772192 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 83927040 used= 83886080 count= 2 avg_used= 41943040 block_cnt= 2 chunk_cnt= 2 mod=LogGroupBuffer [MEMORY] hold= 67190784 used= 67109344 count= 4 avg_used= 16777336 block_cnt= 4 chunk_cnt= 4 mod=MrgMemPoolChk [MEMORY] hold= 916421104 used= 899221089 count= 9449 avg_used= 95165 mod=SUMMARY [2025-08-14 09:01:52.733580] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= MEMSTORE_CTX_ID hold= 415236096 used= 411992064 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 411992064 used= 411803568 count= 198 avg_used= 2079816 block_cnt= 198 chunk_cnt= 198 mod=Memstore [MEMORY] hold= 411992064 used= 411803568 count= 198 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:01:52.733588] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1002 ctx_id= EXECUTE_CTX_ID hold= 1114112 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 7 washed_size= 7274496 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:52.733596] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= TRANS_CTX_MGR_ID hold= 950272 used= 933888 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1146880 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 933888 used= 921984 count= 2 avg_used= 460992 mod=SUMMARY [2025-08-14 09:01:52.733606] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= PLAN_CACHE_CTX_ID hold= 211812352 used= 201995008 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 176930880 used= 160228962 count= 13450 avg_used= 11912 block_cnt= 13450 chunk_cnt= 101 mod=SqlPhyPlan [MEMORY] hold= 201995008 used= 183048143 count= 17906 avg_used= 10222 mod=SUMMARY [2025-08-14 09:01:52.733613] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= WORK_AREA hold= 1441792 used= 0 limit= 429496725 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 14 washed_blocks= 54 washed_size= 27918336 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:52.733622] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1002 ctx_id= GLIBC hold= 12582912 used= 11425280 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 11425280 used= 6322556 count= 19216 avg_used= 329 mod=SUMMARY [2025-08-14 09:01:52.733632] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= CO_STACK hold= 149913600 used= 147087360 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 68 washed_blocks= 70 washed_size= 3178496 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 147087360 used= 146813760 count= 285 avg_used= 515136 block_cnt= 285 chunk_cnt= 73 mod=CoStack [MEMORY] hold= 147087360 used= 146813760 count= 285 avg_used= 515136 mod=SUMMARY [2025-08-14 09:01:52.733645] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= KVSTORE_CACHE_ID hold= 48234496 used= 48234496 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 48234496 used= 0 count= 23 avg_used= 0 mod=SUMMARY [2025-08-14 09:01:52.733654] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1002 ctx_id= META_OBJ_CTX_ID hold= 37748736 used= 35741632 limit= 1717986900 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 35741632 used= 32457488 count= 496 avg_used= 65438 mod=SUMMARY [2025-08-14 09:01:52.733662] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= TX_CALLBACK_CTX_ID hold= 7766016 used= 6934400 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 17 washed_size= 6914048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 6934400 used= 6729728 count= 848 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:52.733671] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1002 ctx_id= LOB_CTX_ID hold= 147456 used= 16384 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 16384 used= 15872 count= 2 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:52.733682] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1002 ctx_id= RPC_CTX_ID hold= 5980160 used= 1720320 limit= 8589934500 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 6 washed_blocks= 6 washed_size= 10797056 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1720320 used= 1124864 count= 76 avg_used= 14800 mod=SUMMARY [2025-08-14 09:01:52.733693] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= TX_DATA_TABLE hold= 5718016 used= 4521856 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 9 washed_blocks= 16 washed_size= 13156352 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4521856 used= 4480128 count= 40 avg_used= 112003 mod=SUMMARY [2025-08-14 09:01:52.733703] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= MDS_DATA_ID hold= 3047424 used= 1998848 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 4 washed_size= 1146880 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1998848 used= 1936384 count= 244 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:52.733711] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1002 ctx_id= MDS_CTX_ID hold= 147456 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:52.733723] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:52.733731] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1002 ctx_id= MERGE_NORMAL_CTX_ID hold= 1032192 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 12 washed_size= 13647872 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:52.737292] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:52.739177] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=353631762, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=251618, succ_cluster_heartbeat_count=1475, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161006924036004, v:0}, local_cluster_delta=1104929524, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:01:52.739287] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=31] get wrs ts(ls_id={id:1}, delta=362996928, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:52.739301] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:00:16.745976] EDIAG [LIB] alloc (ob_malloc_allocator.cpp:924) [471][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=1985673][errcode=-4013] LOGGER COST TOO MUCH TIME, cost: 1021350, owner: ObLog, click_count: 3, time dist:[FORMAT_END=979822, ALLOC_END=41524, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0xf9b6d19 0xf9b6367 0xf9b6058 0xf989e6a 0x973c6f6 0x973c18d 0x7f9973af5dfa 0x7f9973af5efd 0x9796af8 0x1fb130a7 0x98164f4 0x9816209 0x981582f 0x1bdf8786 0x1fb1c7eb 0x1fb1ca60 0x1231e896 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:52.739145] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=564038691, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=253019, succ_cluster_heartbeat_count=2182, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161000606550000, v:0}, local_cluster_delta=1111170657, force_self_check=true, weak_read_refresh_interval=100000) [2025-08-14 09:01:52.739856] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1001}, delta=368011463, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:52.739875] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:52.739910] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=362998094, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:52.739927] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:52.740289] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=537] get wrs ts(ls_id={id:1}, delta=565000771, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:52.740315] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:52.740907] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=565001856, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:52.740922] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:52.741479] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=565001856, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:52.741516] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=25] current monitor number(seq_=-1) [2025-08-14 09:01:52.741606] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:52.742748] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:52.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.745346] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:52.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.745571] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:52.747279] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:52.748629] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:52.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.748645] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8095] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:52.748699] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891968390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.748756] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=54][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ac790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.748775] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891991430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196ce10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.749604] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:52.749658] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=13] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:01:52.750413] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:52.750584] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=10] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:52.751395] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1001}, delta=368012548, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:52.751438] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=40] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:52.741492] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 08:59:45.720658] EDIAG [SQL] start_stmt (ob_result_set.cpp:348) [1029][ObStorageHADiag][T1][YB42AC4D0007-00063C4966799D69-0-0] [lt=2068113][errcode=-4283] LOGGER COST TOO MUCH TIME, cost: 126840312, owner: ObLog, click_count: 7, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=1495, ObMalloc#start=126838799, ObMalloc#end=4, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x147c2277 0x933771a 0x932e8ab 0x96ac524 0x96a672d 0x96a5033 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x2044675f 0x1d70b480 0x1d7075ca 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:52.732575] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=501942][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 08:59:45.720658] WDIAG [SQL] start_stmt (ob_result_set.cpp:348) [1029][ObStorageHADiag][T1][YB42AC4D0007-00063C4966799D69-0-0] [lt=2068113][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:483, schema_version:1753069640157096, object_type:1, is_db_explicit:false, is_existed:true}]) [2025-08-14 09:01:52.752896] WDIAG [SQL] do_open_plan (ob_result_set.cpp:575) [1029][ObStorageHADiag][T1][YB42AC4D0007-00063C4966799D69-0-0] [lt=126840312][errcode=-4283] fail start stmt(ret=-4283) [2025-08-14 09:01:52.752906] WDIAG [SQL] open (ob_result_set.cpp:137) [1029][ObStorageHADiag][T1][YB42AC4D0007-00063C4966799D69-0-0] [lt=9][errcode=-4283] execute plan failed(ret=-4283) [2025-08-14 09:01:52.752919] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=26] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:01:52.732319] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=2010443] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:52.732944] WDIAG pkts_wq_flush (packet_server.c:52) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] PNIO rpc resp is expired, expire_us=1755162111765773, sock_id=131071 [2025-08-14 09:01:52.748953] WDIAG [SQL] open_plan (ob_result_set.cpp:104) [1003][BlackListServic][T1][YB42AC4D0007-00063C495F09BD68-0-0] [lt=994156][errcode=-4012] query is timeout(ret=-4012, timeout_ts=1755162082740150, start_time=1755162080764848) [2025-08-14 09:01:52.753013] WDIAG [SQL] open (ob_result_set.cpp:137) [1003][BlackListServic][T1][YB42AC4D0007-00063C495F09BD68-0-0] [lt=4053][errcode=-4012] execute plan failed(ret=-4012) [2025-08-14 09:01:52.754316] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:52.754893] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8611] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:52.754920] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:52.750315] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=10] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:01:52.745327] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:52.755534] WDIAG ev_invoke_pending (ev.c:2136) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10190][errcode=0] epoll handler too much time :10224 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:52.732284] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=2002045] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:52.733071] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:01:52.756398] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23276][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:01:52.756422] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:01:52.756432] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:00:41.841, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=08:59:41.841], idx=471, next_allow_print_ts="09:01:11.841") [2025-08-14 09:01:52.756443] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:01:52.756542] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:52.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.757265] WDIAG [COORDINATOR] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [471][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=1021350][errcode=0] cost too much time:ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_(), (this=|threshold=1.00s|start at 09:01:52.757|total=130.92s) [2025-08-14 09:01:52.757282] WDIAG [COORDINATOR] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [471][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] cost too much time:ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_failure(), (this=|threshold=1.00s|start at 09:01:52.757|total=130.92s) [2025-08-14 09:01:52.757291] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [471][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:0, time_interval:100.00ms, expected_run_time:09:01:52.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.757303] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [471][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:01:52.757|total=130.92s) [2025-08-14 09:01:52.756468] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:01:52.757389] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=916][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:01:52.757408] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:01:52.757419] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:01:52.757432] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:01:52.757442] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:01:52.757451] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:00:52.998, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:22.998], idx=771, next_allow_print_ts="09:01:22.998") [2025-08-14 09:01:52.761521] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [1003][BlackListServic][T1][YB42AC4D0007-00063C495F09BD68-0-0] [lt=5][errcode=-4012] open result set failed(ret=-4012) [2025-08-14 09:01:52.757467] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=08:59:57.942, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:27.942], idx=868, next_allow_print_ts="09:00:27.942") [2025-08-14 09:01:52.761699] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=4071][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:01:52.761720] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:01:52.761736] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:01:52.764328] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=44] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:52.765793] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=22] PNIO [ratelimit] time: 1755162112765790, bytes: 0, bw: 0.000000 MB/s, add_ts: 1008099, add_bytes: 0 [2025-08-14 09:01:52.765827] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919aac90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.765850] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b9510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.765862] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ad090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.765873] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e62b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891972910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.765883] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73940b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7ee690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.765894] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7eeb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.765904] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73936b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7eec90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.765914] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f0c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.771587] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=1040140] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:52.771606] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=19] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:52.771613] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:52.771618] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:52.771624] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:52.771629] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:52.771634] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:52.771640] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:52.771645] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=4] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:01:52.771653] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:01:52.771659] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:01:52.771682] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:52.772392] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:52.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.772459] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:52.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.761446] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=23246] tx gc loop thread is running(MTL_ID()=1) [2025-08-14 09:01:52.761461] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=20607] tx gc loop thread is running(MTL_ID()=1001) [2025-08-14 09:01:52.774248] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [1029][ObStorageHADiag][T1][YB42AC4D0007-00063C4966799D69-0-0] [lt=6][errcode=-4283] open result set failed(ret=-4283) [2025-08-14 09:01:52.431024] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:52.430732] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:52.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.774542] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=342996][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:52.774544] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=343290][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:52.430|207=343.81ms|210=0us|222=2us|total=343.82ms) [2025-08-14 09:01:52.741600] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=969777] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:52.774560] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4389] timer task use too much time(end=1755162112774559, start=1755162112430730, delta=343829) [2025-08-14 09:01:52.774251] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=12770] try gc retain ctx [2025-08-14 09:01:52.774572] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=32919][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162112741508], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:52.774587] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=335] try to retry start_working [2025-08-14 09:01:52.774589] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162112741508], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:52.774605] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162112741508]}) [2025-08-14 09:01:52.774618] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts statistics(tenant_id=1001, gts_rpc_cnt=0, get_gts_cache_cnt=17, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:01:52.774631] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162112741508]}) [2025-08-14 09:01:52.774642] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1001}) [2025-08-14 09:01:52.774652] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1001}) [2025-08-14 09:01:52.774828] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:01:52.774829] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:801) [1029][ObStorageHADiag][T1][YB42AC4D0007-00063C4966799D69-0-0] [lt=28][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"select time_to_usec(gmt_create) as create_ts from __all_storage_ha_error_diagnose_history where gmt_create > usec_to_time(0) order by gmt_create limit 100"}) [2025-08-14 09:01:52.774843] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=14] get number(ret=-4038, service_type_=0, range=1, base_id=1755162112769239000, start_id=0, end_id=140295968341544) [2025-08-14 09:01:52.774230] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12763] try gc retain ctx [2025-08-14 09:01:52.774928] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=695] try to retry start_working [2025-08-14 08:58:46.794400] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:763) [784][T1002_TBalance][T1002][YB42AC4D0007-00063C4964C9A63F-0-0] [lt=11] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT distinct unit_group_id, status from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1002) order by unit_group_id "}, process_time=2908809) [2025-08-14 08:58:46.794400] EDIAG [SERVER] process_final (ob_inner_sql_connection.cpp:763) [784][T1002_TBalance][T1002][YB42AC4D0007-00063C4964C9A63F-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 185699662, owner: ObLog, click_count: 5, time dist:[FORMAT_END=185699652, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1275158c 0x96a79f6 0x96a5754 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x12586792 0x2012b496 0x2012ab70 0x104ea510 0x10539f82 0x10531684 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:52.775045] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:801) [1003][BlackListServic][T1][YB42AC4D0007-00063C495F09BD68-0-0] [lt=18][errcode=-4012] result set open failed(ret=-4012, executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}) [2025-08-14 09:01:52.775889] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ac010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.775937] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72765b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b5e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.775952] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ac610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.775979] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891969410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.775997] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f0d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.781994] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=17] gc stale ls task succ [2025-08-14 09:01:52.782124] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=32] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:52.782151] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=20] temporary file module use no blocks [2025-08-14 09:01:52.782161] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:52.782183] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=21] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:52.782205] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=21] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623014221828, flush_io_finished_ret_:0, flush_io_finished_round_:26285, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:52.782266] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=34] gc stale ls task succ [2025-08-14 09:01:52.782641] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=23] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:52.782669] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=24] temporary file module use no blocks [2025-08-14 09:01:52.782674] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=5] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:52.783647] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:52.784873] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] [sql nio session](*s={this:0x7f98f72d0030, session_id:3221589415, trace_id:YB42AC4D0007-00063C4965FA43DC-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:245}, err:0, last_decode_time:1755161751714366, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1662, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:01:52.784905] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=31] [sql nio session](*s={this:0x7f98f72d5030, session_id:3221590200, trace_id:YB42AC4D0007-00063C4965FA43E0-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:248}, err:0, last_decode_time:1755161776777167, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1593, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:01:52.785952] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919aab10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.774, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.785990] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739ef90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.786009] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a45b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7283390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.786026] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7eee10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.786044] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73931b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a1e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.792722] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=956812] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:01:52.796014] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ac490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.796044] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196c210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.796060] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7273c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196c090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.796078] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d27b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72eb290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.796097] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d94b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.796115] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7283510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.780, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.796126] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739bbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728b990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.780, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.796141] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739fa10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.796158] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f727dc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f729ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.796180] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f727d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f729af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.796193] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891921c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.802707] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:52.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.806075] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891814fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7292d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.793, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.806104] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891921b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.812906] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=936909] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:01:52.816092] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7facb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7281710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:52.820736] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=928577] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:01:52.840004] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25] get_max_applied_scn(scn={val:1755161749741193000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:50, palf_committed_end_lsn_:{lsn:1604090090}, palf_committed_end_scn_:{val:1755161749741193000, v:0}, last_check_scn_:{val:1755161749741193001, v:0}, max_applied_cb_scn_:{val:1755161749741193000, v:0}}) [2025-08-14 09:01:52.840040] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=31] get wrs ts(ls_id={id:1}, delta=363098216, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:52.840052] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:52.840669] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28] get wrs ts(ls_id={id:1001}, delta=368112598, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:52.840689] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:52.841027] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10164] get_max_applied_scn(scn={val:1755161547738516001, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:50, palf_committed_end_lsn_:{lsn:5282737655}, palf_committed_end_scn_:{val:1755161547738516001, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161547738516001, v:0}}) [2025-08-14 09:01:52.841052] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] get_min_unreplayed_log_info(lsn={lsn:5282737655}, scn={val:1755100947825033001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161624848121, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:13063726, ret_code_:0}}, next_to_submit_lsn_:{lsn:5282737655}, next_to_submit_scn_:{val:1755100947825033001, v:0}, base_lsn_:{lsn:4831543296}, base_scn_:{val:1755063445733337001, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7605000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:5282737655}, end_lsn:{lsn:5282737655}, read_buf:{buf_len_:3747840, buf_:0x7f98f7605000}, block_size:67104768, log_storage_:0x7f993e529bf0, log_storage_:{log_tail:{lsn:5282737655}, readable_log_tail:{lsn:5282737655}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:5234171904}, min_scn:{val:1755152244802535004, v:0}, curr_block_id:78, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:206, min_block_id:42, max_block_id:79, curr_writable_block_id:78}, logical_block_size_:67104768, curr_block_writable_size_:18539017, block_header_serialize_buf_:0x7f993e52a8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:85, read_io_size:301367296, read_disk_cost_ts:2364604, start_lsn:{lsn:4831543296}}}}}}) [2025-08-14 09:01:52.841484] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=869161][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb38b8, timer:0x7f9966a59cb0, task:0x7f99333bfa78, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr15MediumCheckTaskE", scheduled_time:1755161750786742, pushed_time:0, delay:20000000}, running_task_token={this:0x7f9931bb37d8, timer:0x7f9966a59cb0, task:0x7f99333bfa38, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr14MediumLoopTaskE", scheduled_time:1755161743804000, pushed_time:1755161743806666, delay:120000000}, total_delay=362054740, threashold=1000000, tenant_id=1001) [2025-08-14 09:01:52.841699] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=82] get wrs ts(ls_id={id:1}, delta=565102967, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:52.841719] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:52.845745] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=4979][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c0b90, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571c0e70, time_wheel:0x7f9959fef990, thread_pool:0x7f9959fef170, is_running:1, time_interval:3.00s, expected_run_time:09:01:52.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.863408] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=36745][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:01:52.863604] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=41414][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:01:52.891442] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [402][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=1795854][errcode=0] EASY SLOW: thread: 7f9953cfb300, lastrun: 1755162112.755196 cost: 0.136244 loop:26439, slowcnt: 1 [2025-08-14 09:01:52.940060] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=363197912, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:52.940087] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:52.940687] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1001}, delta=368211281, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:52.940704] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:52.941653] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=565202211, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:52.941671] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:52.941715] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6db8, timer:0x7f994b1dfb70, task:0x7f99201e33a0, task_type:"N9oceanbase5table20ObTableClientInfoMgr27ObTableClientInfoRetireTaskE", scheduled_time:1755161750768689, pushed_time:0, delay:60000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=362173025, threashold=1000000, tenant_id=1002) [2025-08-14 09:01:52.945442] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:52.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.950372] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:52.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.966625] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=22] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:01:52.993823] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:52.989, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.993991] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:52.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.007256] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199fbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919acc10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:52.995, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:53.013962] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:53.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.024025] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:53.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.031477] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=31][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a5a78, timer:0x7f996714bf20, task:0x7f994b3fce90, task_type:"N9oceanbase7storage21ObTenantFreezeInfoMgr24UpdateLSResvSnapshotTaskE", scheduled_time:1755161931783996, pushed_time:0, delay:10000000}, running_task_token={this:0x7f994e1a5998, timer:0x7f996714bf20, task:0x7f994b3fce78, task_type:"N9oceanbase7storage21ObTenantFreezeInfoMgr10ReloadTaskE", scheduled_time:1755161924772450, pushed_time:1755161924772520, delay:3000000}, total_delay=181247480, threashold=1000000, tenant_id=1) [2025-08-14 09:01:53.041364] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9229-0-0] [lt=41][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162113041141) [2025-08-14 09:01:53.041393] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9229-0-0] [lt=28][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162113041141}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162112941087}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162112941087}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:01:53.041478] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [317][pnio1][T1][YB42AC4D0007-00063C49607B9229-0-0] [lt=13][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1, delta=1280642, last_succ_cluster_heartbeat_tstamp_=1755162111760834) [2025-08-14 09:01:53.041744] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=565302677, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:53.041769] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:53.041800] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161101770923, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755161572801635) [2025-08-14 09:01:53.042389] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=565303736, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:53.042406] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:53.045502] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:52.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.050413] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:52.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.094407] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:53.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.142475] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=565403014, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:53.142512] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=34] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:53.145582] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:53.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.150474] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:53.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.215069] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:53.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.242534] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=565503363, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:53.242565] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:53.245280] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:53.239, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.245478] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:53.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.245644] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:53.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.250538] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:53.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.251837] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=19153] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:53.251863] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=24] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3799) [2025-08-14 09:01:53.251872] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:01:53.251894] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=16] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:53.251906] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=11] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3800) [2025-08-14 09:01:53.251919] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:01:53.252088] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:53.252108] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:227) [1031][PxTargetMgr0][T1002][Y0-0000000000000000-0-0] [lt=16] refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3801) [2025-08-14 09:01:53.252119] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:01:53.255543] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:53.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.261900] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:01:53.261930] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:01:53.265403] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:53.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.275544] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:53.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.305703] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:53.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.322511] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [571][T1_ArcSender][T1][YB42AC4D0007-00063C495BA99C16-0-0] [lt=27][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:01:53.342616] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=565603383, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:53.342643] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:53.345718] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:53.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.350578] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:53.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.445781] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:53.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.450640] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:53.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.496892] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:53.489, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.497102] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:53.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.517001] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:53.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.527042] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:53.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.545808] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:53.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.550676] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:53.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.597513] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:53.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.645838] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:53.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.650748] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:53.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.684933] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:01:53.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.774951] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] finish update min start scn(max_decided_scn={val:1755161547738516001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1755161493768447002, v:0}, keep_alive_scn_:{val:1755161493768447002, v:0}, update_ts_:1755162112769239}) [2025-08-14 09:01:53.716545] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:946) [1029][ObStorageHADiag][T1][YB42AC4D0007-00063C4966799D69-0-0] [lt=26][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"select time_to_usec(gmt_create) as create_ts from __all_storage_ha_error_diagnose_history where gmt_create > usec_to_time(0) order by gmt_create limit 100"}, retry_cnt=0, local_sys_schema_version=1753073979411448, local_tenant_schema_version=1753073979411448) [2025-08-14 09:01:53.716590] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1018) [784][T1002_TBalance][T1002][YB42AC4D0007-00063C4964C9A63F-0-0] [lt=185699662][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT distinct unit_group_id, status from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1002) order by unit_group_id "}, aret=-4012, ret=-4012) [401]OB_MALLOC COST TOO MUCH TIME, cost_time=959993, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=60, WASH_BLOCK_END=959933], seq:[0, 1, 2] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=960223, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=64, WASH_BLOCK_END=960157, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1, 2, 3, 4] [2025-08-14 09:01:52.774860] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:01:53.261951] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [651]OB_MALLOC COST TOO MUCH TIME, cost_time=877044, tenant_id=500, label=CommonArray, ctx_id=0, prio=0, size=15872, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [832]OB_MALLOC COST TOO MUCH TIME, cost_time=964294, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=50, WASH_BLOCK_END=964241, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1, 2, 3, 4] [857]OB_MALLOC COST TOO MUCH TIME, cost_time=702481, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:01:52.754958] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:53.716715] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=960302][errcode=0] epoll handler too much time :961762 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:52.790211] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [409][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=23] [sql nio session](*s={this:0x7f98f7334030, session_id:3221557375, trace_id:YB42AC4D0007-00063C4962CA3040-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:262}, err:0, last_decode_time:1755161121788142, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:2389, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:01:53.720577] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [409][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=928958] [sql nio session](*s={this:0x7f98f7333030, session_id:3221589989, trace_id:YB42AC4D0007-00063C4962CA3068-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:273}, err:0, last_decode_time:1755161766744050, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1903, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:01:53.720588] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [409][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] [sql nio session](*s={this:0x7f98f7339030, session_id:3221590023, trace_id:YB42AC4D0007-00063C4962CA3069-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:254}, err:0, last_decode_time:1755161784744265, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1683, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:01:52.755098] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:53.720881] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=964318] temporary file module use no blocks [2025-08-14 09:01:53.720889] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:53.737373] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:53.743246] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:53.743432] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:52.782691] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=16] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:53.743527] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=959383] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623014222315, flush_io_finished_ret_:0, flush_io_finished_round_:26294, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:53.743701] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=363999654, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:53.743722] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:53.743839] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [950][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F26-0-0] [lt=2461][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1002, svr="172.77.0.7:2882", version={val:1755161758221798000, v:0}, valid_part_count=0, total_part_count=2, generate_timestamp=1755162113040093) [2025-08-14 09:01:53.743896] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [950][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F26-0-0] [lt=55][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1002, req={req_server:"172.77.0.7:2882", version:{val:1755161758221798000, v:0}, valid_part_count:0, total_part_count:2, generate_timestamp:1755162113040093}, twrs={inited:true, tenant_id:1002, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162112940047}, server_version_for_stat:{version:{val:18446744073709551615, v:3}, total_part_count:1, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162113743680}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [567]OB_MALLOC COST TOO MUCH TIME, cost_time=302036, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:01:53.745161] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=566002331, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:53.745193] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:53.745693] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:53.745796] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=566002331, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:53.745814] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:53.733087] INFO [PALF] check_parent_health (log_config_mgr.cpp:2875) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=22] re_register_parent reason(palf_id=1, self="172.77.0.7:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1755162099867155, last_first_register_time_us=1755158728414267, register_time_us=1755162099867155, curr_time_us=1755162112755583) [2025-08-14 09:01:52.755199] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=20] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:53.746113] WDIAG ev_invoke_pending (ev.c:2136) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=989416][errcode=0] epoll handler too much time :990916 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:53.746142] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EE99-0-0] [lt=11] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-08-14 09:01:53.746208] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:53.746515] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:53.747389] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:53.744340] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] get wrs ts(ls_id={id:1001}, delta=369013023, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:53.718264] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:53.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.743986] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [316][pnio1][T1002][YB42AC4D0007-00063C49612B8F26-0-0] [lt=19998][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1002, delta=1990964, last_succ_cluster_heartbeat_tstamp_=1755162111753021) [2025-08-14 09:01:52.793382] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=20] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:53.747742] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=29429][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:53.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.874600] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:52.782870] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=30] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:53.747810] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=963481][errcode=0] epoll handler too much time :964947 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:53.747809] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=871887] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:01:53.747827] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:52.974707] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:52.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:52.881977] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:01:52.793, task_priority:1, with_handle_protected:1}) [659]OB_MALLOC COST TOO MUCH TIME, cost_time=970319, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=970311], seq:[0] [2025-08-14 09:01:53.747877] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=864589][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:52.881|207=865.90ms|210=0us|222=1us|total=865.91ms) [2025-08-14 09:01:53.747871] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=771993][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:52.974|207=773.16ms|210=0us|222=1us|total=773.17ms) [2025-08-14 09:01:53.747890] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task use too much time(end=1755162113747890, start=1755162112881974, delta=865916) [2025-08-14 09:01:53.747893] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4389] timer task use too much time(end=1755162113747893, start=1755162112974706, delta=773187) [2025-08-14 09:01:53.716701] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=454059][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:01:52.752945] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:8607757, max_lsn:1604090331, max_scn:{val:1755161756730847001, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:1556737504}, reuse_lsn:{lsn:1604090331}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:1562147291}}, last_submit_lsn:{lsn:1604090203}, last_submit_end_lsn:{lsn:1604090331}, last_submit_log_id:8607757, last_submit_log_pid:50, max_flushed_lsn:{lsn:1604090203}, max_flushed_end_lsn:{lsn:1604090331}, max_flushed_log_pid:50, committed_end_lsn:{lsn:1604090090}, last_slide_log_id:8607755, last_slide_scn:{val:1755161749741193000, v:0}, last_slide_lsn:{lsn:1604089968}, last_slide_end_lsn:{lsn:1604090090}, last_slide_log_pid:50, last_slide_log_accum_checksum:1446576759, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f990bfe4530}, [StateMgr]={this:0x7f990bfe89b0, self:"172.77.0.7:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"PENDING", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:51}, leader:"0.0.0.0:0", leader_epoch:-1, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:1604090331}, scan_disk_log_finished:true, last_check_start_id:8607756, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161901828039, palf_role_change_cb:0x7f990bfef470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:50, prev:{version:1, config:{config_version:{proposal_id:49, config_seq:45}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:50, config_seq:46}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_mode_pid:50}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_end_lsn:{lsn:1604088821}, prev_mode_pid:50}, reconfig_barrier:{prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_end_lsn:{lsn:1604088821}, prev_mode_pid:50}, persistent_config_version:{proposal_id:50, config_seq:46}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755161738854289, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755161738854289, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f990bfe5c70}, [ModeMgr]={palf_id:1, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:50, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:1543409664}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:51}, log_config_meta:{version:3, proposal_id:50, prev:{version:1, config:{config_version:{proposal_id:49, config_seq:45}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:50, config_seq:46}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_mode_pid:50}, log_snapshot_meta:{version:2, base_lsn:{lsn:1543409664}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:712704}, readable_log_tail:{lsn:712704}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:1610514432}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/meta", dir_fd:225, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66392064, block_header_serialize_buf_:0x7f990bfeabb8, flashback_version:0}, log_storage:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1755161901828095, this:0x7f990bfe90f0}, [Reconfirm]={palf_id:1, self:"172.77.0.7:2882", state:"RECONFIRM_MODE_META", new_proposal_id:51, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"172.77.0.7:2882", majority_max_accept_pid:50, majority_max_lsn:{lsn:1604090331}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:1755161901828373, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f990bfe8b30}) [2025-08-14 09:01:53.716664] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:946) [1003][BlackListServic][T1][YB42AC4D0007-00063C495F09BD68-0-0] [lt=18][errcode=-4012] execute failed(ret=-4012, tenant_id=1, executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, retry_cnt=0, local_sys_schema_version=1753073979411448, local_tenant_schema_version=1753073979411448) [2025-08-14 09:01:53.747964] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=31208][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:01:53.747977] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:01:53.747972] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=993509] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161974728707, last_flushed_end_lsn:{lsn:1604090331}}]) [2025-08-14 09:01:53.747995] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:01:53.747990] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16] current log_task status(palf_id=1, self="172.77.0.7:2882", log_id=8607756, log_task={header:{begin_lsn:{lsn:1604090090}, end_lsn:{lsn:1604090203}, log_id:8607756, min_scn:{val:1755161749741193001, v:0}, max_scn:{val:1755161749741193001, v:0}, data_len:57, proposal_id:50, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:1604090090}, data_checksum:2031283194, accum_checksum:759186326, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1755161750716165, freeze_ts:1755161750716173, submit_ts:1755161750716183, flushed_ts:1755161853765770, gen_to_freeze cost time:8, gen_to_submit cost time:18, submit_to_flush cost time:103049587}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161974728707, last_flushed_end_lsn:{lsn:1604090331}}]) [2025-08-14 09:01:53.748009] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:01:53.748025] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:01:53.748035] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] get_fetch_log_dst failed or invalid(ret=0, palf_id=1001, self="172.77.0.7:2882", fetch_log_dst="172.77.0.7:2882") [2025-08-14 09:01:53.748036] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:01:53.748048] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:01:53.748049] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:00:52.998, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:22.998], idx=771, next_allow_print_ts="09:01:22.998") [2025-08-14 09:01:53.748066] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=08:59:57.942, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:27.942], idx=868, next_allow_print_ts="09:00:27.942") [2025-08-14 09:01:53.720901] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:53.748202] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27259] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623014193244, flush_io_finished_ret_:0, flush_io_finished_round_:26258, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [383]OB_MALLOC COST TOO MUCH TIME, cost_time=966814, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=966802], seq:[0] [2025-08-14 09:01:53.748065] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1001, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:8571658, max_lsn:7323687369, max_scn:{val:1755161749741193002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:7278610047}, reuse_lsn:{lsn:7323686131}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:7281744329}}, last_submit_lsn:{lsn:7323686131}, last_submit_end_lsn:{lsn:7323687369}, last_submit_log_id:8571658, last_submit_log_pid:49, max_flushed_lsn:{lsn:7323686009}, max_flushed_end_lsn:{lsn:7323686131}, max_flushed_log_pid:49, committed_end_lsn:{lsn:7323686009}, last_slide_log_id:8571656, last_slide_scn:{val:1755161747742105001, v:0}, last_slide_lsn:{lsn:7323685887}, last_slide_end_lsn:{lsn:7323686009}, last_slide_log_pid:49, last_slide_log_accum_checksum:2505458373, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f990112a530}, [StateMgr]={this:0x7f990112e9b0, self:"172.77.0.7:2882", palf_id:1001, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"PENDING", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:49}, leader:"0.0.0.0:0", leader_epoch:-1, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:7323687369}, scan_disk_log_finished:true, last_check_start_id:8571657, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161767744317, palf_role_change_cb:0x7f9901135470, allow_vote:true}, [ConfigMgr]={palf_id:1001, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:49, prev:{version:1, config:{config_version:{proposal_id:48, config_seq:47}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:49, config_seq:48}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_mode_pid:49}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_end_lsn:{lsn:7323685240}, prev_mode_pid:49}, reconfig_barrier:{prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_end_lsn:{lsn:7323685240}, prev_mode_pid:49}, persistent_config_version:{proposal_id:49, config_seq:48}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755161739755312, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755161739755312, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f990112bc70}, [ModeMgr]={palf_id:1001, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, accepted_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1001, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1753069671378640001, v:0}, base_lsn_for_block_gc:{lsn:7247314944}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:49}, log_config_meta:{version:3, proposal_id:49, prev:{version:1, config:{config_version:{proposal_id:48, config_seq:47}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:49, config_seq:48}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_mode_pid:49}, log_snapshot_meta:{version:2, base_lsn:{lsn:7247314944}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1753069671378640000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}}, log_meta_storage:{log_tail:{lsn:765952}, readable_log_tail:{lsn:765952}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7381524480}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/meta", dir_fd:232, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66338816, block_header_serialize_buf_:0x7f9901130bb8, flashback_version:0}, log_storage:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, palf_epoch:2, last_purge_throttling_ts:1755161769777679, this:0x7f990112f0f0}, [Reconfirm]={palf_id:1001, self:"172.77.0.7:2882", state:"WAITING_LOG_FLUSHED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:1755161769777679, this:0x7f990112eb30}) [2025-08-14 09:01:53.748248] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=181] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [2025-08-14 09:01:53.748257] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [383]OB_MALLOC COST TOO MUCH TIME, cost_time=966835, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=966834], seq:[0, 1] [2025-08-14 09:01:53.748259] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11] current log_task status(palf_id=1001, self="172.77.0.7:2882", log_id=8571657, log_task={header:{begin_lsn:{lsn:7323686009}, end_lsn:{lsn:7323686131}, log_id:8571657, min_scn:{val:1755161749741193000, v:0}, max_scn:{val:1755161749741193000, v:0}, data_len:66, proposal_id:49, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:7323686009}, data_checksum:2222604892, accum_checksum:1531236791, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1755161749745909, freeze_ts:1755161749745914, submit_ts:1755161749745940, flushed_ts:1755161785758706, gen_to_freeze cost time:5, gen_to_submit cost time:31, submit_to_flush cost time:36012766}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [2025-08-14 09:01:53.748273] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:01:53.748286] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:01:52.774600] INFO [STORAGE.TRANS] refresh_elr_tenant_config_ (ob_tx_elr_util.cpp:42) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=9] refresh tenant config success(tenant_id=1001, *this={last_refresh_ts:1755162112769239, can_tenant_elr:true}) [2025-08-14 09:01:53.748330] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=972250] get_max_applied_scn(scn={val:1755161719775179002, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:45, palf_committed_end_lsn_:{lsn:8461098908}, palf_committed_end_scn_:{val:1755161719775179002, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161719775179002, v:0}}) [2025-08-14 09:01:53.722116] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:53.748382] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26225][errcode=0] epoll handler too much time :26275 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:53.748349] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=15] get_min_unreplayed_log_info(lsn={lsn:8461098908}, scn={val:1755100947600130001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161920778323, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:23084275, ret_code_:0}}, next_to_submit_lsn_:{lsn:8461098908}, next_to_submit_scn_:{val:1755100947600130001, v:0}, base_lsn_:{lsn:7515734016}, base_scn_:{val:1753840867171901045, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6e05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:8461098908}, end_lsn:{lsn:8461098908}, read_buf:{buf_len_:3747840, buf_:0x7f98f6e05000}, block_size:67104768, log_storage_:0x7f99297a3bf0, log_storage_:{log_tail:{lsn:8461098908}, readable_log_tail:{lsn:8461098908}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8455200768}, min_scn:{val:1755160384851871002, v:0}, curr_block_id:126, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:215, min_block_id:93, max_block_id:127, curr_writable_block_id:126}, logical_block_size_:67104768, curr_block_writable_size_:61206628, block_header_serialize_buf_:0x7f99297a48b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:198, read_io_size:699822080, read_disk_cost_ts:4178035, start_lsn:{lsn:7515734016}}}}}}) [2025-08-14 09:01:53.748408] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=57] finish update min start scn(max_decided_scn={val:1755161719775179002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1755161287778691002, v:0}, keep_alive_scn_:{val:1755161287778691002, v:0}, update_ts_:1755162113747618}) [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=969979, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=969977], seq:[0] [2025-08-14 09:01:53.749158] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=24] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:01:53.752495] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:53.752551] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [574]OB_MALLOC COST TOO MUCH TIME, cost_time=967076, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=967070], seq:[0] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=967085, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=967084], seq:[0, 1] [2025-08-14 09:01:53.753349] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=13] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:53.753370] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=20] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3802) [2025-08-14 09:01:53.762284] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=5155] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:01:53.762417] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:53.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.770647] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:53.772163] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:53.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.782016] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=24] gc stale ls task succ [2025-08-14 09:01:53.802370] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:53.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.847868] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=8] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:01:53.847890] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:53.847967] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:01:53.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.946386] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:53.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.952480] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:53.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.891999] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [402][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] EASY SLOW: thread: 7f9953cfb300, lastrun: 1755162113.755434 cost: 0.136564 loop:26440, slowcnt: 1 [2025-08-14 09:01:53.782325] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=15] gc stale ls task succ [2025-08-14 09:01:53.783661] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:54.037392] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=253347][errcode=0] epoll handler too much time :253739 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:54.685492] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:01:54.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.749797] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:53.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.847993] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:53.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.753378] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:01:53.947926] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:54.720472] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=771374][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:54.720450] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=871134][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:53.847|207=872.46ms|210=0us|222=2us|total=872.50ms) [2025-08-14 09:01:54.720491] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4389] timer task use too much time(end=1755162114720490, start=1755162113847991, delta=872499) [2025-08-14 09:01:54.023659] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:54.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:54.720529] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=695806][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:54.236|207=696.86ms|210=1us|222=1us|total=696.88ms) [2025-08-14 09:01:54.720541] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task use too much time(end=1755162114720540, start=1755162114023657, delta=696883) [393]OB_MALLOC COST TOO MUCH TIME, cost_time=966793, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=966776, ALLOC_CHUNK_END=5], seq:[0, 1] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=966835, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=966810, WASH_BLOCK_END=25], seq:[0, 1, 2] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=964827, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=964815, ALLOC_CHUNK_END=6], seq:[0, 1] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=948060, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=948051, ALLOC_CHUNK_END=4], seq:[0, 1] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=964876, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=964841, WASH_BLOCK_END=35], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=948099, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=948075, WASH_BLOCK_END=24], seq:[0, 1, 2] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=970924, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=970922], seq:[0] [717]OB_MALLOC COST TOO MUCH TIME, cost_time=938155, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=938144, ALLOC_CHUNK_END=6], seq:[0, 1] [717]OB_MALLOC COST TOO MUCH TIME, cost_time=938172, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=938163, WASH_BLOCK_END=9], seq:[0, 1, 2] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=937674, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=937665, ALLOC_CHUNK_END=4], seq:[0, 1] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=937692, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=937682, WASH_BLOCK_END=10], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=753664, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=753654, ALLOC_CHUNK_END=5], seq:[0, 1] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=753681, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=753672, WASH_BLOCK_END=9], seq:[0, 1, 2] [513]OB_MALLOC COST TOO MUCH TIME, cost_time=688921, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=688912, ALLOC_CHUNK_END=4], seq:[0, 1] [513]OB_MALLOC COST TOO MUCH TIME, cost_time=688935, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=688927, WASH_BLOCK_END=8], seq:[0, 1, 2] [797]OB_MALLOC COST TOO MUCH TIME, cost_time=149388, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=149380, ALLOC_CHUNK_END=4], seq:[0, 1] [797]OB_MALLOC COST TOO MUCH TIME, cost_time=149406, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=149396, WASH_BLOCK_END=9], seq:[0, 1, 2] [659]OB_MALLOC COST TOO MUCH TIME, cost_time=989341, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=11155, WASH_BLOCK_END=5156, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=970344], seq:[0, 1, 2, 3, 4] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=876793, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=980130, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=980121], seq:[0] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=1006919, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=21566, WASH_BLOCK_END=5196, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=980150], seq:[0, 1, 2, 3, 4] [2025-08-14 09:01:53.721960] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1006956, owner: ObLog, click_count: 5, time dist:[FORMAT_END=24, ObMalloc#start=0, ObMalloc#end=1006928, ALLOC_END=3, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [385]OB_MALLOC COST TOO MUCH TIME, cost_time=983202, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=970933], seq:[0, 1] [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=987094, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=969990], seq:[0, 1] [2025-08-14 09:01:54.720561] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=428210, expected_run_ts_us_=1755162114292349, ObClockGenerator::getRealClock()=1755162114720560, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:54.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:54.720576] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:54.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:54.046448] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:53.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:54.720610] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=673140][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:54.464|207=674.16ms|210=0us|222=2us|total=674.17ms) [2025-08-14 09:01:54.052543] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:53.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:54.720635] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=667080][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:54.525|207=668.09ms|210=0us|222=1us|total=668.10ms) [2025-08-14 09:01:54.720646] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task use too much time(end=1755162114720646, start=1755162114052541, delta=668105) [2025-08-14 09:01:54.248459] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:01:54.720671] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=471492][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:01:54.720689] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:01:54.720705] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:01:54.720719] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:01:53.752345] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:54.720906] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=967096][errcode=0] epoll handler too much time :968570 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:53.752387] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:54.720620] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task use too much time(end=1755162114720620, start=1755162114046446, delta=674174) [2025-08-14 09:01:54.721682] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:53.720386] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:54.721792] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=999892][errcode=0] epoll handler too much time :1001416 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:54.721814] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:53.752478] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:54.721990] WDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=968046][errcode=0] epoll handler too much time :969518 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:54.720727] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:01:54.722021] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=1279][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:01:53.782894] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:54.722036] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:01:54.722043] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=937730][errcode=0] epoll handler too much time :939159 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:54.722049] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:01:54.722059] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:01:54.722068] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:00:52.998, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:22.998], idx=771, next_allow_print_ts="09:01:22.998") [2025-08-14 09:01:54.722083] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=08:59:57.942, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:27.942], idx=868, next_allow_print_ts="09:00:27.942") [2025-08-14 09:01:53.967259] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=31] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:01:54.572596] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fe410, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe6f0, time_wheel:0x7f9922df9d40, thread_pool:0x7f9922df9520, is_running:1, time_interval:2.00s, expected_run_time:09:01:54.432, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:54.722305] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=149480][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:54.572|207=149.71ms|210=0us|222=2us|total=149.72ms) [2025-08-14 09:01:54.722320] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] timer task use too much time(end=1755162114722319, start=1755162114572593, delta=149726) [2025-08-14 09:01:54.720480] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=965624] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:54.722373] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=1888] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3803) [2025-08-14 09:01:54.722382] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:01:54.722554] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=6] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:54.722566] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=11] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3804) [2025-08-14 09:01:54.722571] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:01:54.720520] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=1004453] finish to resize log external storage handler(current_ts=1755162113733429, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-08-14 09:01:54.723224] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=2697] snapshot_traffic_info once [2025-08-14 09:01:53.747599] INFO [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:171) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=940381] handle gts request statistics(total_rt=941844, total_cnt=1, avg_rt=4.709220000000000000e+05, avg_cnt=1.000000000000000056e-01) [2025-08-14 09:01:54.723286] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=974209][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162112774818], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:54.723302] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162112774818], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:54.723316] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162112774818]}) [2025-08-14 09:01:54.723327] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts statistics(tenant_id=1002, gts_rpc_cnt=0, get_gts_cache_cnt=169, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:01:54.723362] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=390] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:54.723378] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:54.723400] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=21] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623015221999, flush_io_finished_ret_:0, flush_io_finished_round_:26286, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:54.722333] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:01:54.725087] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=2741][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:01:54.725105] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:01:53.721960] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728bb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.718, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730470] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1006956][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:53.721|207=1.01s|210=1us|222=2us|total=1.01s) [2025-08-14 09:01:54.730499] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4389] timer task use too much time(end=1755162114730499, start=1755162113721958, delta=1008541) [2025-08-14 09:01:54.730519] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] task scheduled too delay(scheduled_delay=1010711, expected_run_ts_us_=1755162113719806, ObClockGenerator::getRealClock()=1755162114730518, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180fbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739fe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.719, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730534] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180fbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739fe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.719, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730553] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] task scheduled too delay(scheduled_delay=982810, expected_run_ts_us_=1755162113747742, ObClockGenerator::getRealClock()=1755162114730553, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b9b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730562] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b9b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730572] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=982137, expected_run_ts_us_=1755162113748434, ObClockGenerator::getRealClock()=1755162114730571, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891973990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730592] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891973990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730601] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=981525, expected_run_ts_us_=1755162113749075, ObClockGenerator::getRealClock()=1755162114730601, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728b090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730610] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728b090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730619] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=974892, expected_run_ts_us_=1755162113755726, ObClockGenerator::getRealClock()=1755162114730619, ret=0, ret="OB_SUCCESS", *this={this:0x7f98918105b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a0490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730628] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918105b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a0490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730639] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=965244, expected_run_ts_us_=1755162113765395, ObClockGenerator::getRealClock()=1755162114730639, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891972310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730648] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891972310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730657] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=967024, expected_run_ts_us_=1755162113763633, ObClockGenerator::getRealClock()=1755162114730657, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ed990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730666] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ed990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730675] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=962401, expected_run_ts_us_=1755162113768273, ObClockGenerator::getRealClock()=1755162114730675, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891815730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7292a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730685] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891815730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7292a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730701] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] task scheduled too delay(scheduled_delay=960305, expected_run_ts_us_=1755162113770395, ObClockGenerator::getRealClock()=1755162114730700, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72eca90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730716] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72eca90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730725] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=960107, expected_run_ts_us_=1755162113770617, ObClockGenerator::getRealClock()=1755162114730725, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e94b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739e810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730734] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e94b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739e810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730744] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=959958, expected_run_ts_us_=1755162113770786, ObClockGenerator::getRealClock()=1755162114730744, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739e690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730752] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739e690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730762] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=955640, expected_run_ts_us_=1755162113775121, ObClockGenerator::getRealClock()=1755162114730762, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180ea30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739f410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730771] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180ea30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739f410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730781] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=947911, expected_run_ts_us_=1755162113782869, ObClockGenerator::getRealClock()=1755162114730781, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ea510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730791] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ea510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730801] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=946695, expected_run_ts_us_=1755162113784105, ObClockGenerator::getRealClock()=1755162114730800, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891825990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730816] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891825990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730825] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=941737, expected_run_ts_us_=1755162113789087, ObClockGenerator::getRealClock()=1755162114730825, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196be10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730834] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196be10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730846] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=945508, expected_run_ts_us_=1755162113785337, ObClockGenerator::getRealClock()=1755162114730846, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728d430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7290f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730854] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728d430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7290f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730863] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=935098, expected_run_ts_us_=1755162113795764, ObClockGenerator::getRealClock()=1755162114730863, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a1690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730871] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a1690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730888] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] task scheduled too delay(scheduled_delay=934981, expected_run_ts_us_=1755162113795906, ObClockGenerator::getRealClock()=1755162114730888, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73922b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a1510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730898] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73922b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a1510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730907] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=934871, expected_run_ts_us_=1755162113796036, ObClockGenerator::getRealClock()=1755162114730907, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a1390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730915] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a1390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730931] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] task scheduled too delay(scheduled_delay=931794, expected_run_ts_us_=1755162113799136, ObClockGenerator::getRealClock()=1755162114730930, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a1210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.799, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730939] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a1210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.799, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730949] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=938015, expected_run_ts_us_=1755162113792934, ObClockGenerator::getRealClock()=1755162114730949, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7289890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730958] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7289890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730967] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=921756, expected_run_ts_us_=1755162113809210, ObClockGenerator::getRealClock()=1755162114730967, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891973e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.809, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730975] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891973e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.809, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730983] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=926123, expected_run_ts_us_=1755162113804860, ObClockGenerator::getRealClock()=1755162114730983, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eaa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7ef410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.730993] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eaa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7ef410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.731009] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] task scheduled too delay(scheduled_delay=893469, expected_run_ts_us_=1755162113837539, ObClockGenerator::getRealClock()=1755162114731009, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73945b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f1c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.837, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.731018] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73945b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f1c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.837, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.731027] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=819259, expected_run_ts_us_=1755162113911768, ObClockGenerator::getRealClock()=1755162114731027, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f1390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.911, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.731038] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f1390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:53.911, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.731063] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f1e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:54.718, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.731072] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739fd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:54.726, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.731080] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a67b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7397290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:54.728, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:53.747459] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=3110] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:53.843463] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a4d58, timer:0x7f994b1dfb70, task:0x7f991f004030, task_type:"N9oceanbase8observer20ObTableQueryASyncMgrE", scheduled_time:1755161750817104, pushed_time:0, delay:0}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=363026358, threashold=1000000, tenant_id=1002) [2025-08-14 09:01:54.732371] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=887560][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6178, timer:0x7f99200ff180, task:0x7f99200ff220, task_type:"N9oceanbase8observer18ObTableLoadService21ObClientTaskPurgeTaskE", scheduled_time:1755162052780445, pushed_time:0, delay:1000000}, running_task_token={this:0x7f991b7a5fb8, timer:0x7f99200ff180, task:0x7f99200ff1f0, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162050755138, pushed_time:1755162084760013, delay:1000000}, total_delay=61951926, threashold=1000000, tenant_id=1002) [2025-08-14 09:01:54.473407] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fa190, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa470, time_wheel:0x7f99341cdbf0, thread_pool:0x7f99341cd3d0, is_running:1, time_interval:2.00s, expected_run_time:09:01:54.334, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:54.732516] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=258714][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:54.473|207=259.11ms|210=0us|222=1us|total=259.12ms) [2025-08-14 09:01:54.732527] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task use too much time(end=1755162114732527, start=1755162114473404, delta=259123) [2025-08-14 09:01:54.732620] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1808) [784][T1002_TBalance][T1002][YB42AC4D0007-00063C4964C9A63F-0-0] [lt=23][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT distinct unit_group_id, status from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1002) order by unit_group_id ) [2025-08-14 09:01:54.733834] INFO [PALF] register_parent_ (log_config_mgr.cpp:2707) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=12856] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.77.0.7:2882", reason=1, leader="172.77.0.7:2882", register_time_us=1755162113747625, last_submit_register_req_time_us=1755162113747625, register_parent_reason=1) [2025-08-14 09:01:54.733857] INFO [PALF] check_parent_health (log_config_mgr.cpp:2884) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=20] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.77.0.7:2882") [2025-08-14 09:01:54.733870] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=8] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:12632581, max_lsn:8461098908, max_scn:{val:1755161719775179002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:8214679957}, reuse_lsn:{lsn:8461098908}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:8419155868}}, last_submit_lsn:{lsn:8461098786}, last_submit_end_lsn:{lsn:8461098908}, last_submit_log_id:12632581, last_submit_log_pid:45, max_flushed_lsn:{lsn:8461098786}, max_flushed_end_lsn:{lsn:8461098908}, max_flushed_log_pid:45, committed_end_lsn:{lsn:8461098908}, last_slide_log_id:12632581, last_slide_scn:{val:1755161719775179002, v:0}, last_slide_lsn:{lsn:8461098786}, last_slide_end_lsn:{lsn:8461098908}, last_slide_log_pid:45, last_slide_log_accum_checksum:1091335677, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f992979c530}, [StateMgr]={this:0x7f99297a09b0, self:"172.77.0.7:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:46}, leader:"0.0.0.0:0", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:8461098908}, scan_disk_log_finished:true, last_check_start_id:12632582, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161899748977, palf_role_change_cb:0x7f99297a7470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:45, prev:{version:1, config:{config_version:{proposal_id:43, config_seq:41}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:45, config_seq:42}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:41, prev_lsn:{lsn:8461098218}, prev_mode_pid:45}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:41, prev_lsn:{lsn:8461098218}, prev_end_lsn:{lsn:8461098377}, prev_mode_pid:45}, reconfig_barrier:{prev_log_proposal_id:41, prev_lsn:{lsn:8461098218}, prev_end_lsn:{lsn:8461098377}, prev_mode_pid:45}, persistent_config_version:{proposal_id:45, config_seq:42}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755162113747625, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755162113747625, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f992979dc70}, [ModeMgr]={palf_id:1, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:45, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:46, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:46, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:1753719139394335001, v:0}, min_block_id:93, min_block_min_scn:{val:1753712284262753001, v:0}, base_lsn_for_block_gc:{lsn:8186781696}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:46}, log_config_meta:{version:3, proposal_id:45, prev:{version:1, config:{config_version:{proposal_id:43, config_seq:41}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:45, config_seq:42}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:41, prev_lsn:{lsn:8461098218}, prev_mode_pid:45}, log_snapshot_meta:{version:2, base_lsn:{lsn:8186781696}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:46, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:655360}, readable_log_tail:{lsn:655360}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8522305536}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/meta", dir_fd:214, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66449408, block_header_serialize_buf_:0x7f99297a2bb8, flashback_version:0}, log_storage:{log_tail:{lsn:8461098908}, readable_log_tail:{lsn:8461098908}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8455200768}, min_scn:{val:1755160384851871002, v:0}, curr_block_id:126, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:215, min_block_id:93, max_block_id:127, curr_writable_block_id:126}, logical_block_size_:67104768, curr_block_writable_size_:61206628, block_header_serialize_buf_:0x7f99297a48b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1755161900739661, this:0x7f99297a10f0}, [Reconfirm]={palf_id:1, self:"172.77.0.7:2882", state:"FETCH_MAX_LOG_LSN", new_proposal_id:46, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"172.77.0.7:2882", majority_max_accept_pid:45, majority_max_lsn:{lsn:8461098908}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:1755161954782466, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f99297a0b30}) [2025-08-14 09:01:54.734067] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=191] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161956777957, last_flushed_end_lsn:{lsn:8461098908}}]) [2025-08-14 09:01:54.734080] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=10] current log_task status(palf_id=1, self="172.77.0.7:2882", log_id=12632582, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161956777957, last_flushed_end_lsn:{lsn:8461098908}}]) [2025-08-14 09:01:53.749438] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:54.734160] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=983233][errcode=0] epoll handler too much time :984730 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:54.723337] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162112774818]}) [2025-08-14 09:01:54.734436] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11079][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1002}) [2025-08-14 09:01:54.734445] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1002}) [2025-08-14 09:01:54.724045] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=566980888, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:54.736308] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12231] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:53.762360] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:54.736348] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=972515][errcode=0] epoll handler too much time :974008 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:53.993766] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:53.989, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:54.736383] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=741492][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:53.993|207=742.62ms|210=0us|222=1us|total=742.63ms) [2025-08-14 09:01:54.736404] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4389] timer task use too much time(end=1755162114736404, start=1755162113993764, delta=742640) [2025-08-14 09:01:54.736412] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=0] task scheduled too delay(scheduled_delay=730924, expected_run_ts_us_=1755162114005487, ObClockGenerator::getRealClock()=1755162114736412, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:54.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:54.736423] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:54.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:54.736433] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=647116, expected_run_ts_us_=1755162114089316, ObClockGenerator::getRealClock()=1755162114736433, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:54.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:54.736441] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:54.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:54.736452] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=532155, expected_run_ts_us_=1755162114204297, ObClockGenerator::getRealClock()=1755162114736452, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:54.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:54.736463] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:54.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:54.736480] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd110, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd3f0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:10.00s, expected_run_time:09:01:54.726, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.742906] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=15] current monitor number(seq_=-1) [2025-08-14 09:01:54.739920] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:54.740601] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161101770923, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755161572801635) [2025-08-14 09:01:54.741166] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] get wrs ts(ls_id={id:1}, delta=566998219, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:54.741181] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:54.741227] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919d9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b4610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:54.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.741260] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7277c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72eac90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:54.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.720217] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=968950][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:53.749|207=970.42ms|210=1us|222=1us|total=992.55ms) [2025-08-14 09:01:54.742358] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=22108][errcode=-4389] timer task use too much time(end=1755162114742358, start=1755162113749795, delta=992563) [2025-08-14 09:01:54.742374] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=997739, expected_run_ts_us_=1755162113744633, ObClockGenerator::getRealClock()=1755162114742373, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:53.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:54.742394] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:53.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:54.742427] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:54.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:54.721863] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=967598][errcode=0] epoll handler too much time :969069 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:54.742677] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:54.743676] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:54.743704] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=22] temporary file module use no blocks [2025-08-14 09:01:54.743713] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:54.743726] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:54.743741] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623016183322, flush_io_finished_ret_:0, flush_io_finished_round_:26295, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:54.745230] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:53.747962] EDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1076) [1003][BlackListServic][T1][YB42AC4D0007-00063C495F09BD68-0-0] [lt=31246][errcode=-4012] LOGGER COST TOO MUCH TIME, cost: 1001161, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1001149, ObMalloc#start=2, ObMalloc#end=5, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x12ead58e 0x981b5d1 0x124ddd52 0x96a5a41 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x2615d428 0x1c013aca 0x1c01349f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:00:14.721899] EDIAG [STORAGE] add_table (ob_tablet_table_store_iterator.cpp:242) [14945][T1_TimerWK8_MergeLoop][T1][YB42AC4D0007-00063C4978199C2C-0-0] [lt=1997896][errcode=-4012] LOGGER COST TOO MUCH TIME, cost: 6036860, owner: ObLog, click_count: 5, time dist:[FORMAT_END=6036852, ObMalloc#start=1, ObMalloc#end=6, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1bd2d8a6 0x972ecc3 0x972e828 0x972ee6c 0x9519a4d 0x9361c6d 0x94fbe0f 0x94b4970 0x9766028 0x9765ae5 0x976579e 0x9764de3 0x975ab8f 0x97f7c68 0x97f78b3 0x97f7636 0x97f5260 0x97f4a4a 0x97f4859 0x97f4305 0x1aee6975 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:54.745760] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:54.746284] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:54.746476] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:54.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:54.748089] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=952934] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:54.748573] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=24] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:54.748595] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=19] temporary file module use no blocks [2025-08-14 09:01:54.748605] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:54.748619] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:54.748634] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623016187996, flush_io_finished_ret_:0, flush_io_finished_round_:26259, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:54.749225] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:54.749699] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:01:54.749724] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:01:54.750282] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:54.750354] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:53.747962] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1076) [1003][BlackListServic][T1][YB42AC4D0007-00063C495F09BD68-0-0] [lt=31246][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4012, err_:"OB_TIMEOUT", retry_type:0, client_ret:-4012}, need_retry=false) [2025-08-14 09:01:54.750805] INFO [STORAGE] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:123) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=982198] wead read timestamp is too old(timestamp={val:1755161744727824001, v:0}, gts_scn={val:1755161762721798002, v:0}, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1755161744727824001, v:0}}) [2025-08-14 09:01:54.750838] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:53.748808] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1076) [1029][ObStorageHADiag][T1][YB42AC4D0007-00063C4966799D69-0-0] [lt=29][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2025-08-14 09:01:54.751069] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:54.751257] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ad990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:54.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.751282] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ad810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:54.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.752049] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=31] get_min_unreplayed_log_info(lsn={lsn:1604088821}, scn={val:1755100948296325001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873479, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:12492179, ret_code_:0}}, next_to_submit_lsn_:{lsn:1604088821}, next_to_submit_scn_:{val:1755100948296325001, v:0}, base_lsn_:{lsn:1409200128}, base_scn_:{val:1753840802540914000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:1604088821}, end_lsn:{lsn:1604088821}, read_buf:{buf_len_:3747840, buf_:0x7f98f6a05000}, block_size:67104768, log_storage_:0x7f990bfebbf0, log_storage_:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:42, read_io_size:147709952, read_disk_cost_ts:1061450, start_lsn:{lsn:1409200128}}}}}}) [2025-08-14 09:01:54.752133] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=81] get wrs ts(ls_id={id:1}, delta=365010382, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:54.752149] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:54.752140] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [950][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F27-0-0] [lt=31][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1002, svr="172.77.0.7:2882", version={val:1755161758221798000, v:0}, valid_part_count=0, total_part_count=2, generate_timestamp=1755162114751984) [2025-08-14 09:01:54.752170] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [950][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F27-0-0] [lt=30][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1002, req={req_server:"172.77.0.7:2882", version:{val:1755161758221798000, v:0}, valid_part_count:0, total_part_count:2, generate_timestamp:1755162114751984}, twrs={inited:true, tenant_id:1002, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162113743680}, server_version_for_stat:{version:{val:18446744073709551615, v:3}, total_part_count:1, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162114752018}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:01:54.752280] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [316][pnio1][T1002][YB42AC4D0007-00063C49612B8F27-0-0] [lt=3749][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1002, delta=1008295, last_succ_cluster_heartbeat_tstamp_=1755162113743985) [2025-08-14 09:01:54.752476] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:54.752654] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=7] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:01:54.752773] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1001}, delta=370024866, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:54.752787] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:54.752806] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [WRS] update tenant weak read server version(tenant_id=1002, server_version={version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162114752018}, version_delta=356531008) [2025-08-14 09:01:54.752830] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=365011497, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:54.752847] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:54.754437] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20601] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:54.754471] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:54.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:54.754521] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=737] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:54.753450] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=370024866, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:54.760146] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6683] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:54.761313] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7277230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ea690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:54.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.761345] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7272330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919a5090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:54.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.761360] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919822b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891970390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:54.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.761384] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891969a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:54.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.761398] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d71b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891969890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:54.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.761410] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196b390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:54.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.761423] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a27b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ed810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:54.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.761425] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:54.761436] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a40b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7283690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:54.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.761449] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7faf30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7281590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:54.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.762444] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=27] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:01:54.763854] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1079) [784][T1002_TBalance][T1002][YB42AC4D0007-00063C4964C9A63F-0-0] [lt=15][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2025-08-14 09:01:54.763955] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1748) [784][T1002_TBalance][T1002][YB42AC4D0007-00063C4964C9A63F-0-0] [lt=93][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1753069619, tenant_id=1) [2025-08-14 09:01:54.766575] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:54.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:54.770788] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:54.771627] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7352330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196fc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:54.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.760378] WDIAG [SQL] do_close_plan (ob_result_set.cpp:844) [1003][BlackListServic][T1][YB42AC4D0007-00063C495F09BD68-0-0] [lt=1001161][errcode=-4006] exec result is null(ret=-4006) [2025-08-14 09:01:54.773374] WDIAG [SQL] do_close (ob_result_set.cpp:935) [1003][BlackListServic][T1][YB42AC4D0007-00063C495F09BD68-0-0] [lt=12972][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-08-14 09:01:54.773397] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [1003][BlackListServic][T1][YB42AC4D0007-00063C495F09BD68-0-0] [lt=17][errcode=-4012] result set close failed(ret=-4012) [2025-08-14 09:01:54.773408] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:199) [1003][BlackListServic][T1][YB42AC4D0007-00063C495F09BD68-0-0] [lt=9][errcode=-4012] result set close failed(ret=-4012) [2025-08-14 09:01:54.773416] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:951) [1003][BlackListServic][T1][YB42AC4D0007-00063C495F09BD68-0-0] [lt=8][errcode=-4012] failed to close result(close_ret=-4012, ret=-4012) [2025-08-14 09:01:54.769713] WDIAG [STORAGE] add_tables (ob_tablet_table_store_iterator.cpp:301) [14945][T1_TimerWK8_MergeLoop][T1][YB42AC4D0007-00063C4978199C2C-0-0] [lt=6036860][errcode=-4012] fail to add sstable to iterator(ret=-4012, i=0) [2025-08-14 09:01:54.776031] WDIAG [STORAGE] get_mds_sstables (ob_tablet_table_store.cpp:1280) [14945][T1_TimerWK8_MergeLoop][T1][YB42AC4D0007-00063C4978199C2C-0-0] [lt=6305][errcode=-4012] failed to add ddl sstables to iterator(ret=-4012) [2025-08-14 09:01:54.776050] WDIAG [STORAGE] inner_get_mds_sstables (ob_tablet.cpp:6949) [14945][T1_TimerWK8_MergeLoop][T1][YB42AC4D0007-00063C4978199C2C-0-0] [lt=16][errcode=-4012] fail to get read tables(ret=-4012) [2025-08-14 09:01:54.776065] WDIAG [STORAGE] get_mds_sstables_from_tablet (ob_tablet_handle.cpp:335) [14945][T1_TimerWK8_MergeLoop][T1][YB42AC4D0007-00063C4978199C2C-0-0] [lt=12][errcode=-4012] fail to get mds sstables(ret=-4012, snapshot_version=9223372036854775807) [2025-08-14 09:01:54.776311] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:54.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:54.779870] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=948893] PNIO [ratelimit] time: 1755162114779869, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007614, add_bytes: 0 [2025-08-14 09:01:54.781689] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919aca90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:54.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.781727] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f734fc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891918610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:54.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.782082] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=12] gc stale ls task succ [2025-08-14 09:01:54.782110] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:54.782376] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=253254] gc stale ls task succ [2025-08-14 09:01:54.783093] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:54.791755] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891969110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:54.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.791793] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196c510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:54.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.801814] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72eb110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:54.794, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.806513] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:54.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:54.811884] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196b690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:54.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.811920] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7282d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:54.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.811931] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7289710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:54.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.811959] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7397b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:54.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.811975] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918299b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891916210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:54.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.820558] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:54.820569] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:54.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:54.832009] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ec910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:54.825, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.832045] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ec790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:54.826, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.832056] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7282f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:54.820, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.832074] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72879b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728a010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:54.828, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.840786] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B922F-0-0] [lt=29][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162114840659) [2025-08-14 09:01:54.840819] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B922F-0-0] [lt=31][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162114840659}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162114740624}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162114740624}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:01:54.840909] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [317][pnio1][T1][YB42AC4D0007-00063C49607B922F-0-0] [lt=24][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1, delta=1799432, last_succ_cluster_heartbeat_tstamp_=1755162113041476) [2025-08-14 09:01:54.841313] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] get wrs ts(ls_id={id:1}, delta=567102729, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:54.841346] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:54.841985] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=567102729, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:54.842009] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:54.848503] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd070, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fd350, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:5.00s, expected_run_time:09:01:54.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:54.848545] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:01:54.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:54.852112] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7289590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:54.846, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:54.852920] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=365111718, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:54.852944] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:54.853547] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=370125087, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:54.853565] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:54.920528] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:54.920630] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:54.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:54.942032] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=567203110, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:54.942067] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:54.953024] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] get wrs ts(ls_id={id:1}, delta=365210975, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:54.953045] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:54.953645] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1001}, delta=370225404, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:54.953663] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:54.997153] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:54.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:54.997782] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:54.989, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.017917] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:55.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.020574] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=63][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:55.020690] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:54.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.022333] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=753717] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:01:55.027691] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:55.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.042114] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=567302508, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:55.042139] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:55.053084] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=365311589, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:55.053107] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:55.053742] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] get wrs ts(ls_id={id:1001}, delta=370324958, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:55.053762] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:55.098438] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:55.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.120634] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:55.120751] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:55.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.142184] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=567402586, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:55.142217] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:55.153154] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] get wrs ts(ls_id={id:1}, delta=365411624, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:55.153174] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:55.153251] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [950][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F2B-0-0] [lt=34][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1002, svr="172.77.0.7:2882", version={val:1755161758221798000, v:0}, valid_part_count=0, total_part_count=2, generate_timestamp=1755162115153115) [2025-08-14 09:01:55.153277] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [950][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F2B-0-0] [lt=25][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1002, req={req_server:"172.77.0.7:2882", version:{val:1755161758221798000, v:0}, valid_part_count:0, total_part_count:2, generate_timestamp:1755162115153115}, twrs={inited:true, tenant_id:1002, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162115053072}, server_version_for_stat:{version:{val:18446744073709551615, v:3}, total_part_count:1, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162115153142}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:01:55.153779] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] get wrs ts(ls_id={id:1001}, delta=370424993, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:55.153792] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:55.153817] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=365411624, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:55.153828] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:55.154398] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1001}, delta=370426051, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:55.154410] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:55.219183] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:55.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.220666] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:55.220815] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:55.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.222627] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:55.222664] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=36] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3805) [2025-08-14 09:01:55.222672] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:01:55.222682] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=6] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:55.222694] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=12] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3806) [2025-08-14 09:01:55.222704] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:01:55.222914] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=13] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:55.222931] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=16] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3807) [2025-08-14 09:01:55.222936] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:01:55.225359] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:01:55.225397] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:01:55.225417] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:01:55.234692] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=5] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:55.234719] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162115234687], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:55.234754] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162115234687], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:55.234773] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162115234687]}) [2025-08-14 09:01:55.234789] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=77, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:01:55.234814] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162115234687]}) [2025-08-14 09:01:55.234834] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=16] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:55.234843] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162115234832], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:55.248609] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=55][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:55.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.249378] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=70][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:55.239, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.253909] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=365511737, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:55.253934] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:55.254529] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=370526152, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:55.254546] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:55.258669] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=53][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:55.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.269430] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:55.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.279068] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:55.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.309210] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:55.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.421032] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=5][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:55.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.422319] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1042][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:55.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.490072] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:55.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.490567] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:55.489, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.510645] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:55.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.521091] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:55.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.522382] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:55.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.591071] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:55.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.621155] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:55.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.622418] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:55.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.686051] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:01:55.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.686213] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=47589][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:1, time_interval:3.00s, expected_run_time:09:01:55.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.714044] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:55.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.719445] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:55.721061] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=967108] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:55.721224] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:55.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.721623] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:55.722451] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:55.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.723015] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:55.723038] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=21] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3808) [2025-08-14 09:01:55.723045] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:01:55.723066] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:55.723079] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=13] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3809) [2025-08-14 09:01:55.723084] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:01:55.723274] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:55.723291] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=16] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3810) [2025-08-14 09:01:55.723296] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:01:55.723528] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=24] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:55.225435] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:01:55.234860] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162115234832]}) [2025-08-14 09:01:55.724108] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=497911][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:01:55.724455] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=342][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:01:55.724476] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:01:55.724491] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:01:55.724508] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:01:55.724521] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=488741][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:01:55.724528] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:01:55.724535] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:01:55.724540] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:00:52.998, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:22.998], idx=771, next_allow_print_ts="09:01:22.998") [2025-08-14 09:01:55.724557] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162115724513], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:55.724565] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=08:59:57.942, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:27.942], idx=868, next_allow_print_ts="09:00:27.942") [2025-08-14 09:01:55.724572] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162115724513], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:55.724587] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162115724513]}) [2025-08-14 09:01:54.723230] EDIAG [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=940194][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1007418, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1007416, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b51df0f 0x986bcde 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:54.734662] EDIAG [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1001365, owner: ObLog, click_count: 5, time dist:[FORMAT_END=998268, ALLOC_END=3097, APPEND_END=0, ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b51df0f 0x986bcde 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [567]OB_MALLOC COST TOO MUCH TIME, cost_time=495331, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [857]OB_MALLOC COST TOO MUCH TIME, cost_time=384583, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=950435, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=950426], seq:[0] [2025-08-14 09:01:55.724734] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:01:55.724752] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:01:55.724765] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:01:55.520209] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:55.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.725127] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=204605][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:55.520|207=204.92ms|210=0us|222=1us|total=204.93ms) [2025-08-14 09:01:55.725143] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] timer task use too much time(end=1755162115725142, start=1755162115520207, delta=204935) [2025-08-14 09:01:55.726173] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] [COMPACTION DAG STATUS] (start_time=1755162107720133, end_time=1755162115726171, dag_status={}) [2025-08-14 09:01:55.320727] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:55.726759] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=405416][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:55.320878] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:55.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.727167] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=405673][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:55.320|207=406.29ms|210=0us|222=2us|total=406.30ms) [2025-08-14 09:01:55.727180] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task use too much time(end=1755162115727179, start=1755162115320876, delta=406303) [2025-08-14 09:01:54.723230] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=940194] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136826952869001, v:0}, clog_base_lsn:{lsn:5099962368}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1755071646732636000, v:0}, all_id_meta:{id_meta:[{limited_id:1755161555795565000, latest_log_ts:{val:1755161545796642001, v:0}}, {limited_id:22000001, latest_log_ts:{val:1755135645787806003, v:0}}, {limited_id:60000001, latest_log_ts:{val:1755135646400653001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:50, palf_env_:0x7f9950bf4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-08-14 09:01:55.737204] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:54.734662] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=12] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136829134139001, v:0}, clog_base_lsn:{lsn:8186781696}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1753840867372367034, v:0}, all_id_meta:{id_meta:[{limited_id:1755161729775179000, latest_log_ts:{val:1755161719775179001, v:0}}, {limited_id:18000001, latest_log_ts:{val:1755157440846556001, v:0}}, {limited_id:60000001, latest_log_ts:{val:1755135657266496001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:46, palf_env_:0x7f99321fa030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-08-14 09:01:55.737575] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=1001365] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:55.737594] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=8] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:55.738378] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=40] get wrs ts(ls_id={id:1}, delta=567999206, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:55.738398] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:55.738431] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161101770923, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755161572801635) [2025-08-14 09:01:55.739003] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=567999206, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:55.739016] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:55.737089] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1007418] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:55.734428] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6311] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:01:55.739250] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=2138] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:55.744642] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:55.744683] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:55.752361] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:55.752760] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=13] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [374]OB_MALLOC COST TOO MUCH TIME, cost_time=989907, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=50, WASH_BLOCK_END=989853, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=3], seq:[0, 1, 2, 3, 4] [2025-08-14 09:01:55.764266] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=15] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:01:55.772586] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:55.731460] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [784][T1002_TBalance][T1002][YB42AC4D0007-00063C4964C9A63F-0-0] [lt=12][errcode=-4012] query failed(ret=-4012, conn=0x7f9966b3c3b0, start=1755161923885572, sql=SELECT distinct unit_group_id, status from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1002) order by unit_group_id ) [2025-08-14 09:01:55.776542] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:55.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.782147] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=18] gc stale ls task succ [2025-08-14 09:01:55.743215] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:55.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.772673] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [784][T1002_TBalance][T1002][YB42AC4D0007-00063C4964C9A63F-0-0] [lt=41149][errcode=-4012] read failed(ret=-4012) [2025-08-14 09:01:55.723551] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=17] temporary file module use no blocks [2025-08-14 09:01:55.790513] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=28] PNIO [ratelimit] time: 1755162115790511, bytes: 0, bw: 0.000000 MB/s, add_ts: 1010642, add_bytes: 0 [2025-08-14 09:01:55.800619] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:55.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.826801] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:55.832210] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=4964][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:55.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.837620] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=4] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:55.839312] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=4137] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:55.852769] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c0b90, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571c0e70, time_wheel:0x7f9959fef990, thread_pool:0x7f9959fef170, is_running:1, time_interval:3.00s, expected_run_time:09:01:55.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.852815] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:01:55.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.922575] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:55.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.937687] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=14] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:55.939339] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:56.021949] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:56.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.022635] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:55.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.122696] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:56.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.222759] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:56.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.273447] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:56.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.303599] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:56.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.322825] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:56.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.422889] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:56.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.522913] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:56.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.524885] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:56.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.622939] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:56.564, task_priority:1, with_handle_protected:1}) [1003]OB_MALLOC COST TOO MUCH TIME, cost_time=978089, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=978088], seq:[0] [303]OB_MALLOC COST TOO MUCH TIME, cost_time=980878, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=53882], seq:[0, 1] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=882691, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1], seq:[0, 1] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=939265, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=939259], seq:[0] [2025-08-14 09:01:56.693947] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:01:56.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.037727] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=11] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:56.717996] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=679230] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:56.039423] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:56.224933] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:01:56.718181] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=492495][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:01:56.718214] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:01:56.718225] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:01:56.718256] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:01:55.727222] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=50][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72760b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b5b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.719, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.718675] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=989950][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:55.727|207=991.45ms|210=0us|222=2us|total=991.48ms) [2025-08-14 09:01:56.718699] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4389] timer task use too much time(end=1755162116718698, start=1755162115727220, delta=991478) [2025-08-14 09:01:56.718712] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=980465, expected_run_ts_us_=1755162115738245, ObClockGenerator::getRealClock()=1755162116718711, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7283810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.718729] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7283810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.718740] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=982263, expected_run_ts_us_=1755162115736477, ObClockGenerator::getRealClock()=1755162116718740, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728b690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.718751] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728b690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.718767] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] task scheduled too delay(scheduled_delay=980259, expected_run_ts_us_=1755162115738508, ObClockGenerator::getRealClock()=1755162116718767, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739be30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728b810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.718777] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739be30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728b810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.718787] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=987521, expected_run_ts_us_=1755162115731265, ObClockGenerator::getRealClock()=1755162116718787, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7281e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.718797] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7281e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.718807] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=976223, expected_run_ts_us_=1755162115742583, ObClockGenerator::getRealClock()=1755162116718807, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7edc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f0910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.718817] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7edc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f0910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.718826] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=975967, expected_run_ts_us_=1755162115742859, ObClockGenerator::getRealClock()=1755162116718826, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ebe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f0a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.718836] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ebe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f0a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.718846] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=970575, expected_run_ts_us_=1755162115748270, ObClockGenerator::getRealClock()=1755162116718846, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891971110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.718855] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891971110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.718866] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=968484, expected_run_ts_us_=1755162115750381, ObClockGenerator::getRealClock()=1755162116718866, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891968c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.750, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.718876] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891968c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.750, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.718886] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=963664, expected_run_ts_us_=1755162115755221, ObClockGenerator::getRealClock()=1755162116718886, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7283990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.718895] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7283990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.718905] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=961301, expected_run_ts_us_=1755162115757603, ObClockGenerator::getRealClock()=1755162116718905, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728d1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7291110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.718915] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728d1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7291110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.718925] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=954653, expected_run_ts_us_=1755162115764272, ObClockGenerator::getRealClock()=1755162116718925, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b4790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.718935] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b4790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.718945] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=954362, expected_run_ts_us_=1755162115764582, ObClockGenerator::getRealClock()=1755162116718945, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919afe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b9990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.718954] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919afe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b9990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.718964] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=950879, expected_run_ts_us_=1755162115768085, ObClockGenerator::getRealClock()=1755162116718964, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d62b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891968f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.718974] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d62b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891968f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.718984] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=950821, expected_run_ts_us_=1755162115768162, ObClockGenerator::getRealClock()=1755162116718984, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891970210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.718993] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891970210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719003] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=946888, expected_run_ts_us_=1755162115772115, ObClockGenerator::getRealClock()=1755162116719003, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891968b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719013] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891968b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719022] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=943879, expected_run_ts_us_=1755162115775143, ObClockGenerator::getRealClock()=1755162116719022, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196b210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719032] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196b210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719042] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=948812, expected_run_ts_us_=1755162115770229, ObClockGenerator::getRealClock()=1755162116719042, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7280990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719052] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7280990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719062] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=947314, expected_run_ts_us_=1755162115771747, ObClockGenerator::getRealClock()=1755162116719061, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7397a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719071] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7397a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719081] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=934357, expected_run_ts_us_=1755162115784723, ObClockGenerator::getRealClock()=1755162116719080, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7280690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719090] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7280690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719100] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=938039, expected_run_ts_us_=1755162115781061, ObClockGenerator::getRealClock()=1755162116719100, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7efe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719109] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7efe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719119] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=936461, expected_run_ts_us_=1755162115782658, ObClockGenerator::getRealClock()=1755162116719119, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f0010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719128] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f0010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719138] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=922315, expected_run_ts_us_=1755162115796823, ObClockGenerator::getRealClock()=1755162116719138, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7282c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719148] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7282c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719157] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=916940, expected_run_ts_us_=1755162115802217, ObClockGenerator::getRealClock()=1755162116719157, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728b210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.802, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719167] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728b210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.802, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719177] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=904846, expected_run_ts_us_=1755162115814330, ObClockGenerator::getRealClock()=1755162116719176, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728a190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.814, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719186] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728a190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.814, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719206] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=901385, expected_run_ts_us_=1755162115817820, ObClockGenerator::getRealClock()=1755162116719205, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7282010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.817, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719215] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7282010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.817, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719225] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=900251, expected_run_ts_us_=1755162115818973, ObClockGenerator::getRealClock()=1755162116719225, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7ee810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719246] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7ee810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719256] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=900138, expected_run_ts_us_=1755162115819118, ObClockGenerator::getRealClock()=1755162116719256, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7ee990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.819, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719266] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7ee990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.819, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719276] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=905783, expected_run_ts_us_=1755162115813492, ObClockGenerator::getRealClock()=1755162116719276, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a0a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.813, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719285] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a0a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.813, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719295] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=890460, expected_run_ts_us_=1755162115828835, ObClockGenerator::getRealClock()=1755162116719295, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919831b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891970c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.828, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719305] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919831b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891970c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.828, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719315] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=884275, expected_run_ts_us_=1755162115835039, ObClockGenerator::getRealClock()=1755162116719315, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891973c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.835, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719324] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891973c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.835, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719334] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=871647, expected_run_ts_us_=1755162115847687, ObClockGenerator::getRealClock()=1755162116719334, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891968210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.847, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719344] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891968210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.847, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719353] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=874606, expected_run_ts_us_=1755162115844747, ObClockGenerator::getRealClock()=1755162116719353, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891970b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.844, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719363] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891970b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.844, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719373] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=874398, expected_run_ts_us_=1755162115844974, ObClockGenerator::getRealClock()=1755162116719373, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7282190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.844, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719382] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7282190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:55.844, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.719411] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ebbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7ee510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:56.615, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:55.726978] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:213) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-08-14 09:01:56.719511] INFO [COMMON] handle (memory_dump.cpp:544) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=991032] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [1003]OB_MALLOC COST TOO MUCH TIME, cost_time=982406, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=978098], seq:[0, 1] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=955633, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=950462], seq:[0, 1] [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=985021, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=19, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=984980], seq:[0, 1, 2, 3, 4] [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=985084, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=985038], seq:[0, 1] [621]OB_MALLOC COST TOO MUCH TIME, cost_time=984914, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=984888], seq:[0, 1, 2, 3, 4] [621]OB_MALLOC COST TOO MUCH TIME, cost_time=984923, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=984922], seq:[0, 1] [857]OB_MALLOC COST TOO MUCH TIME, cost_time=984410, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=15, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=984368], seq:[0, 1, 2, 3, 4] [857]OB_MALLOC COST TOO MUCH TIME, cost_time=984424, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=984423], seq:[0, 1] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=971694, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=971670], seq:[0, 1, 2] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=971724, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=971722], seq:[0, 1] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=971669, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=971645], seq:[0, 1, 2] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=971679, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=971678], seq:[0, 1] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=971694, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=971681], seq:[0, 1, 2] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=971704, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=971703], seq:[0, 1] [389]OB_MALLOC COST TOO MUCH TIME, cost_time=971507, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=971490], seq:[0, 1, 2] [389]OB_MALLOC COST TOO MUCH TIME, cost_time=971533, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=971515], seq:[0, 1] [387]OB_MALLOC COST TOO MUCH TIME, cost_time=971400, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=971385], seq:[0, 1, 2] [387]OB_MALLOC COST TOO MUCH TIME, cost_time=971411, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=971410], seq:[0, 1] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=971419, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=971405], seq:[0, 1, 2] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=971427, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=971426], seq:[0, 1] [383]OB_MALLOC COST TOO MUCH TIME, cost_time=971364, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=971351], seq:[0, 1, 2] [383]OB_MALLOC COST TOO MUCH TIME, cost_time=971374, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=971374], seq:[0, 1] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=971316, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=971301], seq:[0, 1, 2] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=971326, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=971325], seq:[0, 1] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=941594, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=939281], seq:[0, 1] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=970891, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=970867], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=970900, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=970899], seq:[0, 1] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=968762, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=968742], seq:[0, 1, 2] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=968775, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=968774], seq:[0, 1] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=968721, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=968706], seq:[0, 1, 2] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=968729, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=968729], seq:[0, 1] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=946009, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=945990], seq:[0, 1, 2] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=946018, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=946017], seq:[0, 1] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=941628, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=941623], seq:[0] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=941637, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=941637], seq:[0, 1] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=941365, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=941360], seq:[0] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=941372, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=941372], seq:[0, 1] [651]OB_MALLOC COST TOO MUCH TIME, cost_time=868036, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=868031], seq:[0] [651]OB_MALLOC COST TOO MUCH TIME, cost_time=868044, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=868043], seq:[0, 1] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=969005, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=968982, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3] [1031]OB_MALLOC COST TOO MUCH TIME, cost_time=502132, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=502123, ALLOC_CHUNK_END=4], seq:[0, 1] [381]OB_MALLOC COST TOO MUCH TIME, cost_time=500903, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=500893, ALLOC_CHUNK_END=5], seq:[0, 1] [683]OB_MALLOC COST TOO MUCH TIME, cost_time=289621, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=289612, ALLOC_CHUNK_END=5], seq:[0, 1] [797]OB_MALLOC COST TOO MUCH TIME, cost_time=202825, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=202815, ALLOC_CHUNK_END=5], seq:[0, 1] [2025-08-14 09:01:56.718271] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:01:56.720338] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=2057][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:01:56.720361] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:01:56.720373] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:01:56.720384] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:01:56.720402] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:01:56.720410] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:00:52.998, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:22.998], idx=771, next_allow_print_ts="09:01:22.998") [2025-08-14 09:01:56.721176] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=21] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:56.721583] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:55.926854] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:56.722401] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=794342][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:56.720434] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=08:59:57.942, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:27.942], idx=868, next_allow_print_ts="09:00:27.942") [2025-08-14 09:01:56.723694] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=3104][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:01:56.723712] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:01:56.723723] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:01:55.737225] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:981) [1003][BlackListServic][T1][YB42AC4D0007-00063C495F09BD68-0-0] [lt=8][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, record_ret=-4012, ret=-4012) [2025-08-14 09:01:56.723176] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=568002055, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:56.723834] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=650] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=5, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:56.724433] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] get wrs ts(ls_id={id:1}, delta=568002055, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:56.724447] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:56.723226] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:56.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.738422] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=993664] current monitor number(seq_=-1) [2025-08-14 09:01:55.738619] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9233-0-0] [lt=28][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162115241601) [2025-08-14 09:01:55.739154] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=365996529, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:56.725087] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=984440] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:56.725076] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9233-0-0] [lt=984958][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162115241601}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162116723875}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162116723875}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:01:56.725210] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:763) [1003][BlackListServic][T0][YB42AC4D0007-00063C495F09BD68-0-0] [lt=985063] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, process_time=35960361) [2025-08-14 09:01:56.725227] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1018) [1003][BlackListServic][T0][YB42AC4D0007-00063C495F09BD68-0-0] [lt=17][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, aret=-4012, ret=-4012) [2025-08-14 09:01:56.725251] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1808) [1003][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2025-08-14 09:01:56.725260] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1079) [1003][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2025-08-14 09:01:56.725269] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1748) [1003][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1753069619, tenant_id=1) [2025-08-14 09:01:56.725278] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [317][pnio1][T1][YB42AC4D0007-00063C49607B9233-0-0] [lt=32][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1, delta=1884369, last_succ_cluster_heartbeat_tstamp_=1755162114840908) [2025-08-14 09:01:56.725279] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [1003][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4012] query failed(ret=-4012, conn=0x7f98f673ed90, start=1755162080740150, sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2025-08-14 09:01:56.725292] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:114) [1003][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4012] read failed(ret=-4012) [2025-08-14 09:01:55.752110] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=30] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:56.725324] WDIAG ev_invoke_pending (ev.c:2136) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=971743][errcode=0] epoll handler too much time :973225 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:55.752218] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:55.752272] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=52] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:56.725460] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=971718][errcode=0] epoll handler too much time :973196 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:55.752511] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:55.752661] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:56.725566] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=971427][errcode=0] epoll handler too much time :972927 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:55.763107] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:56.725575] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=961014][errcode=0] epoll handler too much time :962480 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:55.877524] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=41][errcode=0] EASY SLOW: thread: 7f99547f7300, lastrun: 1755162115.752759 cost: 0.124763 loop:26488, slowcnt: 1 [2025-08-14 09:01:55.752695] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:56.725604] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=971440][errcode=0] epoll handler too much time :972916 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:55.752784] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:56.725642] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=971389][errcode=0] epoll handler too much time :972865 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:55.782644] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=17] gc stale ls task succ [2025-08-14 09:01:55.782353] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=39074][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:55.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.725719] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=941937][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:55.782|207=943.37ms|210=0us|222=1us|total=943.38ms) [2025-08-14 09:01:56.725731] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task use too much time(end=1755162116725730, start=1755162115782351, delta=943379) [2025-08-14 09:01:56.725744] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] task scheduled too delay(scheduled_delay=742211, expected_run_ts_us_=1755162115983531, ObClockGenerator::getRealClock()=1755162116725743, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:55.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.725756] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:55.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.753411] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:55.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.725822] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=970940][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:55.753|207=972.41ms|210=0us|222=1us|total=972.42ms) [2025-08-14 09:01:56.725834] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task use too much time(end=1755162116725834, start=1755162115753409, delta=972425) [2025-08-14 09:01:55.788799] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=65148] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:56.725841] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=0] task scheduled too delay(scheduled_delay=970353, expected_run_ts_us_=1755162115755487, ObClockGenerator::getRealClock()=1755162116725841, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:55.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.725845] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=935626] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:56.725851] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:55.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.725875] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=0] task scheduled too delay(scheduled_delay=636559, expected_run_ts_us_=1755162116089316, ObClockGenerator::getRealClock()=1755162116725875, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:56.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.725886] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:56.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.725868] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=23] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623017163193, flush_io_finished_ret_:0, flush_io_finished_round_:26287, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:56.725897] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=521600, expected_run_ts_us_=1755162116204297, ObClockGenerator::getRealClock()=1755162116725897, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:56.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.755682] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:55.755769] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=12] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:56.725977] WDIAG ev_invoke_pending (ev.c:2136) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=968743][errcode=0] epoll handler too much time :970211 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:56.722554] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5660] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:01:55.783032] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:56.726112] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=941654][errcode=0] epoll handler too much time :943089 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:56.726118] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3506] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:01:55.783334] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:56.726146] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=941388][errcode=0] epoll handler too much time :942820 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:56.726161] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:01:56.720456] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [419]OB_MALLOC COST TOO MUCH TIME, cost_time=703411, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=703408], seq:[0] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=703423, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=703422], seq:[0, 1] [2025-08-14 09:01:55.856810] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EE9B-0-0] [lt=21] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-08-14 09:01:56.022950] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=16] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:01:56.728326] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=568002055, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:56.728351] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:55.932251] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:55.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.732705] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=799241][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:55.932|207=800.45ms|210=0us|222=2us|total=800.47ms) [2025-08-14 09:01:56.732721] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4389] timer task use too much time(end=1755162116732720, start=1755162115932249, delta=800471) [393]OB_MALLOC COST TOO MUCH TIME, cost_time=976016, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=969026, WASH_BLOCK_END=6978, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=11], seq:[0, 1, 2, 3, 4] [683]OB_MALLOC COST TOO MUCH TIME, cost_time=296414, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=289634, WASH_BLOCK_END=6780], seq:[0, 1, 2] [797]OB_MALLOC COST TOO MUCH TIME, cost_time=209665, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=202832, WASH_BLOCK_END=6833], seq:[0, 1, 2] [1031]OB_MALLOC COST TOO MUCH TIME, cost_time=511140, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=502150, WASH_BLOCK_END=8990], seq:[0, 1, 2] [381]OB_MALLOC COST TOO MUCH TIME, cost_time=509786, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=500913, WASH_BLOCK_END=8873], seq:[0, 1, 2] [2025-08-14 09:01:56.725907] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:56.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:55.755742] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:56.733266] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=976046][errcode=0] epoll handler too much time :977531 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:56.436404] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=5][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fa190, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa470, time_wheel:0x7f99341cdbf0, thread_pool:0x7f99341cd3d0, is_running:1, time_interval:2.00s, expected_run_time:09:01:56.334, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.733337] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=296481][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:56.436|207=296.93ms|210=0us|222=1us|total=296.95ms) [2025-08-14 09:01:56.733357] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task use too much time(end=1755162116733357, start=1755162116436402, delta=296955) [2025-08-14 09:01:56.523389] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fe410, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe6f0, time_wheel:0x7f9922df9d40, thread_pool:0x7f9922df9520, is_running:1, time_interval:2.00s, expected_run_time:09:01:56.432, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.733429] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=209720][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:56.523|207=210.04ms|210=0us|222=1us|total=210.05ms) [2025-08-14 09:01:56.733442] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task use too much time(end=1755162116733442, start=1755162116523387, delta=210055) [2025-08-14 09:01:56.725943] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=968795][errcode=0] epoll handler too much time :970273 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:56.223375] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:56.224763] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:56.735385] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=509845][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162116224758], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:56.735377] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:227) [1031][PxTargetMgr0][T1][Y0-0000000000000000-0-0] [lt=511220] refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3811) [2025-08-14 09:01:56.735405] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=19] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162116224758]}) [2025-08-14 09:01:56.735410] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:01:56.735419] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:01:56.735430] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:01:56.735430] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:56.735442] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=12] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3812) [2025-08-14 09:01:56.735445] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:56.735450] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:01:56.735455] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162116735442], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:56.735467] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162116735442]}) [2025-08-14 09:01:56.735490] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162116735442]}) [2025-08-14 09:01:56.735671] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:56.735682] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:01:56.735695] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:01:56.735700] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=29] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3813) [2025-08-14 09:01:56.735709] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162116735678], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:56.735719] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:01:56.735724] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162116735678]}) [2025-08-14 09:01:56.726583] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get_max_applied_scn(scn={val:1755161744727824001, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:49, palf_committed_end_lsn_:{lsn:7323686009}, palf_committed_end_scn_:{val:1755161747742105001, v:0}, last_check_scn_:{val:1755161749741193000, v:0}, max_applied_cb_scn_:{val:1755161744727824001, v:0}}) [2025-08-14 09:01:56.735811] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9208] get_min_unreplayed_log_info(lsn={lsn:7323685240}, scn={val:1755100946796590002, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873485, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:8946764, ret_code_:0}}, next_to_submit_lsn_:{lsn:7323685240}, next_to_submit_scn_:{val:1755100946796590002, v:0}, base_lsn_:{lsn:7180210176}, base_scn_:{val:1753840803166611000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:7323685240}, end_lsn:{lsn:7323685240}, read_buf:{buf_len_:3747840, buf_:0x7f98f7a05000}, block_size:67104768, log_storage_:0x7f9901131bf0, log_storage_:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:28, read_io_size:98508800, read_disk_cost_ts:645339, start_lsn:{lsn:7180210176}}}}}}) [2025-08-14 09:01:54.776077] WDIAG [STORAGE] prepare_mds_tables (ob_multiple_merge.cpp:1662) [14945][T1_TimerWK8_MergeLoop][T1][YB42AC4D0007-00063C4978199C2C-0-0] [lt=10][errcode=-4012] fail to get mds sstables(ret=-4012, get_table_param=0x7f97efaced40, snapshot_version=9223372036854775807) [2025-08-14 09:01:55.921356] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:55.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.736206] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=813606][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:55.921|207=814.84ms|210=0us|222=2us|total=814.89ms) [2025-08-14 09:01:56.736255] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=48][errcode=-4389] timer task use too much time(end=1755162116736254, start=1755162115921354, delta=814900) [2025-08-14 09:01:56.736856] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=91] get wrs ts(ls_id={id:1001}, delta=371012747, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:56.736931] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=72] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:56.737250] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:56.737944] WDIAG [STORAGE.TRANS] do_thread_task_ (ob_black_list.cpp:219) [1003][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4012] fail to execute sql(ret=-4012, ret="OB_TIMEOUT", sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2025-08-14 09:01:56.737968] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:396) [1003][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=23] start to print blacklist info [2025-08-14 09:01:55.783311] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [723][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=16] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=16, total_destructed_detector=16, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f99333562b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-08-14 09:01:56.738474] WDIAG [DETECT] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [723][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=953718][errcode=0] cost too much time:ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t), (this=|threshold=100.00ms|start at 09:01:56.738|total=955.24ms) [2025-08-14 09:01:56.739216] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919abd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:56.721, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.739253] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919abb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:56.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.739268] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b4f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:56.726, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.739282] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fdc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7282310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:56.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.739296] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7282490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:56.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.741732] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [14780][T1001_TimerWK4_SSTableGC][T1001][YB42AC4D0007-00063C4974099C4E-0-0] [lt=14][errcode=0] cost too much time:mds_dump_node.h:int oceanbase::storage::mds::MdsDumpNode::init(const uint8_t, const uint8_t, const UserMdsNode &, oceanbase::common::ObIAllocator &) [K = oceanbase::compaction::ObMediumCompactionInfoKey, V = oceanbase::compac..., (this=|threshold=1.00ms|start at 09:01:52.732|219=1.01s|238=2.99s|total=4.01s) [2025-08-14 09:01:56.742886] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:56.719468] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4015] clock out of order(cur_ts=1755162115782611, cur_ts_=1755162115740571, delta=42040) [2025-08-14 09:01:56.718033] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=677568] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:56.744420] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:56.744898] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8870] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:55.813059] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [851][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=28] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=16, total_destructed_detector=16, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f9922fee2b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-08-14 09:01:56.746051] WDIAG [DETECT] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [851][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=931581][errcode=0] cost too much time:ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t), (this=|threshold=100.00ms|start at 09:01:56.746|total=933.01ms) [2025-08-14 09:01:56.746195] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=18] update_check_range(min_check_version=2540, max_check_version=2540, global_version=2637) [2025-08-14 09:01:56.747859] WDIAG [SHARE] read_unit_groups (ob_unit_table_operator.cpp:1298) [784][T1002_TBalance][T1002][YB42AC4D0007-00063C4964C9A63F-0-0] [lt=14662][errcode=-4012] execute sql failed(sql=SELECT distinct unit_group_id, status from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1002) order by unit_group_id , ret=-4012) [2025-08-14 09:01:56.749886] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:56.749950] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:56.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.750015] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728be10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:56.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.750034] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72847b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728a310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:56.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.750507] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:56.750544] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:56.750768] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:56.751169] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119812272, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-08-14 09:01:56.751223] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] get wrs ts(ls_id={id:1}, delta=367009625, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:56.751247] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:56.751865] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1001}, delta=372022994, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:56.751880] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:56.752423] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:56.754114] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=7] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:56.754643] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:56.754755] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=10] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:01:56.754852] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10] LogLoopThread round_cost_time(us)(round_cost_time=8) [2025-08-14 09:01:56.754866] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:01:56.754883] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:01:56.756074] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=37] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:56.756100] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=21] temporary file module use no blocks [2025-08-14 09:01:56.756111] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:56.756127] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:56.756146] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=19] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623018195495, flush_io_finished_ret_:0, flush_io_finished_round_:26261, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:56.742920] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:56.756811] WDIAG ev_invoke_pending (ev.c:2136) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13868][errcode=0] epoll handler too much time :13898 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:56.747815] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=7337][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:56.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.757790] WDIAG [MVCC] refresh_ (ob_multi_version_garbage_collector.cpp:534) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4012] collect snapshot info sql failed(ret=-4012, this={this:0x7f99200ff540, last_study_timestamp:1755161007611441, last_refresh_timestamp:1755161188828450, last_reclaim_timestamp:1755161664766510, last_sstable_overflow_timestamp:0, has_error_when_study:true, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1755161006924036004, v:0}, is_inited:true}, collector={reserved_snapshot_version:{val:4611686018427387903, v:0}, reserved_snapshot_type:0, reserved_status:-1, reserved_create_time:0, reserved_addr:"0.0.0.0:0", is_this_server_disabled:false, status:0}) [2025-08-14 09:01:56.760079] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b9e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:56.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.760116] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891806f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:56.751, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.760131] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739fb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:56.750, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.761662] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:56.763192] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=17] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:01:56.763802] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:56.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.764303] INFO [COMMON] print_io_status (ob_io_manager.cpp:2639) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:0, norm_iops:0, bw:255, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:10623, total:10631) [2025-08-14 09:01:56.764834] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [411][sql_nio4][T0][Y0-0000000000000000-0-0] [lt=33] new mysql sessid created(conn.sessid_=3221590965, support_ssl=false) [2025-08-14 09:01:56.766308] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=16] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:01:56.768593] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9048][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:56.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.770107] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73539b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7396b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:56.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.770898] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:56.772698] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] [COMPACTION DAG STATUS] (start_time=1755162107720054, end_time=1755162116772697, dag_status={}) [2025-08-14 09:01:56.776350] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] [COMPACTION DAG STATUS] (start_time=1755162107720127, end_time=1755162116776349, dag_status={}) [2025-08-14 09:01:56.778185] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:56.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.778588] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=11] blacklist_loop exec finished(cost_time=16, is_enabled=true, send_cnt=0) [2025-08-14 09:01:56.782210] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:56.782208] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:56.782222] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=13] gc stale ls task succ [2025-08-14 09:01:56.782700] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=941624] gc stale ls task succ [2025-08-14 09:01:56.795103] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=22] PNIO [ratelimit] time: 1755162116795101, bytes: 0, bw: 0.000000 MB/s, add_ts: 1004590, add_bytes: 0 [2025-08-14 09:01:56.800324] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f1510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:56.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.803428] INFO handle (memory_dump.cpp:671) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=55] statistics: tenant_cnt: 6, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 168 272 0 1 2 110 1 0 1 3 7 0 0 1 4 1 1 0 1 5 86 6 0 1 6 13 0 0 1 7 22 2 0 1 8 53 1 0 1 12 17 3 0 1 13 10 1 0 1 14 1 1 0 1 15 1 1 0 1 16 5 6 0 1 18 4 3 0 1 20 2 1 0 1 22 1 1 0 1 24 1 0 0 1 25 6 0 0 500 0 563 276 0 500 7 192 6 0 500 8 41 1 0 500 9 2 1 0 500 10 2 1 0 500 16 50 4 0 500 17 52 7 0 500 22 38 64 0 500 23 27 10 0 508 0 2 7 0 508 8 5 1 0 509 0 2 7 0 509 8 5 1 0 1001 0 147 251 0 1001 2 26 1 0 1001 3 5 0 0 1001 4 1 1 0 1001 5 85 6 0 1001 6 11 0 0 1001 7 30 2 0 1001 8 36 1 0 1001 12 17 3 0 1001 13 3 1 0 1001 14 1 1 0 1001 16 6 2 0 1001 18 4 3 0 1001 20 4 1 0 1001 25 8 0 0 1002 0 262 278 0 1002 2 198 1 0 1002 3 4 0 0 1002 4 1 1 0 1002 5 101 6 0 1002 6 14 0 0 1002 7 6 2 0 1002 8 73 1 0 1002 12 18 3 0 1002 13 7 1 0 1002 14 1 2 0 1002 16 8 8 0 1002 18 9 3 0 1002 20 2 1 0 1002 21 1 0 0 1002 24 1 0 0 1002 25 7 0 0 cost_time: 79175 [2025-08-14 09:01:56.803473] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=23] [MEMORY] tenant: 1, limit: 2147483648 hold: 1008979968 cache_hold: 56623104 cache_used: 56623104 cache_item_count: 27 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 444993536 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 230686720 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 1163264 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 483328 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 53714944 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 1458176 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 14204928 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 111149056 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35651584 limit= 429496720 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 8003584 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= PS_CACHE_CTX_ID hold_bytes= 65536 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 8536064 limit= 9223372036854775807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 3481600 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3964928 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 212992 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 884736 limit= 9223372036854775807 [2025-08-14 09:01:56.803516] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 444993536 used= 389560096 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 70 washed_blocks= 287 washed_size= 70221824 [MEMORY] request_cached_chunk_cnt= 4 [MEMORY] hold= 389560096 used= 379314179 count= 32870 avg_used= 11539 mod=SUMMARY [2025-08-14 09:01:56.803543] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=15] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 230686720 used= 228884480 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 block_cnt= 110 chunk_cnt= 110 mod=Memstore [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:01:56.803563] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=14] [MEMORY] tenant_id= 1 ctx_id= EXECUTE_CTX_ID hold= 1163264 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 15 washed_size= 13516800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:56.803580] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:01:56.803599] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 53714944 used= 47487424 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 84 washed_blocks= 334 washed_size= 126640128 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 47487424 used= 43098388 count= 3994 avg_used= 10790 mod=SUMMARY [2025-08-14 09:01:56.803618] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1 ctx_id= WORK_AREA hold= 1458176 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 13 washed_blocks= 66 washed_size= 25804800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:56.803637] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 14204928 used= 10970224 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 277 washed_size= 31932416 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10970224 used= 6078924 count= 18573 avg_used= 327 mod=SUMMARY [2025-08-14 09:01:56.803657] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 111149056 used= 108896256 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 block_cnt= 211 chunk_cnt= 53 mod=CoStack [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 mod=SUMMARY [2025-08-14 09:01:56.803678] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 56623104 used= 56623104 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 56623104 used= 0 count= 27 avg_used= 0 mod=SUMMARY [2025-08-14 09:01:56.803697] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 34365376 limit= 429496720 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 34365376 used= 31086608 count= 475 avg_used= 65445 mod=SUMMARY [2025-08-14 09:01:56.803715] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 8003584 used= 4947392 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 10 washed_blocks= 23 washed_size= 12967936 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4947392 used= 4793344 count= 604 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:56.803731] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:56.803751] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 65536 used= 31024 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 2031616 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 31024 used= 27058 count= 20 avg_used= 1352 mod=SUMMARY [2025-08-14 09:01:56.803771] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=14] [MEMORY] tenant_id= 1 ctx_id= RPC_CTX_ID hold= 8536064 used= 2015232 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2015232 used= 1324288 count= 86 avg_used= 15398 mod=SUMMARY [2025-08-14 09:01:56.803787] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 3481600 used= 2391232 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 10 washed_size= 4907008 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2391232 used= 2367040 count= 36 avg_used= 65751 mod=SUMMARY [2025-08-14 09:01:56.803804] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 3964928 used= 3080192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 3 washed_size= 229376 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3080192 used= 2983936 count= 376 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:56.803824] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1 ctx_id= SCHEMA_SERVICE hold= 212992 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1884160 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:56.803844] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:56.803862] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1 ctx_id= MERGE_NORMAL_CTX_ID hold= 884736 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 6 washed_blocks= 9 washed_size= 11698176 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:56.808366] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=74][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:56.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.810357] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d22b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72eaf90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:56.808, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.818061] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=12] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:56.818097] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=26055] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:56.820414] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ec010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:56.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.820447] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891972790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:56.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.822446] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=9] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:01:56.822481] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:56.824341] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=689900][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a66b8, timer:0x7f9966fddee0, task:0x7f994f1b1e78, task_type:"N9oceanbase5share23ObStorageIOUsageRepoter23TenantIOUsageReportTaskE", scheduled_time:1755161694839037, pushed_time:0, delay:5000000}, running_task_token={this:0x7f994e1a65d8, timer:0x7f9966fddee0, task:0x7f994f1bbcc0, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161694824965, pushed_time:1755161925772608, delay:5000000}, total_delay=421985303, threashold=1000000, tenant_id=1) [2025-08-14 09:01:56.832797] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:56.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.835965] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=22] LogLoopThread round_cost_time(us)(round_cost_time=14) [2025-08-14 09:01:56.836708] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a5df8, timer:0x7f99200ff180, task:0x7f99200ff1c0, task_type:"N9oceanbase8observer18ObTableLoadService15ObHeartBeatTaskE", scheduled_time:1755162055729056, pushed_time:0, delay:10000000}, running_task_token={this:0x7f991b7a5fb8, timer:0x7f99200ff180, task:0x7f99200ff1f0, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162050755138, pushed_time:1755162084760013, delay:1000000}, total_delay=61107651, threashold=1000000, tenant_id=1002) [2025-08-14 09:01:56.840505] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ab290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:56.831, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.840538] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a1c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:56.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.843994] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=367102212, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:56.844023] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:56.844660] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1001}, delta=372116639, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:56.844677] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:56.850566] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196ac10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:56.849, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.854907] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:01:56.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.860633] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a0910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:56.852, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.870680] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891972610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:56.861, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.900882] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919859b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891972490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:56.896, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:56.905101] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:233) [709][T1001_RFLWorker][T1001][YB42AC4D0007-00063C495BE99C16-0-0] [lt=2429] ObRemoteFetchWorker is running(thread_index=0) [2025-08-14 09:01:56.918118] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=23] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:56.918155] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:56.922506] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=8] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:01:56.922531] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:56.925761] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=846798][errcode=0] EASY SLOW: thread: 7f99542fb300, lastrun: 1755162115.752503 cost: 1.173256 loop:26416, slowcnt: 3 [2025-08-14 09:01:56.932833] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:56.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.995324] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:56.989, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.999181] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:56.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:57.015390] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:57.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:57.018155] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=22] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:57.018220] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=11] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:57.022569] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:57.027623] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=703443] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:01:57.029527] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:57.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:57.032892] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:56.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:57.095814] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:57.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:57.118193] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=22] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:57.118248] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=29] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:57.122641] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:57.132939] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:57.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:57.216542] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:57.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:57.218216] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=32] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:57.218307] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=19] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:57.222700] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:57.223880] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:01:57.223912] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:01:57.223937] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:01:57.223954] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:01:57.223967] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:01:57.223987] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:01:57.224003] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:01:57.224026] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:01:57.224044] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:01:57.224065] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:01:57.224082] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:01:57.224107] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:00:52.998, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:22.998], idx=771, next_allow_print_ts="09:01:22.998") [2025-08-14 09:01:57.224127] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=08:59:57.942, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:27.942], idx=868, next_allow_print_ts="09:00:27.942") [2025-08-14 09:01:57.224322] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:01:57.224343] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:01:57.224361] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:01:57.232991] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:57.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:57.235799] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:57.235830] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=30] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3814) [2025-08-14 09:01:57.235838] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:01:57.235855] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=11] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:57.235879] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=24] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3815) [2025-08-14 09:01:57.235887] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:01:57.236020] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=40] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:57.236086] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=12] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:57.236104] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=18] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3816) [2025-08-14 09:01:57.236112] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:01:57.240615] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=64][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:57.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:57.246712] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:57.239, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:57.266791] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:57.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:57.318369] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:57.322744] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:57.333036] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:57.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:57.336596] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:01:57.418441] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=23] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:57.422803] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:57.433099] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:01:57.498087] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:01:57.518496] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=24] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:57.522823] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:57.618561] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=22] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:57.622877] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:57.694457] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:01:57.718769] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=26] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:57.719574] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=2][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:57.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:57.721285] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:57.721992] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:57.722952] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:57.236044] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162117236016], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:57.724366] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=487582][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162117236016], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:57.724390] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162117236016]}) [2025-08-14 09:01:57.724405] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:01:57.724418] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:01:57.250639] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:57.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:57.724551] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:01:57.724577] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:01:57.271067] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:57.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:57.724596] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:01:57.724590] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant: 500, limit: 9223372036854775807 hold: 691453952 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 332902400 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 41041920 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 85983232 limit= 9223372036854775807 [MEMORY] ctx_id= LIBEASY hold_bytes= 3145728 limit= 9223372036854775807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 10485760 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 2752512 limit= 9223372036854775807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 37232640 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 78925824 limit= 9223372036854775807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 98983936 limit= 9223372036854775807 [2025-08-14 09:01:57.724608] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:01:57.724622] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:01:57.724635] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=18] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 332902400 used= 301271792 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 501 washed_blocks= 2233 washed_size= 927375360 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 301271792 used= 293865518 count= 12662 avg_used= 23208 mod=SUMMARY [2025-08-14 09:01:57.724640] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:01:57.724665] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 500 ctx_id= GLIBC hold= 43139072 used= 21963632 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 180 washed_blocks= 818 washed_size= 355319808 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 21963632 used= 10364997 count= 131700 avg_used= 78 mod=SUMMARY [2025-08-14 09:01:57.724674] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:288) [570][T1_ArcFetcher][T1][YB42AC4D0007-00063C495B999C16-0-0] [lt=26065] ObArchiveFetcher is running(thread_index=0) [2025-08-14 09:01:57.724683] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 500 ctx_id= CO_STACK hold= 85983232 used= 83091456 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 83091456 used= 82936896 count= 161 avg_used= 515136 block_cnt= 161 chunk_cnt= 41 mod=CoStack [MEMORY] hold= 83091456 used= 82936896 count= 161 avg_used= 515136 mod=SUMMARY [2025-08-14 09:01:57.724699] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 500 ctx_id= LIBEASY hold= 3145728 used= 3070816 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1048576 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3070816 used= 2859784 count= 223 avg_used= 12824 mod=SUMMARY [2025-08-14 09:01:57.724714] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 500 ctx_id= LOGGER_CTX_ID hold= 14680064 used= 4161536 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4161536 used= 4159488 count= 2 avg_used= 2079744 mod=SUMMARY [2025-08-14 09:01:57.724738] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= RPC_CTX_ID hold= 2752512 used= 1286144 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 50 washed_blocks= 63 washed_size= 102105088 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1286144 used= 849280 count= 53 avg_used= 16024 mod=SUMMARY [2025-08-14 09:01:57.724753] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= PKT_NIO hold= 37232640 used= 35792384 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 42 washed_blocks= 63 washed_size= 82591744 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 35792384 used= 34871392 count= 116 avg_used= 300615 mod=SUMMARY [2025-08-14 09:01:57.724778] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= SCHEMA_SERVICE hold= 78925824 used= 68904576 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 22 washed_size= 5767168 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 68904576 used= 67642943 count= 4856 avg_used= 13929 mod=SUMMARY [2025-08-14 09:01:57.724793] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= UNEXPECTED_IN_500 hold= 98983936 used= 97130832 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 97130832 used= 86748768 count= 3007 avg_used= 28848 mod=SUMMARY [2025-08-14 09:01:57.724809] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant: 508, limit: 1073741824 hold: 15613952 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 5128192 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10485760 limit= 9223372036854775807 [2025-08-14 09:01:57.725594] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=778] [MEMORY] tenant_id= 508 ctx_id= DEFAULT_CTX_ID hold= 5128192 used= 3765328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3765328 used= 3723912 count= 84 avg_used= 44332 mod=SUMMARY [2025-08-14 09:01:57.725620] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 508 ctx_id= CO_STACK hold= 10485760 used= 10321920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10321920 used= 10302720 count= 20 avg_used= 515136 mod=SUMMARY [2025-08-14 09:01:57.725665] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant: 509, limit: 1073741824 hold: 15613952 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 5128192 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10485760 limit= 9223372036854775807 [2025-08-14 09:01:57.725675] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 509 ctx_id= DEFAULT_CTX_ID hold= 5128192 used= 3765328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3765328 used= 3723912 count= 84 avg_used= 44332 mod=SUMMARY [2025-08-14 09:01:57.725701] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 509 ctx_id= CO_STACK hold= 10485760 used= 10321920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10321920 used= 10302720 count= 20 avg_used= 515136 mod=SUMMARY [2025-08-14 09:01:57.724652] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:01:57.728466] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=3796][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:01:57.728482] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:01:57.728495] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:01:57.728512] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:01:57.728521] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:00:52.998, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:22.998], idx=771, next_allow_print_ts="09:01:22.998") [2025-08-14 09:01:57.728535] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=08:59:57.942, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:27.942], idx=868, next_allow_print_ts="09:00:27.942") [2025-08-14 09:01:57.728700] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:01:57.725750] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant: 1001, limit: 1073741824 hold: 606650368 cache_hold: 44040192 cache_used: 44040192 cache_item_count: 21 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 302309376 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 54525952 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 3080192 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 483328 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 45514752 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 5521408 limit= 157286400 [MEMORY] ctx_id= GLIBC hold_bytes= 15499264 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 73424896 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35651584 limit= 214748360 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 3252224 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 12582912 limit= 1073741800 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 5570560 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3866624 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 1179648 limit= 9223372036854775807 [2025-08-14 09:01:57.729947] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4163] [MEMORY] tenant_id= 1001 ctx_id= DEFAULT_CTX_ID hold= 302309376 used= 269290960 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 81 washed_blocks= 406 washed_size= 113410048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 269290960 used= 263164134 count= 3802 avg_used= 69217 mod=SUMMARY [2025-08-14 09:01:57.729961] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= MEMSTORE_CTX_ID hold= 54525952 used= 54099968 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 54099968 used= 54075216 count= 26 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:01:57.729970] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1001 ctx_id= EXECUTE_CTX_ID hold= 3080192 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 8 washed_size= 7405568 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:57.729979] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:01:57.728711] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:01:57.732331] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=3606][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:01:57.733251] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:57.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:57.729988] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= PLAN_CACHE_CTX_ID hold= 45514752 used= 36449984 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 77 washed_blocks= 201 washed_size= 132743168 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 36449984 used= 32916258 count= 3312 avg_used= 9938 mod=SUMMARY [2025-08-14 09:01:57.733301] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=3302] [MEMORY] tenant_id= 1001 ctx_id= WORK_AREA hold= 5521408 used= 0 limit= 157286400 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 9 washed_blocks= 40 washed_size= 17547264 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:57.733314] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1001 ctx_id= GLIBC hold= 15499264 used= 10889728 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 25 washed_blocks= 164 washed_size= 47415296 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10889728 used= 6020141 count= 18313 avg_used= 328 mod=SUMMARY [2025-08-14 09:01:57.733325] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1001 ctx_id= CO_STACK hold= 73424896 used= 71221248 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 32 washed_blocks= 34 washed_size= 2072576 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 71221248 used= 71088768 count= 138 avg_used= 515136 block_cnt= 138 chunk_cnt= 36 mod=CoStack [MEMORY] hold= 71221248 used= 71088768 count= 138 avg_used= 515136 mod=SUMMARY [2025-08-14 09:01:57.733337] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= KVSTORE_CACHE_ID hold= 44040192 used= 44040192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 44040192 used= 0 count= 21 avg_used= 0 mod=SUMMARY [2025-08-14 09:01:57.733346] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 33906624 limit= 214748360 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33906624 used= 30629648 count= 468 avg_used= 65447 mod=SUMMARY [2025-08-14 09:01:57.733354] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= TX_CALLBACK_CTX_ID hold= 3252224 used= 2342592 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 3 washed_blocks= 8 washed_size= 3039232 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2342592 used= 2269696 count= 286 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:57.740322] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a4578, timer:0x7f9966a63f10, task:0x7f9922e05028, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr17RefreshConfigTaskE", scheduled_time:1755162055747671, pushed_time:0, delay:10000000}, running_task_token={this:0x7f991b7a4498, timer:0x7f9966a63f10, task:0x7f9922e05010, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr11TableGCTaskE", scheduled_time:1755162048800605, pushed_time:1755162048800872, delay:20000}, total_delay=61992648, threashold=1000000, tenant_id=1002) [2025-08-14 09:01:57.740374] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=39] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:56.759354] WDIAG [MVCC] repeat_refresh (ob_multi_version_garbage_collector.cpp:225) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1557][errcode=-4012] repeat refresh failed, we will retry immediately(ret=-4012, this={this:0x7f99200ff540, last_study_timestamp:1755161007611441, last_refresh_timestamp:1755161188828450, last_reclaim_timestamp:1755161664766510, last_sstable_overflow_timestamp:0, has_error_when_study:true, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1755161006924036004, v:0}, is_inited:true}, current_timestamp=1755161970782504) [2025-08-14 09:01:57.733362] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:57.740785] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7398] [MEMORY] tenant_id= 1001 ctx_id= RPC_CTX_ID hold= 12582912 used= 368640 limit= 1073741800 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 368640 used= 243840 count= 15 avg_used= 16256 mod=SUMMARY [2025-08-14 09:01:57.740804] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1001 ctx_id= TX_DATA_TABLE hold= 5570560 used= 2342784 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 8 washed_size= 2818048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2342784 used= 2319424 count= 30 avg_used= 77314 mod=SUMMARY [2025-08-14 09:01:57.740817] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= MDS_DATA_ID hold= 3866624 used= 2785280 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 5 washed_size= 4521984 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2785280 used= 2698240 count= 340 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:57.740833] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= MERGE_NORMAL_CTX_ID hold= 1179648 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 8 washed_blocks= 24 washed_size= 15597568 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:57.742475] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:57.725069] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=985105] current monitor number(seq_=-1) [2025-08-14 09:01:57.743722] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:57.743817] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:57.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:57.744254] INFO [MVCC] repeat_reclaim (ob_multi_version_garbage_collector.cpp:266) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=979651] skip repeat reclaim(ret=0, this={this:0x7f99200ff540, last_study_timestamp:1755161007611441, last_refresh_timestamp:1755161188828450, last_reclaim_timestamp:1755161664766510, last_sstable_overflow_timestamp:0, has_error_when_study:true, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1755161006924036004, v:0}, is_inited:true}, current_timestamp=1755162117740488, GARBAGE_COLLECT_EXEC_INTERVAL=600000000) [2025-08-14 09:01:57.740855] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 1002, limit: 8589934592 hold: 1943941120 cache_hold: 48234496 cache_used: 48234496 cache_item_count: 23 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 1007513600 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 415236096 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 1114112 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 950272 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 211812352 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 1441792 limit= 429496725 [MEMORY] ctx_id= GLIBC hold_bytes= 12582912 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 149913600 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 37748736 limit= 1717986900 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 7766016 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 5980160 limit= 8589934500 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 5718016 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3047424 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 1032192 limit= 9223372036854775807 [2025-08-14 09:01:57.720344] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5761] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:57.744483] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24101][errcode=0] epoll handler too much time :24151 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:57.744512] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=3618] [MEMORY] tenant_id= 1002 ctx_id= DEFAULT_CTX_ID hold= 1007513600 used= 916434032 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 13 [MEMORY] hold= 185860096 used= 185534336 count= 114 avg_used= 1627494 block_cnt= 114 chunk_cnt= 104 mod=MysqlRequesReco [MEMORY] hold= 167792640 used= 167772192 count= 1 avg_used= 167772192 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 83927040 used= 83886080 count= 2 avg_used= 41943040 block_cnt= 2 chunk_cnt= 2 mod=LogGroupBuffer [MEMORY] hold= 67190784 used= 67109344 count= 4 avg_used= 16777336 block_cnt= 4 chunk_cnt= 4 mod=MrgMemPoolChk [MEMORY] hold= 916434032 used= 899232713 count= 9437 avg_used= 95287 mod=SUMMARY [2025-08-14 09:01:57.744530] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1002 ctx_id= MEMSTORE_CTX_ID hold= 415236096 used= 411992064 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 411992064 used= 411803568 count= 198 avg_used= 2079816 block_cnt= 198 chunk_cnt= 198 mod=Memstore [MEMORY] hold= 411992064 used= 411803568 count= 198 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:01:57.744541] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= EXECUTE_CTX_ID hold= 1114112 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 7 washed_size= 7274496 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:57.746606] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:57.746933] WDIAG [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:832) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE2-0-0] [lt=37567][errcode=-4012] failed to read task(ret=-4012, ret="OB_TIMEOUT", tenant_id=1002, sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-08-14 09:01:57.729916] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, param_config_:{memory_limit:536870912, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=865568, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:01:57.318255] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=37] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:57.748249] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=429324] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:57.750506] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:57.744565] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=21] [MEMORY] tenant_id= 1002 ctx_id= TRANS_CTX_MGR_ID hold= 950272 used= 933888 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1146880 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 933888 used= 921984 count= 2 avg_used= 460992 mod=SUMMARY [2025-08-14 09:01:57.750925] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6339] [MEMORY] tenant_id= 1002 ctx_id= PLAN_CACHE_CTX_ID hold= 211812352 used= 201995008 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 176930880 used= 160228962 count= 13450 avg_used= 11912 block_cnt= 13450 chunk_cnt= 101 mod=SqlPhyPlan [MEMORY] hold= 201995008 used= 183048143 count= 17906 avg_used= 10222 mod=SUMMARY [2025-08-14 09:01:57.750938] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1002 ctx_id= WORK_AREA hold= 1441792 used= 0 limit= 429496725 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 14 washed_blocks= 54 washed_size= 27918336 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:57.746566] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:57.750947] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= GLIBC hold= 12582912 used= 11425280 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 11425280 used= 6322556 count= 19216 avg_used= 329 mod=SUMMARY [2025-08-14 09:01:57.750958] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= CO_STACK hold= 149913600 used= 147087360 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 68 washed_blocks= 70 washed_size= 3178496 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 147087360 used= 146813760 count= 285 avg_used= 515136 block_cnt= 285 chunk_cnt= 73 mod=CoStack [MEMORY] hold= 147087360 used= 146813760 count= 285 avg_used= 515136 mod=SUMMARY [2025-08-14 09:01:57.750977] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1002 ctx_id= KVSTORE_CACHE_ID hold= 48234496 used= 48234496 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 48234496 used= 0 count= 23 avg_used= 0 mod=SUMMARY [2025-08-14 09:01:57.750992] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1002 ctx_id= META_OBJ_CTX_ID hold= 37748736 used= 35741632 limit= 1717986900 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 35741632 used= 32457488 count= 496 avg_used= 65438 mod=SUMMARY [2025-08-14 09:01:57.751005] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1002 ctx_id= TX_CALLBACK_CTX_ID hold= 7766016 used= 6934400 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 17 washed_size= 6914048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 6934400 used= 6729728 count= 848 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:57.751020] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1002 ctx_id= LOB_CTX_ID hold= 147456 used= 16384 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 16384 used= 15872 count= 2 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:57.751037] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1002 ctx_id= RPC_CTX_ID hold= 5980160 used= 1933312 limit= 8589934500 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 6 washed_blocks= 6 washed_size= 10797056 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1933312 used= 1267200 count= 84 avg_used= 15085 mod=SUMMARY [2025-08-14 09:01:57.751054] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1002 ctx_id= TX_DATA_TABLE hold= 5718016 used= 4521856 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 9 washed_blocks= 16 washed_size= 13156352 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4521856 used= 4480128 count= 40 avg_used= 112003 mod=SUMMARY [2025-08-14 09:01:57.751070] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1002 ctx_id= MDS_DATA_ID hold= 3047424 used= 1998848 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 4 washed_size= 1146880 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1998848 used= 1936384 count= 244 avg_used= 7936 mod=SUMMARY [2025-08-14 09:01:57.751083] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1002 ctx_id= MDS_CTX_ID hold= 147456 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:57.751098] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1002 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:57.751120] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=19] [MEMORY] tenant_id= 1002 ctx_id= MERGE_NORMAL_CTX_ID hold= 1032192 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 12 washed_size= 13647872 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:01:57.724599] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=452845][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:57.271|207=453.53ms|210=0us|222=1us|total=480.31ms) [2025-08-14 09:01:57.724524] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=473166][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:57.250|207=473.88ms|210=0us|222=2us|total=500.93ms) [2025-08-14 09:01:57.751577] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=27011][errcode=-4389] timer task use too much time(end=1755162117751576, start=1755162117250638, delta=500938) [2025-08-14 09:01:57.751604] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] task scheduled too delay(scheduled_delay=268072, expected_run_ts_us_=1755162117483531, ObClockGenerator::getRealClock()=1755162117751603, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:57.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:57.751621] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:57.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:57.752432] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=26742][errcode=-4389] timer task use too much time(end=1755162117752430, start=1755162117271064, delta=481366) [2025-08-14 09:01:57.752457] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=0] task scheduled too delay(scheduled_delay=460107, expected_run_ts_us_=1755162117292349, ObClockGenerator::getRealClock()=1755162117752456, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:57.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:57.752483] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:57.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:57.752612] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:57.753805] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:57.754374] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=24] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:57.751646] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:57.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:57.754875] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=13] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:01:57.724434] INFO [STORAGE.TRANS] statistics (ob_location_adapter.cpp:70) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] location adapter statistics(renew_access=663, total_access=92009, error_count=0, renew_rate=7.205738686e-03) [2025-08-14 09:01:57.756085] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=31596] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:57.756108] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162117756080], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:57.756131] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162117756080], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:57.756178] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=41] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162117756080]}) [2025-08-14 09:01:57.756409] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:01:57.744283] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99319fe960, caller:ob_multi_version_garbage_collector.cpp:start:115, function_type:ZN9oceanbase19concurrency_control30ObMultiVersionGarbageCollector5startEvE6$_1159, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fec40, time_wheel:0x7f99535f7990, thread_pool:0x7f99535f7170, is_running:0, time_interval:60.00s, expected_run_time:09:02:26.435, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:57.759054] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=14744][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:01:57.759|total=149.82s) [2025-08-14 09:01:57.736012] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891970090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:57.724, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:57.759139] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23085][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7ef590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:57.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:57.759164] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7ef710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:57.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:57.736617] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=12] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:57.759285] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=22630] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3817) [2025-08-14 09:01:57.759307] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:01:57.759334] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=13] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:57.759352] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=18] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3818) [2025-08-14 09:01:57.759357] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:01:57.759535] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:57.759552] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=17] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3819) [2025-08-14 09:01:57.759556] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:01:57.725554] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=971696] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:57.726897] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:57.760008] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=33055] temporary file module use no blocks [2025-08-14 09:01:57.760020] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:57.760041] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=20] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:57.760059] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=17] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623019165667, flush_io_finished_ret_:0, flush_io_finished_round_:26288, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:57.745679] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=358722239, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=251630, succ_cluster_heartbeat_count=1478, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161006924036004, v:0}, local_cluster_delta=1110020001, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:01:57.765555] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [857]OB_MALLOC COST TOO MUCH TIME, cost_time=944660, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=944652], seq:[0] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=953786, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=953784], seq:[0] [369]OB_MALLOC COST TOO MUCH TIME, cost_time=948968, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=948961], seq:[0] [2025-08-14 09:01:57.765573] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] temporary file module use no blocks [2025-08-14 09:01:57.765603] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:57.745694] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=569089029, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=253034, succ_cluster_heartbeat_count=2185, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161000606550000, v:0}, local_cluster_delta=1116220995, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:01:57.765721] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:57.766311] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19915] get wrs ts(ls_id={id:1}, delta=570004667, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:57.766339] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=10, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:57.766917] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=570004667, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:57.766937] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:57.770026] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7281410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:57.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:57.770057] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ab110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:57.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:57.770074] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891969710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:57.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:57.770088] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d36b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ebb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:57.767, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:57.770102] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891971d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:57.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:57.763576] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17812] get wrs ts(ls_id={id:1}, delta=368001990, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:57.770763] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=24] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:01:57.746967] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:834) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE2-0-0] [lt=34] load can-execute balance task(ret=-4012, ret="OB_TIMEOUT", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-08-14 09:01:57.772512] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:57.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:57.780067] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b4a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:57.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:57.780095] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72739b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919a5e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:57.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:57.780108] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72779b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72eab10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:57.777, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:57.780119] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728bc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:57.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:57.783265] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:57.790146] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919902b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196c390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:57.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:57.790172] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ec610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:57.785, task_priority:1, with_handle_protected:0}) [660]OB_MALLOC COST TOO MUCH TIME, cost_time=950250, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=950242], seq:[0] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=940741, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=940730, ALLOC_CHUNK_END=5], seq:[0, 1] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=940418, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=940408, ALLOC_CHUNK_END=4], seq:[0, 1] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=921983, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=921972, ALLOC_CHUNK_END=5], seq:[0, 1] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=901835, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=901814, ALLOC_CHUNK_END=5], seq:[0, 1] [513]OB_MALLOC COST TOO MUCH TIME, cost_time=899948, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=899938, ALLOC_CHUNK_END=5], seq:[0, 1] [822]OB_MALLOC COST TOO MUCH TIME, cost_time=869260, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=869251, ALLOC_CHUNK_END=4], seq:[0, 1] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=801245, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=801235, ALLOC_CHUNK_END=5], seq:[0, 1] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=731494, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=731482, ALLOC_CHUNK_END=5], seq:[0, 1] [1031]OB_MALLOC COST TOO MUCH TIME, cost_time=466826, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=466817, ALLOC_CHUNK_END=4], seq:[0, 1] [2025-08-14 09:01:57.790181] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891973810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:57.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.028187] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=22] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:01:58.534655] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fe410, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe6f0, time_wheel:0x7f9922df9d40, thread_pool:0x7f9922df9520, is_running:1, time_interval:2.00s, expected_run_time:09:01:58.432, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:57.926244] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] EASY SLOW: thread: 7f995447d300, lastrun: 1755162117.771161 cost: 0.155080 loop:26492, slowcnt: 1 [2025-08-14 09:01:57.936946] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:57.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:58.720830] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=782693][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:57.936|207=783.88ms|210=0us|222=2us|total=783.91ms) [2025-08-14 09:01:57.800189] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7277730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ea990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:57.794, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.720856] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=-4389] timer task use too much time(end=1755162118720855, start=1755162117936944, delta=783911) [2025-08-14 09:01:58.232740] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:01:57.833317] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:57.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:58.720894] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=886234][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:57.833|207=887.58ms|210=0us|222=1us|total=887.59ms) [2025-08-14 09:01:58.720905] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task use too much time(end=1755162118720905, start=1755162117833315, delta=887590) [2025-08-14 09:01:57.848313] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=9] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:58.720952] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=871305] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:57.993971] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:57.989, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:58.688032] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:1, time_interval:3.00s, expected_run_time:09:01:58.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:58.721395] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:57.856032] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:01:57.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:58.722397] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=865050][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:57.856|207=866.36ms|210=0us|222=2us|total=866.39ms) [2025-08-14 09:01:58.720869] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=919285][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:57.800|207=920.68ms|210=0us|222=1us|total=920.70ms) [2025-08-14 09:01:58.724773] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=3897][errcode=-4389] timer task use too much time(end=1755162118724772, start=1755162117800187, delta=924585) [2025-08-14 09:01:58.724788] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=929772, expected_run_ts_us_=1755162117795014, ObClockGenerator::getRealClock()=1755162118724787, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891825c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:57.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.724829] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891825c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:57.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.724843] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=906836, expected_run_ts_us_=1755162117818007, ObClockGenerator::getRealClock()=1755162118724843, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7283e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:57.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.720465] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:58.722422] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-4389] timer task use too much time(end=1755162118722421, start=1755162117856030, delta=866391) [2025-08-14 09:01:58.720891] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=487409][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [567]OB_MALLOC COST TOO MUCH TIME, cost_time=860756, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1], seq:[0, 1] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=963288, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=953803], seq:[0, 1] [369]OB_MALLOC COST TOO MUCH TIME, cost_time=958474, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=948981], seq:[0, 1] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=959318, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=959312], seq:[0] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=959338, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=959332], seq:[0, 1] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=960047, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=950264], seq:[0, 1] [2025-08-14 09:01:58.720979] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=725906][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:57.993|207=727.01ms|210=0us|222=2us|total=727.03ms) [2025-08-14 09:01:58.728883] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9236-0-0] [lt=32][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162117866435) [2025-08-14 09:01:58.728924] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9236-0-0] [lt=39][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162117866435}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162117766376}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162117766376}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:01:58.729010] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [316][pnio1][T1][YB42AC4D0007-00063C49607B9236-0-0] [lt=11][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1, delta=2003732, last_succ_cluster_heartbeat_tstamp_=1755162116725277) [2025-08-14 09:01:58.729176] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=570004667, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:58.729197] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:58.729832] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28] get wrs ts(ls_id={id:1}, delta=570004667, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:58.729842] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:58.730389] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=570004667, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:58.730398] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:57.765682] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:58.730455] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=963313][errcode=0] epoll handler too much time :964778 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:57.765581] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:58.724853] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7283e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:57.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.727845] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=6854][errcode=-4389] timer task use too much time(end=1755162118727844, start=1755162117993970, delta=733874) [2025-08-14 09:01:58.730530] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=5663][errcode=0] task scheduled too delay(scheduled_delay=910437, expected_run_ts_us_=1755162117820091, ObClockGenerator::getRealClock()=1755162118730529, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7288390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:57.820, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.730551] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7288390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:57.820, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.730563] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=900504, expected_run_ts_us_=1755162117830058, ObClockGenerator::getRealClock()=1755162118730563, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7288210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:57.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:57.770645] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=24590][errcode=-4015] clock out of order(cur_ts=1755162117770643, cur_ts_=1755162117743183, delta=27460) [2025-08-14 09:01:57.771171] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:58.731979] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=959356][errcode=0] epoll handler too much time :960817 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:58.732176] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [571][T1_ArcSender][T1][YB42AC4D0007-00063C495BA99C16-0-0] [lt=25][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:01:58.732197] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=20][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:01:58.733175] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=24][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:01:57.782262] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=26] gc stale ls task succ [2025-08-14 09:01:57.765658] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:58.733717] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=966595][errcode=0] epoll handler too much time :968065 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:57.818835] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=21] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:58.733755] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=913526] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:01:58.733863] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:355) [834][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=11] finish check and clear obj lock(ls->get_ls_id()={id:1001}) [2025-08-14 09:01:58.733881] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [834][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99319fe030, caller:ob_table_lock_service.cpp:start:276, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE6$_1273, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe310, time_wheel:0x7f99535f3990, thread_pool:0x7f99535f3170, is_running:0, time_interval:10.00s, expected_run_time:09:02:06.426, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:58.733905] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [834][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:01:58.733|total=2.15s) [2025-08-14 09:01:58.722016] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:58.733938] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11904][errcode=0] epoll handler too much time :11932 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:57.770174] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6584] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:58.726640] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=5732][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:01:58.734264] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7608][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:01:58.734275] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:01:58.734289] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:01:58.734298] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:01:58.734313] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:01:58.734323] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:01:58.734336] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:01:58.734346] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:01:58.734354] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:00:52.998, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:22.998], idx=771, next_allow_print_ts="09:01:22.998") [2025-08-14 09:01:58.734522] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:01:58.734534] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:01:58.734544] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:01:58.730479] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=963438] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:58.734652] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=4165] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623019184616, flush_io_finished_ret_:0, flush_io_finished_round_:26298, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:3, normal_idle_loop_cnt_:3, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:57.773157] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:57.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:58.734712] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=960099][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:57.773|207=961.55ms|210=0us|222=2us|total=961.57ms) [2025-08-14 09:01:58.734725] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task use too much time(end=1755162118734725, start=1755162117773155, delta=961570) [2025-08-14 09:01:58.734734] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=942385, expected_run_ts_us_=1755162117792349, ObClockGenerator::getRealClock()=1755162118734734, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:57.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:58.734748] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:57.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:58.730545] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=2690][errcode=0] task scheduled too delay(scheduled_delay=725056, expected_run_ts_us_=1755162118005487, ObClockGenerator::getRealClock()=1755162118730544, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:58.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:58.736080] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=5524][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:58.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:58.736109] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] task scheduled too delay(scheduled_delay=646792, expected_run_ts_us_=1755162118089316, ObClockGenerator::getRealClock()=1755162118736109, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:58.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:58.736121] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:58.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:58.736133] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=531835, expected_run_ts_us_=1755162118204297, ObClockGenerator::getRealClock()=1755162118736133, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:58.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:58.736142] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:58.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:58.736158] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc190, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc470, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:10.00s, expected_run_time:09:01:58.720, task_priority:1, with_handle_protected:1}) [730]OB_MALLOC COST TOO MUCH TIME, cost_time=953449, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=940751, WASH_BLOCK_END=12696, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1, 2, 3, 4] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=953002, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=940429, WASH_BLOCK_END=12573], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=933721, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=921999, WASH_BLOCK_END=11722], seq:[0, 1, 2] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=913513, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=901855, WASH_BLOCK_END=11658], seq:[0, 1, 2] [513]OB_MALLOC COST TOO MUCH TIME, cost_time=911626, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=899955, WASH_BLOCK_END=11671], seq:[0, 1, 2] [822]OB_MALLOC COST TOO MUCH TIME, cost_time=880976, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=869268, WASH_BLOCK_END=11708], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=812987, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=801253, WASH_BLOCK_END=11734], seq:[0, 1, 2] [1031]OB_MALLOC COST TOO MUCH TIME, cost_time=478027, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=467036, WASH_BLOCK_END=10991], seq:[0, 1, 2] [2025-08-14 09:01:58.736685] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=962569] get wrs ts(ls_id={id:1001}, delta=373015359, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:58.736707] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:58.736784] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=368001990, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:58.736792] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:58.737379] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=373015359, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:58.737388] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:58.730572] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7288210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:57.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.737490] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=6901][errcode=0] task scheduled too delay(scheduled_delay=784846, expected_run_ts_us_=1755162117952643, ObClockGenerator::getRealClock()=1755162118737489, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196aa90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:57.952, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.737515] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196aa90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:57.952, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.737529] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=784489, expected_run_ts_us_=1755162117953039, ObClockGenerator::getRealClock()=1755162118737528, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7288e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:57.953, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.737541] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7288e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:57.953, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.737554] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=767038, expected_run_ts_us_=1755162117970515, ObClockGenerator::getRealClock()=1755162118737553, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891992fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196de90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:57.970, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.737565] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891992fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196de90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:57.970, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:57.782777] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=24] gc stale ls task succ [2025-08-14 09:01:57.783258] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:58.737751] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=953050][errcode=0] epoll handler too much time :954502 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:58.737790] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [965][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F30-0-0] [lt=857412][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1002, svr="172.77.0.7:2882", version={val:1755161758221798000, v:0}, valid_part_count=0, total_part_count=2, generate_timestamp=1755162118736731) [2025-08-14 09:01:58.737813] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [965][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F30-0-0] [lt=18][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1002, req={req_server:"172.77.0.7:2882", version:{val:1755161758221798000, v:0}, valid_part_count:0, total_part_count:2, generate_timestamp:1755162118736731}, twrs={inited:true, tenant_id:1002, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162118736772}, server_version_for_stat:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162118736772}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:01:57.802720] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=25] PNIO [ratelimit] time: 1755162117802718, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007617, add_bytes: 0 [2025-08-14 09:01:57.823011] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:58.737950] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=913553][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:57.824932] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=40][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a6258, timer:0x7f9966fddee0, task:0x7f994f1b14e0, task_type:"N9oceanbase5table20ObTableObjectPoolMgr26ObTableSessEliminationTaskE", scheduled_time:1755161695725325, pushed_time:0, delay:5000000}, running_task_token={this:0x7f994e1a65d8, timer:0x7f9966fddee0, task:0x7f994f1bbcc0, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161694824965, pushed_time:1755161925772608, delay:5000000}, total_delay=422099605, threashold=1000000, tenant_id=1) [2025-08-14 09:01:58.737998] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=911685][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a64f8, timer:0x7f9966fddee0, task:0x7f994f1b1510, task_type:"N9oceanbase5table20ObTableObjectPoolMgr28ObTableReqResEliminationTaskE", scheduled_time:1755161695734026, pushed_time:0, delay:5000000}, running_task_token={this:0x7f994e1a65d8, timer:0x7f9966fddee0, task:0x7f994f1bbcc0, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161694824965, pushed_time:1755161925772608, delay:5000000}, total_delay=423003972, threashold=1000000, tenant_id=1) [2025-08-14 09:01:58.738013] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a6418, timer:0x7f9966fddee0, task:0x7f994f1b14f8, task_type:"N9oceanbase5table20ObTableObjectPoolMgr27ObTableSessSysVarUpdateTaskE", scheduled_time:1755161695734641, pushed_time:0, delay:5000000}, running_task_token={this:0x7f994e1a65d8, timer:0x7f9966fddee0, task:0x7f994f1bbcc0, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161694824965, pushed_time:1755161925772608, delay:5000000}, total_delay=423003372, threashold=1000000, tenant_id=1) [2025-08-14 09:01:57.855718] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] get_fetch_log_dst failed or invalid(ret=0, palf_id=1, self="172.77.0.7:2882", fetch_log_dst="172.77.0.7:2882") [2025-08-14 09:01:57.923864] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1737][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:57.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:58.738138] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=813041][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:57.923|207=814.27ms|210=0us|222=1us|total=814.29ms) [2025-08-14 09:01:58.738159] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4389] timer task use too much time(end=1755162118738159, start=1755162117923862, delta=814297) [2025-08-14 09:01:57.756429] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=19] get number(ret=-4038, service_type_=0, range=1, base_id=1755162117743183000, start_id=0, end_id=140295968341544) [2025-08-14 09:01:58.738221] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=980297][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:01:58.738243] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162117756403], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:58.738259] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162117756403]}) [2025-08-14 09:01:58.738358] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [317][pnio1][T1002][YB42AC4D0007-00063C49612B8F30-0-0] [lt=11][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1002, delta=3584967, last_succ_cluster_heartbeat_tstamp_=1755162115153390) [2025-08-14 09:01:58.259629] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:58.738414] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=478060] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3820) [2025-08-14 09:01:58.738453] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1}, delta=368001990, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:01:58.731999] WDIAG [BALANCE] do_work (ob_balance_task_execute_service.cpp:153) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE2-0-0] [lt=24655][errcode=-4012] failed to load all balance task(ret=-4012, ret="OB_TIMEOUT", tenant_id_=1002) [2025-08-14 09:01:58.738542] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [882][T1002_ArcSender][T1002][YB42AC4D0007-00063C495EC99C16-0-0] [lt=19] ObArchiveSender is running(thread_index=6) [2025-08-14 09:01:57.840417] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a5ed8, timer:0x7f99200ff180, task:0x7f99200ff1d8, task_type:"N9oceanbase8observer18ObTableLoadService8ObGCTaskE", scheduled_time:1755162056715823, pushed_time:0, delay:30000000}, running_task_token={this:0x7f991b7a5fb8, timer:0x7f99200ff180, task:0x7f99200ff1f0, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162050755138, pushed_time:1755162084760013, delay:1000000}, total_delay=61124593, threashold=1000000, tenant_id=1002) [2025-08-14 09:01:58.436805] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fa190, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa470, time_wheel:0x7f99341cdbf0, thread_pool:0x7f99341cd3d0, is_running:1, time_interval:2.00s, expected_run_time:09:01:58.334, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:58.741564] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=304295][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:58.436|207=304.76ms|210=0us|222=2us|total=304.77ms) [2025-08-14 09:01:58.741582] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] timer task use too much time(end=1755162118741582, start=1755162118436802, delta=304780) [2025-08-14 09:01:58.741642] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:58.742518] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:58.747657] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b9690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:58.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.747692] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891968e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:58.737, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.747709] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e71b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891973210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:58.739, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.747724] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918119b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a1090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:58.737, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.747934] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4375] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:58.748621] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:58.749540] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:58.749796] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:58.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:58.750364] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [474]OB_MALLOC COST TOO MUCH TIME, cost_time=755147, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=731612, WASH_BLOCK_END=23535], seq:[0, 1, 2] [2025-08-14 09:01:57.732470] EDIAG [SHARE] refresh (ob_task_define.cpp:396) [280][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=1033482][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1016859, owner: ObLog, click_count: 7, time dist:[FORMAT_END=992680, ObMalloc#start=1, ObMalloc#end=24149, ObMalloc#start=23, ObMalloc#end=2, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1fd6cd17 0x96c37e0 0x1fda2168 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:58.741617] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=0] get_fetch_log_dst failed or invalid(ret=0, palf_id=1, self="172.77.0.7:2882", fetch_log_dst="172.77.0.7:2882") [2025-08-14 09:01:58.747910] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:58.744804] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:58.738423] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:01:58.750584] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=12123] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:58.750601] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:227) [1031][PxTargetMgr0][T1001][Y0-0000000000000000-0-0] [lt=14] refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3821) [2025-08-14 09:01:58.750611] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:01:58.750790] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:58.750800] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3822) [2025-08-14 09:01:58.750809] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:01:57.732470] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [280][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=1033482][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:12, tid:472}]) [2025-08-14 09:01:57.994537] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=3003][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:57.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:58.750913] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=755230][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:57.994|207=756.38ms|210=0us|222=2us|total=756.40ms) [2025-08-14 09:01:58.750937] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4389] timer task use too much time(end=1755162118750936, start=1755162117994535, delta=756401) [2025-08-14 09:01:58.738464] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:58.694980] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:01:58.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:58.750952] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=506318, expected_run_ts_us_=1755162118244633, ObClockGenerator::getRealClock()=1755162118750951, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:58.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:58.750976] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:58.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:58.750995] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8030, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8310, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:10.00s, expected_run_time:09:01:58.730, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:58.752779] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:58.753529] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12483] get wrs ts(ls_id={id:1001}, delta=374024699, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:01:58.753543] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:58.753595] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:58.753723] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=207] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:58.749194] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:58.738268] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162117756403]}) [2025-08-14 09:01:58.756139] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17841][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1002}) [2025-08-14 09:01:58.756193] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=54] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1002}) [2025-08-14 09:01:58.756271] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:01:58.756338] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=24] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:58.756355] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] temporary file module use no blocks [2025-08-14 09:01:58.756364] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:58.756378] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:58.756391] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623020196022, flush_io_finished_ret_:0, flush_io_finished_round_:26263, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:58.757725] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72774b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ea810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:58.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.757756] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fbe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7280c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:58.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.738482] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2826) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17990] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:411, [delay/us]: prepare:0, schedule:0, submit:600000000, device:10623, total:10631) [2025-08-14 09:01:58.760554] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22021] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:01:58.738497] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:160) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE2-0-0] [lt=6488] [BALANCE_EXECUTE] finish one round(ret=-4012, ret="OB_TIMEOUT", task_array_=[], idle_time_us=100000, task_comment_=) [2025-08-14 09:01:58.762277] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:58.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:58.762622] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:57.751027] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [982][T1002_L0_G0][T1002][YB42AC4D0007-00063C4962CA3040-0-0] [lt=998862][errcode=0] long time to hold one guard(schema_mgr=0x7f98a2767660, tenant_id=1002, version=1755156894100160, cur_timestamp=1755162024718693, ref_timestamp=1755161741867930, lbt()="0x26a1806d 0x974f3e5 0x974e159 0x974e084 0x948fab3 0x97257c0 0x97256f4 0x9725691 0x20b8d66c 0x20c1ea16 0x1f11562c 0x9703542 0x9700852 0x970011d 0x15f6f4c7 0x15ebaffc 0x15eba874 0x9343ecf 0x9341ca3 0x92e8803 0x92e30c4 0x92df309 0x92db9ba 0x92d9f5e 0x9828a7d 0x9298239 0x9285f13 0x927efad 0x926c154 0x12c5a0c9 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3") [2025-08-14 09:01:57.769105] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [784][T1002_TBalance][T1002][YB42AC4D0007-00063C4964C9A63F-0-0] [lt=25][errcode=0] long time to hold one guard(schema_mgr=0x7f98ad7818f0, tenant_id=1, version=1753073979411448, cur_timestamp=1755162117746776, ref_timestamp=1755161925784325, lbt()="0x26a1806d 0x974f3e5 0x974e159 0x974e084 0x948f9df 0x97257c0 0x97256f4 0x9725691 0x2012b6d2 0x2012ab70 0x104ea510 0x10539f82 0x10531684 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3") [2025-08-14 09:01:58.766272] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=31] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:58.766294] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=18] temporary file module use no blocks [2025-08-14 09:01:58.766303] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:58.766314] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:58.766404] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=90] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623020199858, flush_io_finished_ret_:0, flush_io_finished_round_:26289, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:58.767786] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891968990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:58.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.767822] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891973090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:58.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.767839] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7280b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:58.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.768411] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=34016] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:58.769082] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:58.735854] WDIAG [SERVER] set_timeout (ob_inner_sql_connection.cpp:2049) [742][T1001_DBMSSched][T1001][Y0-0000000000000000-0-0] [lt=937164][errcode=-4012] already timeout(ret=-4012, ctx={abs_timeout_us:1755162113770158}, abs_timeout_us=1755162113770158) [2025-08-14 09:01:58.771291] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=15] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:00:25.720034] EDIAG [STORAGE] ~ObITabletFilterOp (ob_tablet_pointer.cpp:751) [512][T1_TimerWK3_SSTableDefragme][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 92914543, owner: ObLog, click_count: 7, time dist:[FORMAT_END=38544, ObMalloc#start=1, ObMalloc#end=956416, ObMalloc#start=91919576, ObMalloc#end=2, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0x1b427bea 0x1d2cd08a 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:58.773427] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=20] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:01:58.773681] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=23] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:01:58.773698] INFO [PALF] check_parent_health (log_config_mgr.cpp:2875) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=8869] re_register_parent reason(palf_id=1, self="172.77.0.7:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1755162113747625, last_first_register_time_us=1755158728414267, register_time_us=1755162113747625, curr_time_us=1755162118750504) [2025-08-14 09:01:58.774151] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=21] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:00:25.720034] INFO [STORAGE] ~ObITabletFilterOp (ob_tablet_pointer.cpp:751) [512][T1_TimerWK3_SSTableDefragme][T1][Y0-0000000000000000-0-0] [lt=11] tablet skip filter destructed(total_cnt=0, skip_cnt=0, not_in_mem_cnt=0, invalid_attr_cnt=0, total_tablet_cnt=136172, total_skip_cnt=367, not_in_mem_tablet_cnt=0, invalid_attr_tablet_cnt=0) [2025-08-14 09:01:58.777866] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919aef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b9090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:58.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.777918] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7270a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919a4190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:58.762, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.777938] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891971710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:58.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.777957] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a0f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:58.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.777972] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7ef890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:58.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.782316] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=949558] gc stale ls task succ [2025-08-14 09:01:58.782842] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:58.782874] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=953477] gc stale ls task succ [2025-08-14 09:01:58.783266] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:58.787928] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f734de30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891919810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:58.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.787960] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891972f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:58.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.787973] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196b090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:58.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.787986] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d59b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ed090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:58.773, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.787998] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7efa10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:58.777, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.797994] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72eba10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:58.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.798027] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f0190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:58.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.808062] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728a790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:58.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.808093] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7281110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:58.790, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.810336] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=933765] PNIO [ratelimit] time: 1755162118810335, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007617, add_bytes: 0 [2025-08-14 09:01:58.820995] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:58.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:58.826653] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=4190][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:01:58.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:58.829958] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] get_max_applied_scn(scn={val:1755161547738516001, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:50, palf_committed_end_lsn_:{lsn:5282737655}, palf_committed_end_scn_:{val:1755161547738516001, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161547738516001, v:0}}) [2025-08-14 09:01:58.829986] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] get_min_unreplayed_log_info(lsn={lsn:5282737655}, scn={val:1755100947825033001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161624848121, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:13063726, ret_code_:0}}, next_to_submit_lsn_:{lsn:5282737655}, next_to_submit_scn_:{val:1755100947825033001, v:0}, base_lsn_:{lsn:4831543296}, base_scn_:{val:1755063445733337001, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7605000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:5282737655}, end_lsn:{lsn:5282737655}, read_buf:{buf_len_:3747840, buf_:0x7f98f7605000}, block_size:67104768, log_storage_:0x7f993e529bf0, log_storage_:{log_tail:{lsn:5282737655}, readable_log_tail:{lsn:5282737655}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:5234171904}, min_scn:{val:1755152244802535004, v:0}, curr_block_id:78, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:206, min_block_id:42, max_block_id:79, curr_writable_block_id:78}, logical_block_size_:67104768, curr_block_writable_size_:18539017, block_header_serialize_buf_:0x7f993e52a8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:85, read_io_size:301367296, read_disk_cost_ts:2364604, start_lsn:{lsn:4831543296}}}}}}) [2025-08-14 09:01:58.830644] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=76] get wrs ts(ls_id={id:1}, delta=571092089, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:58.830661] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:58.838249] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f1690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:58.827, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.838361] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=881022] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:01:58.838387] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=19] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:01:58.848321] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739af30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196bc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:58.837, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.858381] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739e510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:58.841, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.858415] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f0310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:58.849, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.858430] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7280f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:58.849, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.868383] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c0b90, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571c0e70, time_wheel:0x7f9959fef990, thread_pool:0x7f9959fef170, is_running:1, time_interval:3.00s, expected_run_time:09:01:58.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:58.908660] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728a610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:58.892, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.920983] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:58.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:58.921040] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:58.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:58.928786] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7ee090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:58.917, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.938307] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:58.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:58.938846] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7281a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:58.922, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:58.994208] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:58.989, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.014288] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=56][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:59.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.021056] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:58.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.021055] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:58.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.029092] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=12] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:01:59.038375] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:58.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.049332] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc4e38, timer:0x7f9970f40f80, task:0x564d6d07c140, task_type:"N9oceanbase7storage25ObRedefTableHeartBeatTaskE", scheduled_time:1755161637759743, pushed_time:0, delay:30000000}, running_task_token={this:0x7f9970fc5fb8, timer:0x7f9970f40f80, task:0x564d727182f0, task_type:"N9oceanbase5share16ObAshRefreshTaskE", scheduled_time:1755161611717613, pushed_time:1755161614757742, delay:120000000}, total_delay=481289588, threashold=1000000, tenant_id=500) [2025-08-14 09:01:59.094732] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:59.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.121117] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:59.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.121141] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:59.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.138438] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:59.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.215392] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:59.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.221166] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:59.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.221230] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:59.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.238500] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:59.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.245571] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:59.239, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.250894] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:59.250922] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=27] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3823) [2025-08-14 09:01:59.250930] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:01:59.250943] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:59.250959] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=16] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3824) [2025-08-14 09:01:59.250964] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:01:59.251126] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:59.251142] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=16] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3825) [2025-08-14 09:01:59.251150] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:01:59.256500] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=11] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:59.256526] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162119256496], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:59.256546] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162119256496], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:59.256573] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=23] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162119256496]}) [2025-08-14 09:01:59.256584] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162119256496]}) [2025-08-14 09:01:59.256597] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:01:59.256610] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162119256596], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:59.256627] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162119256596]}) [2025-08-14 09:01:59.265695] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:59.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.321196] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:59.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.321252] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:59.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.338566] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:59.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.421262] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=55][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:59.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.421315] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:59.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.438629] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:59.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.499151] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:59.489, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.519254] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=51][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:59.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.521356] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:59.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.521396] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:59.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.538694] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:59.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.599722] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:59.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.621431] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:59.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.621459] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:59.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.638718] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:59.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.695467] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=55907][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:01:59.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.710469] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:59.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.721495] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:59.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.721523] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:59.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.721593] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:58.776610] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:58.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:58.838524] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get_max_applied_scn(scn={val:1755161749741193000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:50, palf_committed_end_lsn_:{lsn:1604090090}, palf_committed_end_scn_:{val:1755161749741193000, v:0}, last_check_scn_:{val:1755161749741193001, v:0}, max_applied_cb_scn_:{val:1755161749741193000, v:0}}) [651]OB_MALLOC COST TOO MUCH TIME, cost_time=871635, tenant_id=500, label=CommonArray, ctx_id=0, prio=0, size=15872, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [2025-08-14 09:01:58.998057] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:58.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.728697] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts statistics(tenant_id=1001, gts_rpc_cnt=0, get_gts_cache_cnt=15, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:01:59.728923] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:01:59.728939] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:01:59.728954] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162119728916], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:59.728977] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162119728916], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:01:59.728992] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162119728916]}) [2025-08-14 09:01:59.729004] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts statistics(tenant_id=1002, gts_rpc_cnt=0, get_gts_cache_cnt=44, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:01:58.776098] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1055) [742][T1001_DBMSSched][T1001][Y0-0000000000000000-0-0] [lt=34451][errcode=-4012] set timeout failed(ret=-4012) [567]OB_MALLOC COST TOO MUCH TIME, cost_time=805567, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:01:59.733178] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] tx gc loop thread is running(MTL_ID()=1) [2025-08-14 09:01:59.733198] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] try gc retain ctx [2025-08-14 09:01:59.733207] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] try to retry start_working [2025-08-14 09:01:59.733227] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=9] tx gc loop thread is running(MTL_ID()=1001) [2025-08-14 09:01:59.733247] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=19] try gc retain ctx [2025-08-14 09:01:59.733254] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=7] try to retry start_working [2025-08-14 09:01:59.733898] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b9390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:59.728, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:59.733926] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f734c030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891807390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:59.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:59.736316] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:59.736336] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=16] temporary file module use no blocks [2025-08-14 09:01:59.736346] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:59.736360] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:59.737422] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] get wrs ts(ls_id={id:1}, delta=571995278, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:59.737442] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:59.737474] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161101770923, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755161572801635) [2025-08-14 09:01:59.738039] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=571995278, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:01:59.738052] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:59.739825] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EE9D-0-0] [lt=869266] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-08-14 09:01:59.742227] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:59.742534] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:59.743813] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=18314] current monitor number(seq_=-1) [2025-08-14 09:01:59.744912] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:59.745990] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5737] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:59.736374] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623021174448, flush_io_finished_ret_:0, flush_io_finished_round_:26299, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:59.743962] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919aae10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:59.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:59.721096] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6076] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:59.746037] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=2067][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891971b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:59.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:59.746066] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e67b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891972c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:59.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:59.746076] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739e390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:59.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:59.740663] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:59.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.746140] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=906750][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:59.746149] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:59.746475] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2600] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:59.748380] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:59.749394] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:59.750413] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:59.753310] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=12] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:59.753592] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:59.754089] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891968690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:59.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:59.754126] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891968510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:59.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:59.754138] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891972a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:59.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:59.754157] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ed390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:59.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:59.756586] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:01:59.756664] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:01:59.756689] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=18] temporary file module use no blocks [2025-08-14 09:01:59.756702] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:01:59.756720] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=17] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:01:59.756733] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623021196184, flush_io_finished_ret_:0, flush_io_finished_round_:26264, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:59.760786] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=5438][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:59.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.721556] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:58.776117] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:01:59.765249] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=987630] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:01:59.020974] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=793371][errcode=0] EASY SLOW: thread: 7f99542fb300, lastrun: 1755162115.752503 cost: 3.268469 loop:26416, slowcnt: 16 [2025-08-14 09:01:59.728675] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=729511][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:58.998|207=730.62ms|210=1us|222=1us|total=767.97ms) [2025-08-14 09:01:59.766029] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=37296][errcode=-4389] timer task use too much time(end=1755162119766027, start=1755162118998054, delta=767973) [2025-08-14 09:01:59.766049] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=521414, expected_run_ts_us_=1755162119244633, ObClockGenerator::getRealClock()=1755162119766048, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:59.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.766068] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:59.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.766090] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:59.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.766610] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7280090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:59.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:59.766661] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=7] get_max_applied_scn(scn={val:1755161719775179002, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:45, palf_committed_end_lsn_:{lsn:8461098908}, palf_committed_end_scn_:{val:1755161719775179002, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161719775179002, v:0}}) [2025-08-14 09:01:59.766691] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=26] get_min_unreplayed_log_info(lsn={lsn:8461098908}, scn={val:1755100947600130001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161920778323, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:23084275, ret_code_:0}}, next_to_submit_lsn_:{lsn:8461098908}, next_to_submit_scn_:{val:1755100947600130001, v:0}, base_lsn_:{lsn:7515734016}, base_scn_:{val:1753840867171901045, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6e05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:8461098908}, end_lsn:{lsn:8461098908}, read_buf:{buf_len_:3747840, buf_:0x7f98f6e05000}, block_size:67104768, log_storage_:0x7f99297a3bf0, log_storage_:{log_tail:{lsn:8461098908}, readable_log_tail:{lsn:8461098908}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8455200768}, min_scn:{val:1755160384851871002, v:0}, curr_block_id:126, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:215, min_block_id:93, max_block_id:127, curr_writable_block_id:126}, logical_block_size_:67104768, curr_block_writable_size_:61206628, block_header_serialize_buf_:0x7f99297a48b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:198, read_io_size:699822080, read_disk_cost_ts:4178035, start_lsn:{lsn:7515734016}}}}}}) [2025-08-14 09:01:59.751243] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:59.767752] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=16482] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3826) [2025-08-14 09:01:59.767763] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:01:59.767786] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:59.767795] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3827) [2025-08-14 09:01:59.767803] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:01:59.767992] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:01:59.768003] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3828) [2025-08-14 09:01:59.768010] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:01:59.769181] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=17] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:01:59.770118] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:59.764276] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:59.771895] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=22] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:01:59.782382] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=29] gc stale ls task succ [2025-08-14 09:01:59.782894] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=16] gc stale ls task succ [2025-08-14 09:01:59.783555] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:59.786738] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72707b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919a4010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:59.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:59.787714] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [556][T1_OBJLockGC][T1][Y0-0000000000000000-0-0] [lt=68][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fafd0, caller:ob_table_lock_service.cpp:start:276, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE6$_1273, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fb2b0, time_wheel:0x7f994b19db50, thread_pool:0x7f994b19d330, is_running:1, time_interval:10.00s, expected_run_time:09:01:59.696, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.796796] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b4d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:59.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:59.806856] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739acb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196bb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:59.798, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:59.816916] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891973690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:59.806, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:59.816948] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891973510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:59.809, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:59.817971] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=19] PNIO [ratelimit] time: 1755162119817969, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007634, add_bytes: 0 [2025-08-14 09:01:59.821588] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:59.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.826945] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196b510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:59.815, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:59.827364] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd070, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fd350, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:5.00s, expected_run_time:09:01:59.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.827402] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:01:59.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.840367] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:01:59.840407] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=0] get_fetch_log_dst failed or invalid(ret=0, palf_id=1001, self="172.77.0.7:2882", fetch_log_dst="172.77.0.7:2882") [2025-08-14 09:01:59.840420] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:01:59.846215] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=6] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:01:59.846266] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=50][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:59.847985] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891973390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:59.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:59.848028] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ed210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:59.832, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:59.858037] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72eb410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:59.841, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:59.868098] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:01:59.859, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:59.921628] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:59.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.921626] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:59.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.946281] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:01:59.946309] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:59.992124] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:59.989, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.997389] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:59.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.012244] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:00.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.021687] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:59.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.021702] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:01:59.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.029937] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=21] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:02:00.046352] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:00.092767] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:00.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.121720] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:00.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.121776] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:00.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.146416] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:00.213448] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:00.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.221784] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:00.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.221841] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:00.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.229201] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=11] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:00.229246] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=41][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162120229195], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:00.240181] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:00.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.243614] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:00.239, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.248797] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:00.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.258871] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:00.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.263736] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:00.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.268099] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:00.268130] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=30] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3829) [2025-08-14 09:02:00.268138] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:00.321871] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:00.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.340246] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=58][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:00.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.421930] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:00.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.440310] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:00.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.490189] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:00.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.495041] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:00.489, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.515138] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:00.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.521975] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:00.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.535826] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fe410, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe6f0, time_wheel:0x7f9922df9d40, thread_pool:0x7f9922df9520, is_running:1, time_interval:2.00s, expected_run_time:09:02:00.432, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.540374] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:00.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.595604] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:00.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.622036] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:00.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.640435] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:00.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.696026] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:00.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.716339] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:00.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:58.841498] EDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=899690][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 1876205, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=6, ALLOC_BLOCK#end=3, FORMAT_END=1876191, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x9757ced 0x97569df 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:59.744365] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4794] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:00.719067] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=973218][errcode=0] epoll handler too much time :974708 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:01:59.783216] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:00.719128] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=934498][errcode=0] epoll handler too much time :935921 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:00.229270] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162120229195], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:00.719214] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=489198] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162120229195]}) [2025-08-14 09:02:00.719248] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=32] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=44, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:02:00.719263] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:00.719283] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=20] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:00.719302] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=8] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:00.719312] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162120719298], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:00.719325] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162120719298], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:00.719337] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162120719298]}) [2025-08-14 09:02:00.719346] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162120719298]}) [2025-08-14 09:02:00.719516] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:00.719531] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:00.719547] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162120719512], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:00.719559] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162120719512]}) [2025-08-14 09:02:00.321856] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:00.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.719758] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=397297][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:00.321|207=397.90ms|210=0us|222=2us|total=397.92ms) [2025-08-14 09:02:00.719776] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] timer task use too much time(end=1755162120719775, start=1755162120321853, delta=397922) [2025-08-14 09:02:00.246485] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:00.719834] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=472633][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:01:58.841498] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=899690][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a7678, timer:0x7f994b1dfb70, task:0x7f9920110450, task_type:"N9oceanbase10rootserver21ObMLogMaintenanceTaskE", scheduled_time:1755161754754755, pushed_time:0, delay:0}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=364086742, threashold=1000000, tenant_id=1002) [2025-08-14 09:02:00.720588] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=1876205][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a7758, timer:0x7f994b1dfb70, task:0x7f9920110530, task_type:"N9oceanbase10rootserver22ObMViewMaintenanceTaskE", scheduled_time:1755161754754764, pushed_time:0, delay:0}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=365965814, threashold=1000000, tenant_id=1002) [2025-08-14 09:01:59.765064] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [512][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=92914543][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f994e1a4738, timer:0x7f9956fe7e10, task:0x7f9969bfd0a0, task_type:"N9oceanbase12blocksstable21ObSharedMacroBlockMgr26ObBlockDefragmentationTaskE", scheduled_time:1755162021723428, pushed_time:1755162021725261, delay:30000000}, delay_in_thread_pool=1033393, threashold=500000) [2025-08-14 09:02:00.720604] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a7bb8, timer:0x7f994b1dfb70, task:0x7f9920110628, task_type:"N9oceanbase10rootserver34ObMViewRefreshStatsMaintenanceTaskE", scheduled_time:1755161754754765, pushed_time:0, delay:0}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=365965839, threashold=1000000, tenant_id=1002) [780]OB_MALLOC COST TOO MUCH TIME, cost_time=973485, tenant_id=500, label=HeapVar, ctx_id=0, prio=0, size=24600, [2025-08-14 09:01:59.728514] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=888627] get wrs ts(ls_id={id:1}, delta=369986319, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:00.721345] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=991327] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:00.721386] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:00.040593] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=775603][errcode=0] EASY SLOW: thread: 7f995407d300, lastrun: 1755162119.753400 cost: 0.287191 loop:26492, slowcnt: 2 [2025-08-14 09:02:00.442857] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fa190, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa470, time_wheel:0x7f99341cdbf0, thread_pool:0x7f99341cd3d0, is_running:1, time_interval:2.00s, expected_run_time:09:02:00.334, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.722009] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=278727][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:00.442|207=279.15ms|210=1us|222=1us|total=279.18ms) [2025-08-14 09:02:00.722043] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=-4389] timer task use too much time(end=1755162120722042, start=1755162120442854, delta=279188) [2025-08-14 09:02:00.722105] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=50][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:00.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.722872] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73516b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196f510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:00.716, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:59.234679] EDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] LOGGER COST TOO MUCH TIME, cost: 1491568, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=6, ALLOC_BLOCK#end=2, FORMAT_END=1491553, ObMalloc#start=0, ObMalloc#end=3, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x13b06ffb 0x958a39e 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:59.727287] EDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=949237][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1000693, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=9110, FORMAT_END=991572, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ed35c 0x97aead0 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:59.234679] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:00.728539] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=1491568][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:02:00.728563] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:00.728576] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:00.728589] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:02:00.728607] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:00.728619] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:00.728638] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:00.728653] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:00.728669] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:00.728682] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:00.728695] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:00:52.998, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:22.998], idx=771, next_allow_print_ts="09:01:22.998") [2025-08-14 09:02:00.728863] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:00.728878] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:00.728895] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:02:00.268153] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:00.729174] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=460322] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3830) [2025-08-14 09:02:00.729184] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:00.729398] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:00.729413] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:227) [1031][PxTargetMgr0][T1002][Y0-0000000000000000-0-0] [lt=12] refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3831) [2025-08-14 09:02:00.729422] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:01:59.727287] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=949237][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:01:58.776|207=950.67ms|210=0us|222=2us|total=1.95s) [2025-08-14 09:02:00.729519] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1000693][errcode=-4389] timer task use too much time(end=1755162120729519, start=1755162118776609, delta=1952910) [2025-08-14 09:02:00.729532] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1937182, expected_run_ts_us_=1755162118792349, ObClockGenerator::getRealClock()=1755162120729531, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:58.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.729551] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:01:58.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.729565] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=1926571, expected_run_ts_us_=1755162118802994, ObClockGenerator::getRealClock()=1755162120729565, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40b7f0, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40bad0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:10.00s, expected_run_time:09:01:58.802, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.729579] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40b7f0, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40bad0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:10.00s, expected_run_time:09:01:58.802, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.729607] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=969576, expected_run_ts_us_=1755162119760031, ObClockGenerator::getRealClock()=1755162120729607, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:59.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.729621] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:01:59.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:59.746036] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24901][errcode=0] epoll handler too much time :24951 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:00.729720] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=982186] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:00.729740] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SERVER] net_in=0kB/s, net_out=0kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=1280000kB/s [2025-08-14 09:01:59.731449] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1748) [742][T1001_DBMSSched][T1001][Y0-0000000000000000-0-0] [lt=953906][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1753069619, tenant_id=1) [2025-08-14 09:01:59.766538] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:00.730688] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=962687] temporary file module use no blocks [2025-08-14 09:02:00.730700] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:00.730715] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:00.730771] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=55] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623021206201, flush_io_finished_ret_:0, flush_io_finished_round_:26290, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:59.753414] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:00.730866] WDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=975975][errcode=0] epoll handler too much time :977464 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:00.740039] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=52][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:00.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.740597] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=56][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b5990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:00.720, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:00.740645] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b5810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:00.727, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:00.740660] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ed690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:00.727, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:00.740672] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72851b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7289410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:00.721, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:00.742358] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:00.730142] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [742][T1001_DBMSSched][T1001][Y0-0000000000000000-0-0] [lt=997181][errcode=-4012] query failed(ret=-4012, conn=0x7f98eb025780, start=1755162118728423, sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1002)) [2025-08-14 09:02:00.743760] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [742][T1001_DBMSSched][T1001][Y0-0000000000000000-0-0] [lt=13595][errcode=-4012] read failed(ret=-4012) [2025-08-14 09:02:00.745342] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:00.746257] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:00.746288] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=21] current monitor number(seq_=-1) [2025-08-14 09:02:00.746342] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:00.747579] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:00:25.720483] EDIAG [OCCAM] operator() (ob_occam_timer.h:85) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 94903911, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=0, FORMAT_END=1030439, ObMalloc#start=1, ObMalloc#end=34095, ALLOC_END=93839372, APPEND_END=1], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ffa3b 0x9835318 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:00.720583] INFO pn_print_diag_info (group.c:775) [322][pnio2][T0][Y0-0000000000000000-0-0] [lt=1024980] PNIO client_send:0/0, client_queue_time=0, cnt=0, server_send:0/0, server_queue_time=0, cnt=0 [2025-08-14 09:02:00.746365] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9635] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:00.748680] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=2304] temporary file module use no blocks [2025-08-14 09:02:00.748702] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=22] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:00.748717] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:00.748731] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623022185793, flush_io_finished_ret_:0, flush_io_finished_round_:26300, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:00.740685] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739e210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:00.737, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:00.749185] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8482][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7281d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:00.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:00.750160] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:00.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.750728] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:00.750848] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891825690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:00.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:00.751630] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=777][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891807210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:00.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:00.751656] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891969590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:00.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:00.751671] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739e090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:00.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:00.748465] INFO [STORAGE] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:123) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] wead read timestamp is too old(timestamp={val:1755161749741193000, v:0}, gts_scn={val:1755161762721798002, v:0}, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1755161749741193000, v:0}}) [2025-08-14 09:02:00.753363] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=3228] get_max_applied_scn(scn={val:1755161744727824001, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:49, palf_committed_end_lsn_:{lsn:7323686009}, palf_committed_end_scn_:{val:1755161747742105001, v:0}, last_check_scn_:{val:1755161749741193000, v:0}, max_applied_cb_scn_:{val:1755161744727824001, v:0}}) [2025-08-14 09:02:00.753388] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] get_min_unreplayed_log_info(lsn={lsn:7323685240}, scn={val:1755100946796590002, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873485, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:8946764, ret_code_:0}}, next_to_submit_lsn_:{lsn:7323685240}, next_to_submit_scn_:{val:1755100946796590002, v:0}, base_lsn_:{lsn:7180210176}, base_scn_:{val:1753840803166611000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:7323685240}, end_lsn:{lsn:7323685240}, read_buf:{buf_len_:3747840, buf_:0x7f98f7a05000}, block_size:67104768, log_storage_:0x7f9901131bf0, log_storage_:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:28, read_io_size:98508800, read_disk_cost_ts:645339, start_lsn:{lsn:7180210176}}}}}}) [2025-08-14 09:02:00.754028] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=79] get wrs ts(ls_id={id:1001}, delta=376023757, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:00.754040] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:00.756743] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=26] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:02:00.739402] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=31] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:00.757683] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18252][errcode=0] epoll handler too much time :18293 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:00.718990] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=72] finish update min start scn(max_decided_scn={val:1755161719775179002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1755161287778691002, v:0}, keep_alive_scn_:{val:1755161287778691002, v:0}, update_ts_:1755162120717958}) [2025-08-14 09:02:00.751403] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=573011830, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:00.759115] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7695] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:00.761252] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d67b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891969290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:00.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:00.748414] INFO [PALF] register_parent_ (log_config_mgr.cpp:2707) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=35] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.77.0.7:2882", reason=1, leader="172.77.0.7:2882", register_time_us=1755162119724899, last_submit_register_req_time_us=1755162119724899, register_parent_reason=1) [2025-08-14 09:02:00.762291] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5891] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:00.759141] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162120750805}, version_delta=573020625) [2025-08-14 09:02:00.763943] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4787] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161101770923, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755161572801635) [2025-08-14 09:02:00.764563] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=573025085, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:00.764690] INFO [PALF] check_parent_health (log_config_mgr.cpp:2884) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=13204] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.77.0.7:2882") [2025-08-14 09:02:00.764577] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:00.750703] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:00.765510] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14782][errcode=0] epoll handler too much time :14817 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:00.765544] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12260][errcode=0] epoll handler too much time :12290 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:00.746714] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:00.765577] WDIAG ev_invoke_pending (ev.c:2136) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18834][errcode=0] epoll handler too much time :18873 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:00.766290] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=43535] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:00.721145] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] finish update min start scn(max_decided_scn={val:1755161547738516001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1755161493768447002, v:0}, keep_alive_scn_:{val:1755161493768447002, v:0}, update_ts_:1755162120720968}) [2025-08-14 09:02:00.740134] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:00.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.767333] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=27154][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:00.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:00:25.720483] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:00:26.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.767988] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=94903911][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:02:00.767|total=97.77s) [2025-08-14 09:02:00.768005] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] task executed too delay(delay_time=97744526, *p_task_={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:0, time_interval:250.00ms, expected_run_time:09:02:01.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.768159] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [272][TimerWK0_TimezoneMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] long time to hold one guard(schema_mgr=0x7f988fae9820, tenant_id=1001, version=1753074886385104, cur_timestamp=1755162027721673, ref_timestamp=1755161952726084, lbt()="0x26a1806d 0x974f3e5 0x974e159 0x974e084 0x948fab3 0x97257c0 0x97256f4 0x9725691 0x200f7258 0x12c537d7 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3") [2025-08-14 09:02:00.769385] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=48] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:00.770069] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:00.771996] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=23] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:00.782246] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:00.782451] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=32] gc stale ls task succ [2025-08-14 09:02:00.782957] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=23] gc stale ls task succ [2025-08-14 09:02:00.783438] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:00.791434] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891807090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:00.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:00.791473] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739aa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196b990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:00.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:00.801495] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ab710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:00.791, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:00.801532] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ecf10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:00.791, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:00.801544] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7280810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:00.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:00.801559] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7288810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:00.799, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:00.811557] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891972d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:00.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:00.811600] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7351e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196f990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:00.803, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:00.811614] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a22b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ed510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:00.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:00.811627] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7288690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:00.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:00.819859] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:00.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.819918] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:00.825564] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=21] PNIO [ratelimit] time: 1755162120825563, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007594, add_bytes: 0 [2025-08-14 09:02:00.825591] INFO pn_print_diag_info (group.c:775) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=25] PNIO client_send:0/0, client_queue_time=0, cnt=0, server_send:0/0, server_queue_time=0, cnt=0 [2025-08-14 09:02:00.830219] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:02:00.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.831670] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ebe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:00.826, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:00.831706] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7350530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196ea90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:00.824, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:00.846729] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:02:00.846771] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=27] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:02:00.872036] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fbbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7280e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:00.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:00.882087] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72860b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7288c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:00.873, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:00.919923] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:00.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.919990] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:00.946727] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:00.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.994599] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:00.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.018709] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:01.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.019975] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:00.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.020054] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:01.022307] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:00.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.030637] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=20] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:02:01.046790] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:00.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.067645] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=743894][errcode=0] EASY SLOW: thread: 7f99542fb300, lastrun: 1755162115.752503 cost: 5.315140 loop:26416, slowcnt: 29 [2025-08-14 09:02:01.070947] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb3a78, timer:0x7f9966f69f20, task:0x7f99333c4e90, task_type:"N9oceanbase7storage21ObTenantFreezeInfoMgr24UpdateLSResvSnapshotTaskE", scheduled_time:1755161699726703, pushed_time:0, delay:10000000}, running_task_token={this:0x7f9931bb3998, timer:0x7f9966f69f20, task:0x7f99333c4e78, task_type:"N9oceanbase7storage21ObTenantFreezeInfoMgr10ReloadTaskE", scheduled_time:1755161691777945, pushed_time:1755161691782442, delay:3000000}, total_delay=421344243, threashold=1000000, tenant_id=1001) [2025-08-14 09:02:01.099251] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:01.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.120012] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:01.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.120118] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:01.122371] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:01.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.146854] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:01.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.219790] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:01.219814] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162121219786], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:01.219837] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162121219786], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:01.220013] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:01.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.222434] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:01.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.229076] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:01.229117] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:02:01.229143] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:01.229159] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:01.229170] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:02:01.229192] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:01.229206] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:01.229246] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:01.229266] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:01.229287] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:01.229305] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:01.229477] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:01.229505] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:01.229516] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:02:01.229506] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:01.229537] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=30] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3832) [2025-08-14 09:02:01.229552] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:01.229568] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:01.229580] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=12] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3833) [2025-08-14 09:02:01.229585] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:01.229792] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=19] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:01.240111] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:01.239, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.246916] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:01.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.260224] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:01.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.322498] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:01.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.334834] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7faa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7281890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:01.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:01.334872] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72842b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7289b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:01.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:01.346981] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:01.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.422555] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:01.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.447045] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:01.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.491645] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=52][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:01.489, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.511733] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:01.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.522589] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:01.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.547110] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:01.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.592143] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:01.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.622650] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:01.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.626469] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7eef90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:01.614, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:01.626502] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eaf30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7ef110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:01.614, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:01.647172] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:01.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.689719] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=32926][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:1, time_interval:3.00s, expected_run_time:09:02:01.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.696635] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:01.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.712893] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:01.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:00.770523] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:00.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.716094] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=944138][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:00.770|207=945.57ms|210=0us|222=2us|total=945.59ms) [2025-08-14 09:02:01.716116] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task use too much time(end=1755162121716115, start=1755162120770521, delta=945594) [2025-08-14 09:02:01.716138] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=0] task scheduled too delay(scheduled_delay=923788, expected_run_ts_us_=1755162120792349, ObClockGenerator::getRealClock()=1755162121716137, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:00.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.716158] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:00.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.229809] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=16] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3834) [2025-08-14 09:02:01.717121] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=486574][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:01.219859] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162121219786]}) [2025-08-14 09:02:01.717155] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=496524] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:01.717167] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162121717152], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:01.717182] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162121717152], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:01.717196] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162121717152]}) [2025-08-14 09:02:01.717410] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:01.717423] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:01.717439] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162121717406], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:01.717452] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162121717406]}) [2025-08-14 09:02:01.246122] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:01.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.717686] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=470848][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:01.246|207=471.56ms|210=0us|222=2us|total=471.57ms) [2025-08-14 09:02:01.717701] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4389] timer task use too much time(end=1755162121717701, start=1755162121246120, delta=471581) [2025-08-14 09:02:01.717712] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=473078, expected_run_ts_us_=1755162121244633, ObClockGenerator::getRealClock()=1755162121717712, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:01.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.717727] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:01.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.220070] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:01.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.717873] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=497049][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:01.220|207=497.80ms|210=0us|222=2us|total=497.81ms) [2025-08-14 09:02:01.717886] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task use too much time(end=1755162121717886, start=1755162121220069, delta=497817) [2025-08-14 09:02:01.220154] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:01.717969] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=497060][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:00.757459] INFO [STORAGE.TRANS] amend (ob_tenant_weak_read_server_version_mgr.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [WRS] tenant has no valid partition on current server, generate weak read snapshot version based on current timestamp(tenant_id=1002, epoch_tstamp_=1755162118838516, total_part_count_=2, valid_inner_part_count_=0, valid_user_part_count_=0, version_={val:1755161758221798000, v:0}, base_version_when_no_valid_partition={val:1755161758221798000, v:0}) [2025-08-14 09:02:01.718672] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=959679] get_min_unreplayed_log_info(lsn={lsn:1604088821}, scn={val:1755100948296325001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873479, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:12492179, ret_code_:0}}, next_to_submit_lsn_:{lsn:1604088821}, next_to_submit_scn_:{val:1755100948296325001, v:0}, base_lsn_:{lsn:1409200128}, base_scn_:{val:1753840802540914000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:1604088821}, end_lsn:{lsn:1604088821}, read_buf:{buf_len_:3747840, buf_:0x7f98f6a05000}, block_size:67104768, log_storage_:0x7f990bfebbf0, log_storage_:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:42, read_io_size:147709952, read_disk_cost_ts:1061450, start_lsn:{lsn:1409200128}}}}}}) [2025-08-14 09:02:01.718753] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=76] get wrs ts(ls_id={id:1}, delta=371974705, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:01.718818] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=61] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:01.719629] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:01.722034] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:01.729888] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:01.729923] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:02:01.729944] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:01.729958] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:01.729971] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:02:01.729989] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:01.730001] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:01.730021] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:01.730034] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:01.730050] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:01.730064] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:01.730253] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:01.730269] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:01.730283] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:02:01.731448] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:01.732460] INFO pn_print_diag_info (group.c:755) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=13] PNIO client:0x7f995f4dd048_"172.77.0.7:49788"_"172.77.0.7:2882"_213_1755161597760925_1, write_queue=0/0, write=497/161573, read=484/110415, doing=0, done=504, write_time=22161148, read_time=28, process_time=20106962 [2025-08-14 09:02:01.732487] INFO pn_print_diag_info (group.c:768) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=24] PNIO server:0x7f995f4dc048_"172.77.0.7:49788"_261_1755161597800497, write_queue=0/0, write=484/110415, read=497/161573, doing=4, done=493, write_time=1097007, read_time=79, process_time=60262103 [2025-08-14 09:02:01.732498] INFO pn_print_diag_info (group.c:775) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=11] PNIO client_send:69965/27971104, client_queue_time=1246042868, cnt=1, server_send:69947/17166834, server_queue_time=63715453, cnt=1 [2025-08-14 09:02:01.737196] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:01.741311] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=26] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [567]OB_MALLOC COST TOO MUCH TIME, cost_time=876111, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:02:01.742061] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=892] get wrs ts(ls_id={id:1}, delta=573977382, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:01.742092] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:01.742574] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:01.744094] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:01.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.745280] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:01.746615] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:01.747044] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:01.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.747316] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:01.748963] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:01.749289] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] get wrs ts(ls_id={id:1001}, delta=376988074, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:01.749313] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:01.749780] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=573977382, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:01.749797] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:01.750692] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:01.750734] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:01.751038] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891968810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:01.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:01.751087] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891970810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:01.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:01.751102] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919827b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891970690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:01.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:01.752469] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9253] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:01.753494] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=8746] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:01.755101] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:01.756893] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:01.757086] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:01.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.761108] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739f890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:01.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:01.763506] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:01.765116] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:01.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.768763] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=14] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:01.768938] INFO pn_print_diag_info (group.c:755) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=8] PNIO client:0x7f995f54ca58_"172.77.0.7:42280"_"172.77.0.7:2882"_226_1755135622705153_1, write_queue=0/0, write=69922/28267302, read=69903/17089196, doing=0, done=69965, write_time=13267093, read_time=16826, process_time=68190586 [2025-08-14 09:02:01.768969] INFO pn_print_diag_info (group.c:768) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=29] PNIO server:0x7f995f54d4a8_"172.77.0.7:42280"_227_1755135622705612, write_queue=0/0, write=69903/17089196, read=69922/28267302, doing=1, done=69921, write_time=3078017, read_time=495566, process_time=115418272 [2025-08-14 09:02:01.769574] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:01.771184] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72742b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b4910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:01.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:01.771222] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919840b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891971590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:01.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:01.771260] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180ef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739f710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:01.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:01.771279] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180ecb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739f590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:01.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:01.771290] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7288b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:01.767, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:01.771305] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a0790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:01.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:01.772533] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=26] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:01.778480] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:01.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.781331] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919925b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196d890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:01.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:01.781363] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a1b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:01.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:01.781378] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7289a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:01.777, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:01.782520] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=36] gc stale ls task succ [2025-08-14 09:02:01.783796] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:01.784898] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=28] gc stale ls task succ [2025-08-14 09:02:01.784943] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:01.791408] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728ca30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7291590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:01.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:01.801468] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739ee10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:01.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:01.808676] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:01.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.811529] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ebd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:01.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:01.811566] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739ec90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:01.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:01.817972] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:01.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.821591] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72874b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7288090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:01.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:01.833218] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=8] PNIO [ratelimit] time: 1755162121833215, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007652, add_bytes: 0 [2025-08-14 09:02:01.833307] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7283090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:01.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:01.833339] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739dc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728a490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:01.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:01.872401] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c0f70, caller:mds_tenant_service.cpp:mtl_start:117, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_145, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571c1250, time_wheel:0x7f9959fef990, thread_pool:0x7f9959fef170, is_running:1, time_interval:15.00s, expected_run_time:09:02:01.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.872437] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c0b90, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571c0e70, time_wheel:0x7f9959fef990, thread_pool:0x7f9959fef170, is_running:1, time_interval:3.00s, expected_run_time:09:02:01.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.903830] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a59b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7282790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:01.895, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:01.994408] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:01.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:01.999168] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:01.989, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:02.029985] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:02.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:02.068448] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] EASY SLOW: thread: 7f995447d300, lastrun: 1755162121.769617 cost: 0.298830 loop:26496, slowcnt: 2 [2025-08-14 09:02:02.069159] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a4f18, timer:0x7f9966fddee0, task:0x7f994b3f6b30, task_type:"N9oceanbase3sql3dtl23ObDTLIntermResultGCTaskE", scheduled_time:1755161700757633, pushed_time:0, delay:10000000}, running_task_token={this:0x7f994e1a65d8, timer:0x7f9966fddee0, task:0x7f994f1bbcc0, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161694824965, pushed_time:1755161925772608, delay:5000000}, total_delay=421311525, threashold=1000000, tenant_id=1) [2025-08-14 09:02:02.151197] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:02.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:02.217291] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:02.217316] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=24] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3835) [2025-08-14 09:02:02.217327] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:02.217340] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:02.217347] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3836) [2025-08-14 09:02:02.217351] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:02.217552] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:02.217560] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3837) [2025-08-14 09:02:02.217565] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:02.245837] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:02.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:02.251278] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:02.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:02.255905] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:02.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:02.271314] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:02.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:02.301492] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:02.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:02.351344] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:02.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:02.423970] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fa190, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa470, time_wheel:0x7f99341cdbf0, thread_pool:0x7f99341cd3d0, is_running:1, time_interval:2.00s, expected_run_time:09:02:02.334, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:02.451410] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:02.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:02.522716] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:02.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:02.540734] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fe410, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe6f0, time_wheel:0x7f9922df9d40, thread_pool:0x7f9922df9520, is_running:1, time_interval:2.00s, expected_run_time:09:02:02.432, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:02.551473] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:02.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:02.652303] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:02.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:02.717651] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:02.717672] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=19] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3838) [2025-08-14 09:02:02.720404] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:02.722208] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:02.734897] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891971a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.726, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:02.734923] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739eb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.721, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:00.768020] EDIAG [ELECT] operator() (election_acceptor.cpp:137) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1964223, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1964222, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xfa03487 0x9835ae8 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [857]OB_MALLOC COST TOO MUCH TIME, cost_time=938206, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:02:01.756848] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1000811, owner: ObLog, click_count: 3, time dist:[FORMAT_END=965428, ALLOC_END=35383, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:01.749822] EDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=958505][errcode=-4015] LOGGER COST TOO MUCH TIME, cost: 1980006, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1980004, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x279e59be 0x971b1c1 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [317]OB_MALLOC COST TOO MUCH TIME, cost_time=948731, tenant_id=1002, label=[L]OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=3968, [2025-08-14 09:02:02.019251] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:02.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:02.741799] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=721452][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:02.192|207=722.55ms|210=0us|222=2us|total=722.57ms) [2025-08-14 09:02:02.741821] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4389] timer task use too much time(end=1755162122741820, start=1755162122019249, delta=722571) [2025-08-14 09:02:02.741836] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=652518, expected_run_ts_us_=1755162122089316, ObClockGenerator::getRealClock()=1755162122741835, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:02.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:02.741852] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:02.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:02.741874] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=0] task scheduled too delay(scheduled_delay=537576, expected_run_ts_us_=1755162122204297, ObClockGenerator::getRealClock()=1755162122741873, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:02.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:02.741886] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:02.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:02.741897] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=502752, expected_run_ts_us_=1755162122239145, ObClockGenerator::getRealClock()=1755162122741897, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:02.239, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:02.497447] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:02.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:02.743228] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=245406][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:02.497|207=245.78ms|210=0us|222=2us|total=245.83ms) [2025-08-14 09:02:02.743283] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=53][errcode=-4389] timer task use too much time(end=1755162122743282, start=1755162122497445, delta=245837) [2025-08-14 09:02:02.697187] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571634a0, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957163780, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:5.00s, expected_run_time:09:02:02.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:02.743311] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=46050][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:02.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:02.743695] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=35] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:02.739085] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:02.745132] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a0610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:02.745456] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:02.749677] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:01.769023] INFO pn_print_diag_info (group.c:775) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=53] PNIO client_send:69965/28279761, client_queue_time=853885982, cnt=1, server_send:69921/17093304, server_queue_time=59350430, cnt=1 [2025-08-14 09:02:01.769626] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:02.750663] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=979552][errcode=0] epoll handler too much time :981043 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:02.731250] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [14780][T1001_TimerWK4_SSTableGC][T1001][YB42AC4D0007-00063C4974099C4E-0-0] [lt=26][errcode=0] cost too much time:mds_table_base.cpp:int oceanbase::storage::mds::MdsTableBase::advance_state_to(oceanbase::storage::mds::MdsTableBase::State) const, (this=|threshold=1.00ms|start at 09:02:01.718|total=1.01s) [2025-08-14 09:02:01.772936] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [512][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=44886][errcode=-4389] timer task too much delay in priority queue(token={this:0x7f994e1a5c38, timer:0x7f994b3feee0, task:0x7f994b3fef20, task_type:"N9oceanbase8observer18ObTableLoadService15ObHeartBeatTaskE", scheduled_time:1755162023721549, pushed_time:1755162047763044, delay:10000000}, delay_in_priority_queue=24041495, threashold=1000000) [2025-08-14 09:02:02.756808] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [512][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=982384][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f994e1a5c38, timer:0x7f994b3feee0, task:0x7f994b3fef20, task_type:"N9oceanbase8observer18ObTableLoadService15ObHeartBeatTaskE", scheduled_time:1755162023721549, pushed_time:1755162047763044, delay:10000000}, delay_in_thread_pool=73004432, threashold=500000) [2025-08-14 09:02:02.757227] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:02.757276] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:02.757425] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:02.757454] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:02.035541] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=22] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:02:01.818038] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:02.758282] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=938818][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:02.758777] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=21] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:02.217691] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=16] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:02.758832] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=540318][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162122217685], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:02.758854] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162122217685]}) [2025-08-14 09:02:02.758863] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162122217685]}) [2025-08-14 09:02:02.758886] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:01.756848] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=13] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:02:01.756915] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=18] temporary file module use no blocks [2025-08-14 09:02:02.768536] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ab590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.750, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:02.768565] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891969e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:02.768577] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d80b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196a190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:02.768588] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196a010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:02.769798] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:02.770890] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:02.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:02.801055] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:02.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:02.840800] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=43] PNIO [ratelimit] time: 1755162122840798, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007583, add_bytes: 0 [2025-08-14 09:02:02.858342] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=9] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:02:02.858367] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:02.958384] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=8] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:02:02.958407] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:03.058155] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=721266] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:02:01.756915] EDIAG [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1001012, , BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x1d0f5b0a 0x1d0f57c6 0x1d17ad7c 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [395]OB_MALLOC COST TOO MUCH TIME, cost_time=991957, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=991955], seq:[0] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=991904, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=991902], seq:[0] [388]OB_MALLOC COST TOO MUCH TIME, cost_time=1006006, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=1005952], seq:[0, 1, 2, 3, 4, 5, 6] [387]OB_MALLOC COST TOO MUCH TIME, cost_time=1001164, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1001120, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [387]OB_MALLOC COST TOO MUCH TIME, cost_time=1001184, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1001174, WASH_BLOCK_END=10], seq:[0, 1, 2] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=994430, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=994365, ALLOC_CHUNK_END=26], seq:[0, 1, 2, 3, 4, 5, 6, 7] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=994476, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=994449, WASH_BLOCK_END=27], seq:[0, 1, 2] [2025-08-14 09:02:02.749707] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1002104, owner: ObLog, click_count: 6, time dist:[FORMAT_END=7, ObMalloc#start=1, ObMalloc#end=1001191, ObMalloc#start=26, ObMalloc#end=48, FREE_END=831], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [567]OB_MALLOC COST TOO MUCH TIME, cost_time=993766, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=993728, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=993786, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=993778, WASH_BLOCK_END=8], seq:[0, 1, 2] [656]OB_MALLOC COST TOO MUCH TIME, cost_time=992516, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=992496, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [656]OB_MALLOC COST TOO MUCH TIME, cost_time=992533, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=992525, WASH_BLOCK_END=8], seq:[0, 1, 2] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=983333, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=983309, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=983349, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=983341, WASH_BLOCK_END=8], seq:[0, 1, 2] [2025-08-14 09:02:01.831471] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=62][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1919600, owner: ObLog, click_count: 10, time dist:[FORMAT_END=926700, ObMalloc#start=1, ObMalloc#end=992540, ObMalloc#start=19, ObMalloc#end=46, ObMalloc#start=130, ObMalloc#end=54, FREE_END=110, ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=1], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:02.230465] EDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] LOGGER COST TOO MUCH TIME, cost: 1521355, owner: ObLog, click_count: 8, time dist:[FORMAT_END=526065, ObMalloc#start=1, ObMalloc#end=994486, ObMalloc#start=23, ObMalloc#end=60, ObMalloc#start=512, ObMalloc#end=54, FREE_END=154], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x13b06ffb 0x958a39e 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:02.741909] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1011037, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1001937, ALLOC_END=9099, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [736]OB_MALLOC COST TOO MUCH TIME, cost_time=984115, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=984092, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=984145, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=984131, WASH_BLOCK_END=14], seq:[0, 1, 2] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=984160, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=984141, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=984180, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=984171, WASH_BLOCK_END=9], seq:[0, 1, 2] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=981444, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=981422, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=981463, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=981454, WASH_BLOCK_END=9], seq:[0, 1, 2] [857]OB_MALLOC COST TOO MUCH TIME, cost_time=970416, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=970393, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [857]OB_MALLOC COST TOO MUCH TIME, cost_time=970434, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=970425, WASH_BLOCK_END=9], seq:[0, 1, 2] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=970504, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=970483, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=970521, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=970513, WASH_BLOCK_END=8], seq:[0, 1, 2] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=970560, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=970543, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=970582, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=970569, WASH_BLOCK_END=13], seq:[0, 1, 2] [2025-08-14 09:02:03.058443] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:02.758891] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162122758883], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:02.768599] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891992330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196d710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:03.731508] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=961448][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:02.768|207=962.90ms|210=1us|222=2us|total=962.93ms) [2025-08-14 09:02:03.731536] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4389] timer task use too much time(end=1755162123731535, start=1755162122768598, delta=962937) [2025-08-14 09:02:03.731547] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=978882, expected_run_ts_us_=1755162122752664, ObClockGenerator::getRealClock()=1755162123731547, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739f110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:03.731573] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739f110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:01.749822] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=958505][errcode=-4015] clock out of order(cur_ts=1755162121749820, cur_ts_=1755162121715898, delta=33922) [2025-08-14 09:02:02.957511] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:02.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:03.734029] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=775340][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:02.957|207=776.52ms|210=0us|222=2us|total=776.54ms) [2025-08-14 09:02:03.734052] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4389] timer task use too much time(end=1755162123734051, start=1755162122957509, delta=776542) [2025-08-14 09:02:03.022284] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:03.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:03.734078] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=710715][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:03.222|207=711.79ms|210=0us|222=2us|total=711.81ms) [2025-08-14 09:02:03.734091] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task use too much time(end=1755162123734090, start=1755162123022282, delta=711808) [2025-08-14 09:02:03.734103] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=5][errcode=0] task scheduled too delay(scheduled_delay=441753, expected_run_ts_us_=1755162123292349, ObClockGenerator::getRealClock()=1755162123734102, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:03.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:03.068987] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] EASY SLOW: thread: 7f99545f7300, lastrun: 1755162122.749700 cost: 0.319285 loop:26493, slowcnt: 3 [2025-08-14 09:02:03.697832] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:03.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:03.737811] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5636] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:03.739647] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6f78, timer:0x7f99329e1b10, task:0x7f98e4e091c0, task_type:"N9oceanbase5table21ObClearTTLHistoryTaskE", scheduled_time:1755162122736611, pushed_time:0, delay:30000000}, running_task_token={this:0x7f991b7a4c78, timer:0x7f99329e1b10, task:0x7f98e4e091f8, task_type:"N9oceanbase5table21ObTTLAllTaskSchedulerE", scheduled_time:1755162064731690, pushed_time:1755162092737016, delay:15000000}, total_delay=1003034, threashold=1000000, tenant_id=1002) [2025-08-14 09:02:03.741355] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:03.743491] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:03.743531] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=32] temporary file module use no blocks [2025-08-14 09:02:03.743539] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:03.750053] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=1001012] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:02.759542] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:02.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:03.750274] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=989232][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:02.759|207=990.73ms|210=0us|222=1us|total=990.75ms) [2025-08-14 09:02:03.750277] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=31] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:03.750290] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4389] timer task use too much time(end=1755162123750289, start=1755162122759540, delta=990749) [2025-08-14 09:02:03.732604] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [512][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f994e1a41f8, timer:0x7f996714bcc0, task:0x7f994b3fab60, task_type:"N9oceanbase3sql27ObSqlPlanMonitorRecycleTaskE", scheduled_time:1755162048762988, pushed_time:1755162048766422, delay:1000000}, delay_in_thread_pool=73990427, threashold=500000) [2025-08-14 09:02:03.719885] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:03.721075] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:03.734118] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:03.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:03.731586] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=970496, expected_run_ts_us_=1755162122761089, ObClockGenerator::getRealClock()=1755162123731586, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891971410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:03.752510] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20890][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891971410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:03.752533] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] task scheduled too delay(scheduled_delay=992511, expected_run_ts_us_=1755162122760021, ObClockGenerator::getRealClock()=1755162123752532, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f0f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:03.753189] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:03.753304] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=8] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:03.753551] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:03.754428] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:0, time_interval:250.00ms, expected_run_time:09:02:02.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:03.754443] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:02:03.754|total=2.99s) [2025-08-14 09:02:03.754458] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] task executed too delay(delay_time=100660460, *p_task_={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:0, time_interval:500.00ms, expected_run_time:09:02:03.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:02.741909] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:02.239, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:03.747026] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:03.750308] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:03.756056] WDIAG ev_invoke_pending (ev.c:2136) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10747][errcode=0] epoll handler too much time :10775 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:02.783810] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=26] gc stale ls task succ [2025-08-14 09:02:03.756106] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=970793][errcode=0] epoll handler too much time :972269 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:03.756223] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=970984] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:03.755134] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=316][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:03.756298] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=1157][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:02:03.756321] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:03.756333] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:03.756352] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:03.756366] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:03.756391] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:03.756404] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:01.779005] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:02:03.756583] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:03.756598] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:03.756611] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:02:03.759257] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16] LogLoopThread round_cost_time(us)(round_cost_time=8) [2025-08-14 09:02:03.759499] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=238] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:02:03.761054] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=41] get wrs ts(ls_id={id:1001}, delta=379031721, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:03.761086] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:03.761118] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=365539319, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=251635, succ_cluster_heartbeat_count=1479, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161006924036004, v:0}, local_cluster_delta=1116837081, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:02:02.764487] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:03.762380] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=996385][errcode=0] epoll handler too much time :997904 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:03.762405] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:03.730701] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=970335] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162122758883]}) [2025-08-14 09:02:03.762456] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=31704] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162122758883]}) [2025-08-14 09:02:03.762469] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1001}) [2025-08-14 09:02:03.762480] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1001}) [2025-08-14 09:02:03.762679] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:03.769752] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=13] LogLoopThread round_cost_time(us)(round_cost_time=6) [2025-08-14 09:02:03.749914] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:03.750311] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=766778, expected_run_ts_us_=1755162122983531, ObClockGenerator::getRealClock()=1755162123750310, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:02.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:03.770095] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=19753][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:02.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:03.750065] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:03.770833] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=20736] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623023196596, flush_io_finished_ret_:0, flush_io_finished_round_:26266, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:03.752128] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32193][errcode=0] epoll handler too much time :32254 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:03.752295] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=31172][errcode=0] epoll handler too much time :31234 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:03.752545] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f0f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:03.158504] EDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] LOGGER COST TOO MUCH TIME, cost: 1561224, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=10, ALLOC_BLOCK#end=91, FORMAT_END=610117, ALLOC_END=951005, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b52e837 0x996d04c 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:03.771463] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18883][errcode=0] task scheduled too delay(scheduled_delay=999593, expected_run_ts_us_=1755162122771868, ObClockGenerator::getRealClock()=1755162123771462, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b9c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:03.771490] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b9c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:03.753354] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1002104][errcode=0] epoll handler too much time :1003653 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:03.771571] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18178] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:01.747836] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [564][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=826] XA scheduler heartbeat task statistics(avg_time=4784) [2025-08-14 09:02:03.754500] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1011037][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:02.741|207=1.01s|210=1us|222=1us|total=1.01s) [2025-08-14 09:02:03.782197] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:03.158504] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:02.925582] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1793838, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=9, ALLOC_BLOCK#end=82, FORMAT_END=842721, ALLOC_END=951026, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [395]OB_MALLOC COST TOO MUCH TIME, cost_time=1961768, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=991972], seq:[0, 1] [2025-08-14 09:02:02.758639] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1961793, owner: ObLog, click_count: 5, time dist:[FORMAT_END=8, ObMalloc#start=1, ObMalloc#end=1961782, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [770]OB_MALLOC COST TOO MUCH TIME, cost_time=1961568, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=991913], seq:[0, 1] [2025-08-14 09:02:01.918034] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=50][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2801203, owner: ObLog, click_count: 5, time dist:[FORMAT_END=839626, ObMalloc#start=0, ObMalloc#end=1961576, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [388]OB_MALLOC COST TOO MUCH TIME, cost_time=1975723, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1006023], seq:[0, 1] [2025-08-14 09:02:02.744819] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1975743, owner: ObLog, click_count: 5, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=1975730, ALLOC_END=3, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [567]OB_MALLOC COST TOO MUCH TIME, cost_time=984969, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=2], seq:[0, 1] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=1021154, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1021147], seq:[0] [857]OB_MALLOC COST TOO MUCH TIME, cost_time=979508, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:02:02.925582] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:02.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:03.769979] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=20029] temporary file module use no blocks [2025-08-14 09:02:02.758639] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:04.723424] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1961793][errcode=0] epoll handler too much time :1964791 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:01.918034] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=50][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:01.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:04.723498] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=2801203][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:01.918|207=2.81s|210=0us|222=1us|total=2.81s) [2025-08-14 09:02:04.723514] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4389] timer task use too much time(end=1755162124723514, start=1755162121918032, delta=2805482) [2025-08-14 09:02:02.744819] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:04.723568] WDIAG ev_invoke_pending (ev.c:2136) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1975743][errcode=0] epoll handler too much time :1978770 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:04.723581] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:03.771502] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=995466, expected_run_ts_us_=1755162122776036, ObClockGenerator::getRealClock()=1755162123771502, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72751b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b5210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.723600] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=950658][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72751b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b5210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.723619] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:03.771|207=952.11ms|210=0us|222=2us|total=952.12ms) [2025-08-14 09:02:04.723628] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] timer task use too much time(end=1755162124723628, start=1755162123771501, delta=952127) [2025-08-14 09:02:04.723635] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] task scheduled too delay(scheduled_delay=1947498, expected_run_ts_us_=1755162122776136, ObClockGenerator::getRealClock()=1755162124723635, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d31b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72eb890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.723647] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d31b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72eb890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.723657] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1949707, expected_run_ts_us_=1755162122773949, ObClockGenerator::getRealClock()=1755162124723657, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73959b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f1090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.773, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.723666] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73959b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f1090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.773, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.694337] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:1, time_interval:3.00s, expected_run_time:09:02:04.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:03.772212] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=17683][errcode=-4389] timer task use too much time(end=1755162123772211, start=1755162122741897, delta=1030314) [2025-08-14 09:02:04.720350] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19238] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:03.762693] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=13] get number(ret=-4038, service_type_=0, range=1, base_id=1755162123759545000, start_id=0, end_id=140295968341544) [2025-08-14 09:02:04.722185] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=950766] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:04.722159] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1793838][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:02.925|207=1.80s|210=0us|222=2us|total=1.80s) [2025-08-14 09:02:04.723444] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:04.726595] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=962442][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:04.723675] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1936340, expected_run_ts_us_=1755162122787334, ObClockGenerator::getRealClock()=1755162124723675, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7283210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.740265] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16560][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7283210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.740294] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] task scheduled too delay(scheduled_delay=1952789, expected_run_ts_us_=1755162122787504, ObClockGenerator::getRealClock()=1755162124740293, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72856b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7289290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.740306] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72856b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7289290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.740424] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=994282] get_max_applied_scn(scn={val:1755161547738516001, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:50, palf_committed_end_lsn_:{lsn:5282737655}, palf_committed_end_scn_:{val:1755161547738516001, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161547738516001, v:0}}) [2025-08-14 09:02:04.724544] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=950883][errcode=0] task scheduled too delay(scheduled_delay=1969056, expected_run_ts_us_=1755162122755487, ObClockGenerator::getRealClock()=1755162124724543, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:02.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:04.741663] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:04.742176] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119812272, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-08-14 09:02:04.742205] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get_max_applied_scn(scn={val:1755161749741193000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:50, palf_committed_end_lsn_:{lsn:1604090090}, palf_committed_end_scn_:{val:1755161749741193000, v:0}, last_check_scn_:{val:1755161749741193001, v:0}, max_applied_cb_scn_:{val:1755161749741193000, v:0}}) [2025-08-14 09:02:01.779005] EDIAG [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1974464, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1974452, ObMalloc#start=2, ObMalloc#end=5, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1f9d7fcf 0x99b8be2 0x1f981833 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:01.735486] EDIAG [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3012870, owner: ObLog, click_count: 3, time dist:[FORMAT_END=3012868, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x99721a1 0x997a72e 0x2006ed17 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [663]OB_MALLOC COST TOO MUCH TIME, cost_time=2015295, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=23725, WASH_BLOCK_END=959577, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1021185], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:02.734614] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [663][T1001_StandbySc][T1001][YB42AC4D0007-00063C495C7A02F6-0-0] [lt=36][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2015322, owner: ObLog, click_count: 5, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=2015306, ALLOC_END=3, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 08:59:50.717276] EDIAG stat_cleanup (debug.c:35) [314][pnio1][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] LOGGER COST TOO MUCH TIME, cost: 133838073, owner: ObLog, click_count: 5, time dist:[FORMAT_END=6776, ObMalloc#start=1, ObMalloc#end=15586, ALLOC_END=133815710, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x27a8a131 0x97bb240 0x97baf6e 0x97b949c 0x97b893b 0x27a8a6d6 0x97b7ba0 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [474]OB_MALLOC COST TOO MUCH TIME, cost_time=768768, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=39, WASH_BLOCK_END=768729], seq:[0, 1, 2] [564]OB_MALLOC COST TOO MUCH TIME, cost_time=985721, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=985713], seq:[0] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=509139, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=23, WASH_BLOCK_END=509116], seq:[0, 1, 2] [656]OB_MALLOC COST TOO MUCH TIME, cost_time=911046, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=42, WASH_BLOCK_END=911003], seq:[0, 1, 2] [2025-08-14 09:02:04.742226] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=375000257, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:04.742266] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=36] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:04.742849] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1001}, delta=380013626, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:04.742867] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:04.742899] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=375000257, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:04.742907] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:04.743469] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7] get wrs ts(ls_id={id:1001}, delta=380015102, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:04.743480] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:04.743514] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:04.744274] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:04.726745] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=4578][errcode=-4389] timer task use too much time(end=1755162124726745, start=1755162122925580, delta=1801165) [2025-08-14 09:02:04.740315] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1952339, expected_run_ts_us_=1755162122787976, ObClockGenerator::getRealClock()=1755162124740315, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7efb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.746333] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=6006][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7efb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.746362] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] task scheduled too delay(scheduled_delay=1956200, expected_run_ts_us_=1755162122790161, ObClockGenerator::getRealClock()=1755162124746362, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919920b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196d590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.790, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.746376] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919920b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196d590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.790, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:03.833319] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=11][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:02:04.746567] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:04.740982] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=16396][errcode=0] task scheduled too delay(scheduled_delay=1651666, expected_run_ts_us_=1755162123089316, ObClockGenerator::getRealClock()=1755162124740982, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:03.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:04.745544] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=13] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:02:04.745666] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=17] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:02:04.745556] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=242] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:02:01.735486] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=37] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:03.777191] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=22] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:03.773284] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=29] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:04.753049] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [14780][T1001_TimerWK4_SSTableGC][T1001][YB42AC4D0007-00063C4974099C4E-0-0] [lt=19406][errcode=0] cost too much time:mds_row.ipp:int oceanbase::storage::mds::MdsRow::get_latest(READ_OP &&) const [K = oceanbase::storage::mds::DummyKey, V = oceanbase::storage::O..., (this=|threshold=5.00ms|start at 09:02:04.716|489=0us|total=36.93ms) [2025-08-14 09:02:04.753360] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5725] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:04.746388] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1906303, expected_run_ts_us_=1755162122840084, ObClockGenerator::getRealClock()=1755162124746387, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72eb710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.840, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.753404] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7003][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72eb710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.840, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.753430] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] task scheduled too delay(scheduled_delay=1907305, expected_run_ts_us_=1755162122846124, ObClockGenerator::getRealClock()=1755162124753429, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d54b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ecd90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.846, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.753457] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d54b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ecd90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.846, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.747157] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=6163][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:03.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:04.754353] WDIAG [COORDINATOR] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [471][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] cost too much time:ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_(), (this=|threshold=1.00s|start at 09:02:04.754|total=1.90s) [2025-08-14 09:02:04.755269] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=983529] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:04.756246] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=982123] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:04.753024] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=978260] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:04.753015] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=974351] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 08:59:50.717276] WDIAG stat_cleanup (debug.c:35) [314][pnio1][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] PNIO [delay_warn] cost too much time: 762006us, procedure: sk_read [2025-08-14 09:02:04.752942] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=3012870] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:04.753468] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1905415, expected_run_ts_us_=1755162122848053, ObClockGenerator::getRealClock()=1755162124753468, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ecc10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.848, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.758551] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=5072][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ecc10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:02.848, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.758591] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] task scheduled too delay(scheduled_delay=1036413, expected_run_ts_us_=1755162123722177, ObClockGenerator::getRealClock()=1755162124758591, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ea390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.722, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.758604] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ea390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.722, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.758616] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1027734, expected_run_ts_us_=1755162123730881, ObClockGenerator::getRealClock()=1755162124758616, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ab410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:03.991481] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:03.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:04.761484] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=768836][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:03.991|207=770.00ms|210=1us|222=1us|total=770.02ms) [2025-08-14 09:02:04.761503] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] timer task use too much time(end=1755162124761502, start=1755162123991479, delta=770023) [2025-08-14 09:02:04.761517] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=516883, expected_run_ts_us_=1755162124244633, ObClockGenerator::getRealClock()=1755162124761516, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:04.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:04.761531] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:04.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:04.763288] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:04.736274] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:04.763356] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27040][errcode=0] epoll handler too much time :27090 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:04.757739] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4716] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:04.763674] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5924] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:04.758626] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ab410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.764328] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=5688][errcode=0] task scheduled too delay(scheduled_delay=1032800, expected_run_ts_us_=1755162123731526, ObClockGenerator::getRealClock()=1755162124764327, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891826cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891918190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.764354] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891826cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891918190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.764368] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1017029, expected_run_ts_us_=1755162123747338, ObClockGenerator::getRealClock()=1755162124764368, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b5390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.757725] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=4694] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:04.764396] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6658] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:04.757841] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=4892] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:04.764421] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6569] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:04.754330] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=7156][errcode=0] task scheduled too delay(scheduled_delay=1550032, expected_run_ts_us_=1755162123204297, ObClockGenerator::getRealClock()=1755162124754329, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:03.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:04.764469] WDIAG [COORDINATOR] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [471][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] cost too much time:ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_failure(), (this=|threshold=1.00s|start at 09:02:04.764|total=1.91s) [2025-08-14 09:02:04.764486] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [471][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:03.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:04.764501] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [471][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:02:04.764|total=1.91s) [2025-08-14 09:02:04.764405] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:04.764427] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:04.764439] WDIAG eloop_delay_warn (debug.h:93) [314][pnio1][T0][Y0-0000000000000000-0-0] [lt=133838073][errcode=0] PNIO [delay_warn] eloop handle events delay high: 134801961, malloc=0/0 write=0/0 read=762006/1 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:04.764378] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b5390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.765015] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=632][errcode=0] task scheduled too delay(scheduled_delay=1018922, expected_run_ts_us_=1755162123746092, ObClockGenerator::getRealClock()=1755162124765014, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919845b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891971890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.765032] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919845b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891971890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.765044] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1010525, expected_run_ts_us_=1755162123754518, ObClockGenerator::getRealClock()=1755162124765043, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ea210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.764924] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=517] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:04.763683] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:04.764934] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=506] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:04.257275] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:04.767282] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=509229][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:02:04.765053] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ea210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.767521] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=2459][errcode=0] task scheduled too delay(scheduled_delay=1012797, expected_run_ts_us_=1755162123754723, ObClockGenerator::getRealClock()=1755162124767520, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ea090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.767539] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ea090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.767551] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1012148, expected_run_ts_us_=1755162123755403, ObClockGenerator::getRealClock()=1755162124767551, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196a910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.767562] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196a910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.768363] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [14780][T1001_TimerWK4_SSTableGC][T1001][YB42AC4D0007-00063C4974099C4E-0-0] [lt=17][errcode=0] cost too much time:mds_unit.ipp:int oceanbase::storage::mds::MdsUnit::get_latest(OP &&) const [K = oceanbase::storage::mds::DummyKey, V = oceanbase::storage::ObTa..., (this=|threshold=10.00ms|start at 09:02:03.749|890=1us|891=0us|total=1.02s) [2025-08-14 09:02:04.740457] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] get_min_unreplayed_log_info(lsn={lsn:5282737655}, scn={val:1755100947825033001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161624848121, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:13063726, ret_code_:0}}, next_to_submit_lsn_:{lsn:5282737655}, next_to_submit_scn_:{val:1755100947825033001, v:0}, base_lsn_:{lsn:4831543296}, base_scn_:{val:1755063445733337001, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7605000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:5282737655}, end_lsn:{lsn:5282737655}, read_buf:{buf_len_:3747840, buf_:0x7f98f7605000}, block_size:67104768, log_storage_:0x7f993e529bf0, log_storage_:{log_tail:{lsn:5282737655}, readable_log_tail:{lsn:5282737655}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:5234171904}, min_scn:{val:1755152244802535004, v:0}, curr_block_id:78, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:206, min_block_id:42, max_block_id:79, curr_writable_block_id:78}, logical_block_size_:67104768, curr_block_writable_size_:18539017, block_header_serialize_buf_:0x7f993e52a8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:85, read_io_size:301367296, read_disk_cost_ts:2364604, start_lsn:{lsn:4831543296}}}}}}) [2025-08-14 09:02:03.855001] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=488][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:02:03.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:04.767306] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:04.767572] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1008196, expected_run_ts_us_=1755162123759376, ObClockGenerator::getRealClock()=1755162124767572, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728a910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.769478] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1900][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728a910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.769497] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] task scheduled too delay(scheduled_delay=1011910, expected_run_ts_us_=1755162123757586, ObClockGenerator::getRealClock()=1755162124769497, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eacb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7ef290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.769508] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eacb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7ef290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.769518] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1001528, expected_run_ts_us_=1755162123767990, ObClockGenerator::getRealClock()=1755162124769518, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891968090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.767, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.783288] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:04.162464] EDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 1553221, owner: ObLog, click_count: 5, time dist:[FORMAT_END=583143, ObMalloc#start=1, ObMalloc#end=21163, ALLOC_END=948904, APPEND_END=9], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x9757ced 0x97569df 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [864]OB_MALLOC COST TOO MUCH TIME, cost_time=955216, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=955214], seq:[0] [864]OB_MALLOC COST TOO MUCH TIME, cost_time=1931357, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=53, WASH_BLOCK_END=976055, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=955243], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:03.783826] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=1000811][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1931392, owner: ObLog, click_count: 5, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=1931368, ALLOC_END=7, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:03.759058] EDIAG [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1956202, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=9, ALLOC_BLOCK#end=2, FORMAT_END=1956184, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1f832f91 0x97c36fe 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:03.730674] EDIAG [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [723][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=98][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1984634, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1984617, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=1, APPEND_END=12], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x973b640 0x973b132 0x973a32d 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [357]OB_MALLOC COST TOO MUCH TIME, cost_time=954977, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=3899], seq:[0, 1] [2025-08-14 09:02:04.162464] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc65d8, timer:0x7f9970f8c420, task:0x564d6e8e3210, task_type:"N9oceanbase3omt19ObTenantTimezoneMgr18DeleteTenantTZTaskE", scheduled_time:1755161702763867, pushed_time:0, delay:5000000}, running_task_token={this:0x7f9970fc66b8, timer:0x7f9970f8c420, task:0x564d6e8e3228, task_type:"N9oceanbase3omt19ObTenantTimezoneMgr18UpdateTenantTZTaskE", scheduled_time:1755161702763810, pushed_time:1755161702764275, delay:5000000}, total_delay=421398596, threashold=1000000, tenant_id=500) [2025-08-14 09:02:04.769821] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28682] get wrs ts(ls_id={id:1}, delta=577030462, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:05.718073] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=946811] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:03.783826] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=1000811] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:02:04.994094] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:04.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:04.881276] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c0b90, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571c0e70, time_wheel:0x7f9959fef990, thread_pool:0x7f9959fef170, is_running:1, time_interval:3.00s, expected_run_time:09:02:04.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:05.718167] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=722974][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:04.994|207=724.07ms|210=0us|222=2us|total=724.11ms) [2025-08-14 09:02:05.718167] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=835622][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:04.881|207=836.89ms|210=0us|222=2us|total=836.92ms) [2025-08-14 09:02:03.759058] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=14] current monitor number(seq_=-1) [2025-08-14 09:02:05.718207] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=39][errcode=-4389] timer task use too much time(end=1755162125718206, start=1755162124994091, delta=724115) [2025-08-14 09:02:05.718208] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=41][errcode=-4389] timer task use too much time(end=1755162125718208, start=1755162124881274, delta=836934) [2025-08-14 09:02:05.135993] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=664142][errcode=0] EASY SLOW: thread: 7f9954779300, lastrun: 1755162124.722431 cost: 0.413559 loop:26486, slowcnt: 4 [2025-08-14 09:02:05.718226] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=473591, expected_run_ts_us_=1755162125244633, ObClockGenerator::getRealClock()=1755162125718225, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:05.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:05.718270] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:05.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:04.753383] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:03.730674] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [723][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=98] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=16, total_destructed_detector=16, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f99333562b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-08-14 09:02:05.718290] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=963448][errcode=0] epoll handler too much time :964912 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:05.718305] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:213) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-08-14 09:02:05.718329] INFO [COMMON] handle (memory_dump.cpp:544) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=22] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-08-14 09:02:05.718332] WDIAG [DETECT] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [723][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=1984634][errcode=0] cost too much time:ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t), (this=|threshold=100.00ms|start at 09:02:05.718|total=3.95s) [2025-08-14 09:02:04.823591] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:04.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:05.718579] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=893632][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:04.823|207=894.99ms|210=0us|222=2us|total=895.00ms) [2025-08-14 09:02:05.718594] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] timer task use too much time(end=1755162125718594, start=1755162124823589, delta=895005) [2025-08-14 09:02:04.756341] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=985082] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:04.769281] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=912895][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:03.855|207=914.28ms|210=0us|222=2us|total=1.86s) [2025-08-14 09:02:05.718752] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=948036][errcode=-4389] timer task use too much time(end=1755162125718751, start=1755162123854999, delta=1863752) [2025-08-14 09:02:05.718764] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=924852, expected_run_ts_us_=1755162124793911, ObClockGenerator::getRealClock()=1755162125718763, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd070, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fd350, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:5.00s, expected_run_time:09:02:04.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:05.718780] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd070, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fd350, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:5.00s, expected_run_time:09:02:04.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:04.734312] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10850][errcode=0] epoll handler too much time :10872 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:04.726594] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=4399] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:05.718793] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=924856, expected_run_ts_us_=1755162124793936, ObClockGenerator::getRealClock()=1755162125718792, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:02:04.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:05.718805] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:02:04.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:05.718817] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=983004] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [660]OB_MALLOC COST TOO MUCH TIME, cost_time=1948375, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=59, WASH_BLOCK_END=1948314, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:03.773139] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=18163][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1948417, owner: ObLog, click_count: 5, time dist:[FORMAT_END=24, ObMalloc#start=1, ObMalloc#end=1948391, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:01.747836] EDIAG [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [564][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=826][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2020719, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2002502, ALLOC_END=18217, APPEND_END=0, ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=991868], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b64d8ba 0x1b64d154 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:04.722444] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19095][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1004493, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1004492, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:03.731021] EDIAG [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [851][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1996286, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1996284, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x973b640 0x973b132 0x973a32d 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [384]OB_MALLOC COST TOO MUCH TIME, cost_time=1946802, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=70, WASH_BLOCK_END=1946731], seq:[0, 1, 2] [683]OB_MALLOC COST TOO MUCH TIME, cost_time=1299798, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=44, WASH_BLOCK_END=1299753], seq:[0, 1, 2] [2025-08-14 09:02:04.429288] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1299861, owner: ObLog, click_count: 5, time dist:[FORMAT_END=33, ObMalloc#start=1, ObMalloc#end=1299806, ALLOC_END=20, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [472]OB_MALLOC COST TOO MUCH TIME, cost_time=1894111, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=39, WASH_BLOCK_END=1894072], seq:[0, 1, 2] [797]OB_MALLOC COST TOO MUCH TIME, cost_time=1169832, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=35, WASH_BLOCK_END=1169797], seq:[0, 1, 2] [400]OB_MALLOC COST TOO MUCH TIME, cost_time=1602665, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=49, WASH_BLOCK_END=1602616], seq:[0, 1, 2] [950]OB_MALLOC COST TOO MUCH TIME, cost_time=975562, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=49, WASH_BLOCK_END=975513], seq:[0, 1, 2] [571]OB_MALLOC COST TOO MUCH TIME, cost_time=983476, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2007, WASH_BLOCK_END=981469], seq:[0, 1, 2] [733]OB_MALLOC COST TOO MUCH TIME, cost_time=983485, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2070, WASH_BLOCK_END=981415], seq:[0, 1, 2] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=983442, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2041, WASH_BLOCK_END=981401], seq:[0, 1, 2] [2025-08-14 09:02:02.783879] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=40][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2943142, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1959671, ObMalloc#start=0, ObMalloc#end=983450, ALLOC_END=20, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [730]OB_MALLOC COST TOO MUCH TIME, cost_time=983498, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2031, WASH_BLOCK_END=981467], seq:[0, 1, 2] [2025-08-14 09:02:02.784936] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=26][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2942176, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1958650, ObMalloc#start=1, ObMalloc#end=983506, ALLOC_END=19, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [477]OB_MALLOC COST TOO MUCH TIME, cost_time=983521, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2026, WASH_BLOCK_END=981495], seq:[0, 1, 2] [317]OB_MALLOC COST TOO MUCH TIME, cost_time=980272, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=42, WASH_BLOCK_END=980230], seq:[0, 1, 2] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=978980, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=41, WASH_BLOCK_END=978939], seq:[0, 1, 2] [2025-08-14 09:02:05.718805] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=990711] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623025189535, flush_io_finished_ret_:0, flush_io_finished_round_:26303, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:3, normal_idle_loop_cnt_:3, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:04.723374] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=35] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:02:05.719364] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] update_check_range(min_check_version=2540, max_check_version=2540, global_version=2638) [2025-08-14 09:02:04.748272] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8949] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:05.719839] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=970098][errcode=0] epoll handler too much time :971576 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:04.766448] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=1520] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:05.719873] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=951983] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:05.719888] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:03.751187] EDIAG [RS.COMPACTION] try_idle (ob_major_merge_scheduler.cpp:226) [1275][T1001_MergeSche][T1001][YB42AC4D0007-00063C4963E99CBE-0-0] [lt=12][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1978364, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1978362, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1125d502 0x1125cf81 0x1125a4c3 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:03.751225] EDIAG [RS.COMPACTION] try_idle (ob_major_merge_scheduler.cpp:226) [1057][T1_MergeSchedul][T1][YB42AC4D0007-00063C4961599CC3-0-0] [lt=20][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1008618, owner: ObLog, click_count: 5, time dist:[FORMAT_END=974062, ObMalloc#start=1, ObMalloc#end=34553, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1125d502 0x1125cf81 0x1125a4c3 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:03.791254] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=970817][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1938525, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1938524, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:03.781275] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1955965, owner: ObLog, click_count: 5, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=1946816, ALLOC_END=9137, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:03.834147] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1903268, owner: ObLog, click_count: 5, time dist:[FORMAT_END=31, ObMalloc#start=1, ObMalloc#end=1894120, ALLOC_END=9116, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:04.559563] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1178985, owner: ObLog, click_count: 5, time dist:[FORMAT_END=26, ObMalloc#start=2, ObMalloc#end=1169840, ALLOC_END=9117, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [717]OB_MALLOC COST TOO MUCH TIME, cost_time=1022731, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=51, WASH_BLOCK_END=1022680], seq:[0, 1, 2] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=985706, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=41, WASH_BLOCK_END=985664], seq:[0, 1, 2] [2025-08-14 09:02:04.126146] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=679837][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1611886, owner: ObLog, click_count: 5, time dist:[FORMAT_END=15, ObMalloc#start=1, ObMalloc#end=1602682, ALLOC_END=9188, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:03.832326] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=7][errcode=-4018] LOGGER COST TOO MUCH TIME, cost: 1905294, owner: ObLog, click_count: 5, time dist:[FORMAT_END=912704, ObMalloc#start=0, ObMalloc#end=983493, ALLOC_END=9097, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3b4b 0x113d34aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [768]OB_MALLOC COST TOO MUCH TIME, cost_time=1036665, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=41, WASH_BLOCK_END=1036624], seq:[0, 1, 2] [2025-08-14 09:02:03.832334] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [571][T1_ArcSender][T1][YB42AC4D0007-00063C495BA99C16-0-0] [lt=13][errcode=-4018] LOGGER COST TOO MUCH TIME, cost: 1905325, owner: ObLog, click_count: 5, time dist:[FORMAT_END=912667, ObMalloc#start=1, ObMalloc#end=983484, ALLOC_END=9173, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3b4b 0x113d34aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [401]OB_MALLOC COST TOO MUCH TIME, cost_time=984587, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=984547], seq:[0, 1, 2] [2025-08-14 09:02:03.771589] EDIAG [RPC] create (ob_poc_rpc_server.cpp:140) [317][pnio1][T0][YB42AC4D0007-00063C49612B8F33-0-0] [lt=980095][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1966177, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=1, FORMAT_END=976827, ObMalloc#start=1, ObMalloc#end=980280, ALLOC_END=9063, APPEND_END=1], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x27a87913 0x97bf45f 0x97bd81d 0x97bac33 0x97ba441 0x27a8a6d6 0x97b7ba0 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:04.702266] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=36255][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1038532, owner: ObLog, click_count: 5, time dist:[FORMAT_END=25, ObMalloc#start=1, ObMalloc#end=1036680, ALLOC_END=1826, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [857]OB_MALLOC COST TOO MUCH TIME, cost_time=981627, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=981779, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:02:04.770264] EDIAG [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1951703, owner: ObLog, click_count: 3, time dist:[FORMAT_END=981535, ALLOC_END=970167, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a5bbf2 0x94da542 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [1031]OB_MALLOC COST TOO MUCH TIME, cost_time=1976007, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2073, WASH_BLOCK_END=981437, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=13220], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:02.717680] EDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4001713, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=0, FORMAT_END=2025686, ObMalloc#start=1, ObMalloc#end=1976018, ALLOC_END=5, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x18257a6e 0x1823b8ca 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:01.718032] EDIAG [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=46083][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5009831, owner: ObLog, click_count: 3, time dist:[FORMAT_END=5009828, ALLOC_END=3, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b51df0f 0x986bcde 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:01.719270] EDIAG [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=39021][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5008759, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=2, FORMAT_END=5008749, ALLOC_END=3, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b51df0f 0x986bcde 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [419]OB_MALLOC COST TOO MUCH TIME, cost_time=1006017, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1006009], seq:[0] [512]OB_MALLOC COST TOO MUCH TIME, cost_time=1005256, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=17, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1005218], seq:[0, 1, 2, 3, 4] [512]OB_MALLOC COST TOO MUCH TIME, cost_time=1985302, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=41, WASH_BLOCK_END=979994, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1005264], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:04.750164] EDIAG [STORAGE] runTimerTask (ob_checkpoint_service.cpp:243) [512][T1_TimerWK3_Flush][T1][Y0-0000000000000000-0-0] [lt=18474][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1985322, owner: ObLog, click_count: 5, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=1985309, ALLOC_END=2, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x1c3fe35c 0x1c3fcfe9 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:05.722976] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6209] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:05.723771] WDIAG pktc_handle_req_queue (pktc_post.h:138) [314][pnio1][T0][Y0-0000000000000000-0-0] [lt=517][errcode=0] PNIO [delay_warn] delay high: 134404958 [2025-08-14 09:02:05.723784] WDIAG pktc_handle_req_queue (pktc_post.h:138) [314][pnio1][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] PNIO [delay_warn] delay high: 134675960 [2025-08-14 09:02:05.723806] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [314][pnio1][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559482, sock=0x7f995f412a58, code=-47 [2025-08-14 09:02:04.766459] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2772] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:05.724319] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=956412] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:05.724328] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:05.724335] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:03.773139] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=18163][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:03.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:05.724527] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1948417][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:03.773|207=1.95s|210=0us|222=2us|total=1.95s) [2025-08-14 09:02:05.724544] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] timer task use too much time(end=1755162125724544, start=1755162123773137, delta=1951407) [2025-08-14 09:02:05.724553] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=0] task scheduled too delay(scheduled_delay=1932203, expected_run_ts_us_=1755162123792349, ObClockGenerator::getRealClock()=1755162125724552, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:03.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:05.724567] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:03.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:04.764449] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10102][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:03.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:05.724653] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=958750][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:04.754|207=970.32ms|210=0us|222=2us|total=970.33ms) [2025-08-14 09:02:05.724662] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] timer task use too much time(end=1755162125724661, start=1755162124754328, delta=970333) [2025-08-14 09:02:05.724679] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=0] task scheduled too delay(scheduled_delay=1735533, expected_run_ts_us_=1755162123989145, ObClockGenerator::getRealClock()=1755162125724679, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:03.989, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:05.724692] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:03.989, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:05.724713] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=998119, expected_run_ts_us_=1755162124726594, ObClockGenerator::getRealClock()=1755162125724713, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd110, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd3f0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:10.00s, expected_run_time:09:02:04.726, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:05.724723] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd110, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd3f0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:10.00s, expected_run_time:09:02:04.726, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:05.724734] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=969246, expected_run_ts_us_=1755162124755487, ObClockGenerator::getRealClock()=1755162125724734, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:04.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:05.724744] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:04.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:05.724761] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=635445, expected_run_ts_us_=1755162125089316, ObClockGenerator::getRealClock()=1755162125724761, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:05.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:05.724771] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:05.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:04.766511] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=1574] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:05.725269] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=957309] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:05.725277] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:04.769431] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=2116][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:05.725352] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=954472][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:02:05.725372] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:05.725382] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:05.725398] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:05.725413] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:05.725427] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:05.725437] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:05.725605] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:05.725617] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:05.725627] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:02:04.769528] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891968090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.767, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.725874] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=954897][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:04.769|207=956.35ms|210=0us|222=3us|total=956.37ms) [2025-08-14 09:02:05.725887] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task use too much time(end=1755162125725887, start=1755162124769517, delta=956370) [2025-08-14 09:02:05.725894] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] task scheduled too delay(scheduled_delay=1946493, expected_run_ts_us_=1755162123779400, ObClockGenerator::getRealClock()=1755162125725894, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ad690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.725906] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ad690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.725919] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1947586, expected_run_ts_us_=1755162123778332, ObClockGenerator::getRealClock()=1755162125725919, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196a490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.725929] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196a490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.725939] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1947358, expected_run_ts_us_=1755162123778581, ObClockGenerator::getRealClock()=1755162125725939, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196a310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.725949] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196a310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.725959] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1955805, expected_run_ts_us_=1755162123770154, ObClockGenerator::getRealClock()=1755162125725959, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7282610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.725971] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7282610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.725981] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1942330, expected_run_ts_us_=1755162123783651, ObClockGenerator::getRealClock()=1755162125725981, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72716b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919a4910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.725991] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72716b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919a4910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726001] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1937973, expected_run_ts_us_=1755162123788027, ObClockGenerator::getRealClock()=1755162125726001, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a0310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726010] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a0310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726022] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1937754, expected_run_ts_us_=1755162123788267, ObClockGenerator::getRealClock()=1755162125726022, ret=0, ret="OB_SUCCESS", *this={this:0x7f98918100b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a0190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726031] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918100b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a0190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726041] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1944829, expected_run_ts_us_=1755162123781212, ObClockGenerator::getRealClock()=1755162125726041, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f1990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726051] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f1990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726061] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1932266, expected_run_ts_us_=1755162123793794, ObClockGenerator::getRealClock()=1755162125726061, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891825510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.793, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726072] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891825510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.793, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726082] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1933682, expected_run_ts_us_=1755162123792400, ObClockGenerator::getRealClock()=1755162125726082, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180fe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a0010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726091] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180fe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a0010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726101] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1920354, expected_run_ts_us_=1755162123805747, ObClockGenerator::getRealClock()=1755162125726101, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a1990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.805, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726113] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a1990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.805, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726123] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1913567, expected_run_ts_us_=1755162123812555, ObClockGenerator::getRealClock()=1755162125726123, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73927b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.812, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726132] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73927b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.812, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726142] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1910676, expected_run_ts_us_=1755162123815466, ObClockGenerator::getRealClock()=1755162125726142, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7399090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.815, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726152] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7399090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.815, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726163] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1909865, expected_run_ts_us_=1755162123816298, ObClockGenerator::getRealClock()=1755162125726163, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7398f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726173] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7398f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726182] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1898853, expected_run_ts_us_=1755162123827329, ObClockGenerator::getRealClock()=1755162125726182, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73954b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f1210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.827, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726192] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73954b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f1210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.827, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726202] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1852518, expected_run_ts_us_=1755162123873684, ObClockGenerator::getRealClock()=1755162125726202, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891970510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.873, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726213] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891970510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:03.873, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726229] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1478056, expected_run_ts_us_=1755162124248172, ObClockGenerator::getRealClock()=1755162125726229, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7ee390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.248, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726247] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7ee390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.248, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726272] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=990742, expected_run_ts_us_=1755162124735529, ObClockGenerator::getRealClock()=1755162125726272, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7270030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919a3b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.735, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726281] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7270030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919a3b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.735, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726293] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=986387, expected_run_ts_us_=1755162124739906, ObClockGenerator::getRealClock()=1755162125726293, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ecb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ac310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.739, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726303] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ecb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ac310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.739, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726313] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=993553, expected_run_ts_us_=1755162124732759, ObClockGenerator::getRealClock()=1755162125726313, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7353730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7396990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726322] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7353730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7396990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726332] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=989470, expected_run_ts_us_=1755162124736861, ObClockGenerator::getRealClock()=1755162125726332, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7efd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726344] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7efd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726353] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=986197, expected_run_ts_us_=1755162124740156, ObClockGenerator::getRealClock()=1755162125726353, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ea30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ac190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726363] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ea30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ac190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726381] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] task scheduled too delay(scheduled_delay=972081, expected_run_ts_us_=1755162124754299, ObClockGenerator::getRealClock()=1755162125726380, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919afbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b9810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726390] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919afbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b9810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726401] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=967869, expected_run_ts_us_=1755162124758532, ObClockGenerator::getRealClock()=1755162125726401, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919854b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891972190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726411] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919854b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891972190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726421] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=967688, expected_run_ts_us_=1755162124758733, ObClockGenerator::getRealClock()=1755162125726421, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891972010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726431] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891972010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726441] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=967470, expected_run_ts_us_=1755162124758970, ObClockGenerator::getRealClock()=1755162125726441, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891971e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726452] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891971e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726462] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=973553, expected_run_ts_us_=1755162124752908, ObClockGenerator::getRealClock()=1755162125726462, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d45b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ec490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726471] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d45b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ec490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726481] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=959889, expected_run_ts_us_=1755162124766592, ObClockGenerator::getRealClock()=1755162125726481, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72747b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b4c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726491] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72747b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b4c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726502] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=960094, expected_run_ts_us_=1755162124766408, ObClockGenerator::getRealClock()=1755162125726502, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7280510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726512] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7280510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726522] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=950901, expected_run_ts_us_=1755162124775620, ObClockGenerator::getRealClock()=1755162125726521, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891825390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726531] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891825390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726541] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=950885, expected_run_ts_us_=1755162124775655, ObClockGenerator::getRealClock()=1755162125726541, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ad510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726552] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ad510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726562] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=954660, expected_run_ts_us_=1755162124771901, ObClockGenerator::getRealClock()=1755162125726562, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891969d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726571] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891969d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726581] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=953610, expected_run_ts_us_=1755162124772971, ObClockGenerator::getRealClock()=1755162125726581, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726591] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726603] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=940744, expected_run_ts_us_=1755162124785859, ObClockGenerator::getRealClock()=1755162125726603, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b5c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726613] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b5c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726623] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=941462, expected_run_ts_us_=1755162124785160, ObClockGenerator::getRealClock()=1755162125726623, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ec310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726632] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ec310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726642] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=922563, expected_run_ts_us_=1755162124804079, ObClockGenerator::getRealClock()=1755162125726642, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919836b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891970f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726654] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919836b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891970f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726664] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=896263, expected_run_ts_us_=1755162124830400, ObClockGenerator::getRealClock()=1755162125726664, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891970e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726673] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891970e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726683] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=895658, expected_run_ts_us_=1755162124831025, ObClockGenerator::getRealClock()=1755162125726683, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72eb590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.831, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726693] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72eb590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.831, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726704] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=864571, expected_run_ts_us_=1755162124862133, ObClockGenerator::getRealClock()=1755162125726704, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7280390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.862, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.726714] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7280390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:04.862, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:04.770935] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=32] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:05.728383] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=955993] temporary file module use no blocks [2025-08-14 09:02:05.728394] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:05.728406] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:05.728417] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623026210634, flush_io_finished_ret_:0, flush_io_finished_round_:26267, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:04.722444] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19095] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:05.728488] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1004493][errcode=0] epoll handler too much time :1006055 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:05.728504] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:03.731021] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [851][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=21] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=16, total_destructed_detector=16, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f9922fee2b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-08-14 09:02:05.730355] WDIAG [DETECT] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [851][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=1996286][errcode=0] cost too much time:ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t), (this=|threshold=100.00ms|start at 09:02:05.730|total=3.95s) [2025-08-14 09:02:04.769751] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [14780][T1001_TimerWK4_SSTableGC][T1001][YB42AC4D0007-00063C4974099C4E-0-0] [lt=15][errcode=0] cost too much time:mds_table_impl.ipp:virtual int oceanbase::storage::mds::MdsTableImpl, oceanbase::st..., (this=|threshold=5.00ms|start at 09:02:00.768|559=0us|total=4.00s) [2025-08-14 09:02:04.429288] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fa190, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa470, time_wheel:0x7f99341cdbf0, thread_pool:0x7f99341cd3d0, is_running:1, time_interval:2.00s, expected_run_time:09:02:04.334, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:02.783879] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=40] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:05.731489] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2943142][errcode=0] epoll handler too much time :2947616 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:05.731505] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:05.731516] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:02.784936] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=26] gc stale ls task succ [2025-08-14 09:02:05.731583] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=2942176] gc stale ls task succ [2025-08-14 09:02:03.791254] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=970817] gc stale ls task succ [2025-08-14 09:02:05.732737] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=1938525] gc stale ls task succ [2025-08-14 09:02:05.725285] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:02:05.737379] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:04.865226] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fc8b0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcb90, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:1, time_interval:3.00s, expected_run_time:09:02:04.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:05.740144] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=873591][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:04.865|207=874.91ms|210=1us|222=1us|total=874.94ms) [2025-08-14 09:02:05.740166] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4389] timer task use too much time(end=1755162125740166, start=1755162124865225, delta=874941) [2025-08-14 09:02:03.781275] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:05.731133] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=1299861][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:04.429|207=1.30s|210=0us|222=1us|total=1.30s) [2025-08-14 09:02:05.740274] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=9126][errcode=-4389] timer task use too much time(end=1755162125740273, start=1755162124429285, delta=1310988) [2025-08-14 09:02:03.834147] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:03.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:04.559563] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fe410, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe6f0, time_wheel:0x7f9922df9d40, thread_pool:0x7f9922df9520, is_running:1, time_interval:2.00s, expected_run_time:09:02:04.432, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:05.740352] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=1178985][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:04.559|207=1.18s|210=1us|222=2us|total=1.18s) [2025-08-14 09:02:05.740360] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] timer task use too much time(end=1755162125740360, start=1755162124559560, delta=1180800) [2025-08-14 09:02:04.126146] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=679837][errcode=0] EASY SLOW: thread: 7f99541f7300, lastrun: 1755162122.758630 cost: 1.367514 loop:26488, slowcnt: 13 [2025-08-14 09:02:03.832326] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=7][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:02:03.832334] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [571][T1_ArcSender][T1][YB42AC4D0007-00063C495BA99C16-0-0] [lt=13][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:02:04.746402] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [950][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F32-0-0] [lt=34][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1002, svr="172.77.0.7:2882", version={val:1755161758221798000, v:0}, valid_part_count=0, total_part_count=2, generate_timestamp=1755162121718595) [2025-08-14 09:02:04.751270] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:03.771589] WDIAG [RPC] create (ob_poc_rpc_server.cpp:140) [317][pnio1][T0][YB42AC4D0007-00063C49612B8F33-0-0] [lt=980095][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1815, fly_ts=950274, send_timestamp=1755162122783717, peer="172.77.0.7:42280", sz=265) [2025-08-14 09:02:05.740817] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [979][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F33-0-0] [lt=65][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1002, svr="172.77.0.7:2882", version={val:1755161758221798000, v:0}, valid_part_count=0, total_part_count=2, generate_timestamp=1755162121818668) [2025-08-14 09:02:05.740118] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:05.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:05.724345] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:02:05.741844] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:04.702266] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=36255][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:04.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:05.742400] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=1038532][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:04.702|207=1.04s|210=0us|222=2us|total=1.04s) [2025-08-14 09:02:05.742417] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] timer task use too much time(end=1755162125742417, start=1755162124702264, delta=1040153) [2025-08-14 09:02:05.719903] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=10] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:02:04.745683] INFO [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:171) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=7716] handle gts request statistics(total_rt=1953608, total_cnt=7, avg_rt=2.442010000000000000e+05, avg_cnt=6.999999999999999556e-01) [2025-08-14 09:02:05.742689] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=995495][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162123762672], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:05.742709] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162123762672], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:05.742727] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162123762672]}) [2025-08-14 09:02:05.742752] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [14780][T1001_TimerWK4_SSTableGC][T1001][YB42AC4D0007-00063C4974099C4E-0-0] [lt=959175][errcode=0] cost too much time:ob_i_tablet_mds_interface.ipp:int oceanbase::storage::ObITabletMdsInterface::get_latest(OP &&, mds::MdsWriter &, mds::TwoPhaseCommitState &, share::SCN &, const int64_t) const [T = oceanbase::storage::ObTabletCreateDeleteMdsUserDat..., (this=|threshold=10.00ms|start at 09:02:00.765|449=2us|458=1us|460=1us|467=4.96s|479=0us|490=9.81ms|total=4.98s) [2025-08-14 09:02:05.743371] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:05.740316] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1903268][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:03.834|207=1.91s|210=1us|222=1us|total=1.91s) [2025-08-14 09:02:05.744682] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=4358][errcode=-4389] timer task use too much time(end=1755162125744681, start=1755162123834145, delta=1910536) [2025-08-14 09:02:05.740229] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1955965][errcode=0] epoll handler too much time :1958963 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:05.744724] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4476] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:05.740787] WDIAG stat_cleanup (debug.c:35) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=1966177][errcode=0] PNIO [delay_warn] cost too much time: 2957021us, procedure: pn_pkts_handle_func [2025-08-14 09:02:05.744754] WDIAG eloop_delay_warn (debug.h:93) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=3958][errcode=0] PNIO [delay_warn] eloop handle events delay high: 2961021, malloc=0/0 write=0/0 read=0/1 server_process=2957021/1 client_cb=0/0 [2025-08-14 09:02:05.740810] INFO [COMMON] print_io_status (ob_io_manager.cpp:2733) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=1974464] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:SHARED_BLOCK_ReaderWriter_IO, mode:LOCAL READ, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:0, norm_iops:0, bw:455, [delay/us]:prepare:975225, schedule:427, submit:1, rt:46449, total:1022102) [2025-08-14 09:02:05.745283] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:05.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:05.745923] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:05.746280] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:05.750557] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:04.764861] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=46] finish to resize log external storage handler(current_ts=1755162124727989, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-08-14 09:02:05.750322] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1236][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:05.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:04.724506] INFO [SQL.PC] need_check_schema_version (ob_plan_cache_value.cpp:2334) [699][T1001_DupTbLeas][T1][YB42AC4D0007-00063C495FB9A46F-0-0] [lt=18] need check schema(new_schema_version=1753073979411448, cached_tenant_schema_version=1753073979411448, is_contain_synonym()=false, contain_sys_name_table_=true, is_contain_tmp_tbl()=false, is_contain_sys_pl_object()=false, need_check=true, constructed_sql_=select * from __all_ls_status where tenant_id = ? and flag like ? order by ls_id limit ?) [2025-08-14 09:02:04.798352] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [412][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=20] [sql nio session](*s={this:0x7f98f733a030, session_id:3221590255, trace_id:YB42AC4D0007-00063C49651A3232-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:257}, err:0, last_decode_time:1755161952736711, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1716, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:05.747864] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196a790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.754767] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=6888][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918114b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a0d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.754796] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728aa90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.754808] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f0490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:05.731999] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EE9F-0-0] [lt=18] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-08-14 09:02:05.757364] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=20] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:05.763347] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:05.770994] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:05.818671] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:05.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:05.818827] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:02:05.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:05.918735] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:05.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:05.944829] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:05.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:05.992270] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=3002][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:05.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:06.044898] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:05.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:06.243677] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:06.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:06.253740] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:06.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:06.495105] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:06.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:06.642879] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:06.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:04.770264] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=11] [sql nio session](*s={this:0x7f98f733f030, session_id:3221589417, trace_id:YB42AC4D0007-00063C496359B268-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:252}, err:0, last_decode_time:1755161813752329, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1778, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:06.725394] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [14780][T1001_TimerWK4_SSTableGC][T1001][YB42AC4D0007-00063C4974099C4E-0-0] [lt=17][errcode=0] cost too much time:ob_i_tablet_mds_interface.cpp:int oceanbase::storage::ObITabletMdsInterface::get_latest_tablet_status(oceanbase::storage::ObTabletCreateDeleteMdsUserData &, mds::MdsWriter &, mds::TwoPhaseCommitState &, share::SCN &, const int64_t)..., (this=|threshold=10.00ms|start at 09:02:00.765|97=1us|total=5.96s) [2025-08-14 09:02:02.717680] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:06.725488] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4001713] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:05.754818] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f0610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.725712] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=969423][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:05.754|207=970.89ms|210=0us|222=2us|total=970.91ms) [2025-08-14 09:02:05.765416] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:05.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:06.018772] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:05.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:06.725809] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=705965][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:06.187|207=707.03ms|210=1us|222=1us|total=707.05ms) [2025-08-14 09:02:06.725819] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task use too much time(end=1755162126725819, start=1755162126018770, delta=707049) [2025-08-14 09:02:06.140745] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=1611886][errcode=0] EASY SLOW: thread: 7f995407d300, lastrun: 1755162124.751261 cost: 1.389481 loop:26497, slowcnt: 4 [2025-08-14 09:02:06.144961] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:06.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:06.725851] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=580009][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:06.144|207=580.89ms|210=0us|222=1us|total=580.90ms) [2025-08-14 09:02:06.725862] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task use too much time(end=1755162126725862, start=1755162126144960, delta=580902) [2025-08-14 09:02:06.225802] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:06.725887] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=499324][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:02:06.725907] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:06.440731] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fa190, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa470, time_wheel:0x7f99341cdbf0, thread_pool:0x7f99341cd3d0, is_running:1, time_interval:2.00s, expected_run_time:09:02:06.334, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:06.540870] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fe410, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe6f0, time_wheel:0x7f9922df9d40, thread_pool:0x7f9922df9520, is_running:1, time_interval:2.00s, expected_run_time:09:02:06.432, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:05.847275] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=17542][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:05.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:06.725969] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=184817][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:06.540|207=185.10ms|210=0us|222=1us|total=185.11ms) [2025-08-14 09:02:06.725979] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task use too much time(end=1755162126725979, start=1755162126540868, delta=185111) [2025-08-14 09:02:06.728507] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=21] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:06.718313] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=1931392] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:02:06.718920] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=40] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:06.734936] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=15988] temporary file module use no blocks [2025-08-14 09:02:06.734948] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:06.734960] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:06.734971] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623028158614, flush_io_finished_ret_:0, flush_io_finished_round_:26304, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:06.725978] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=877370][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:05.847|207=878.70ms|210=0us|222=1us|total=878.73ms) [2025-08-14 09:02:06.719259] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:06.735340] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16054][errcode=0] epoll handler too much time :16090 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:01.718032] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=46083] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136826952869001, v:0}, clog_base_lsn:{lsn:5099962368}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1755071646732636000, v:0}, all_id_meta:{id_meta:[{limited_id:1755161555795565000, latest_log_ts:{val:1755161545796642001, v:0}}, {limited_id:22000001, latest_log_ts:{val:1755135645787806003, v:0}}, {limited_id:60000001, latest_log_ts:{val:1755135646400653001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:50, palf_env_:0x7f9950bf4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-08-14 09:02:01.719270] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=39021] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136829134139001, v:0}, clog_base_lsn:{lsn:8186781696}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1753840867372367034, v:0}, all_id_meta:{id_meta:[{limited_id:1755161729775179000, latest_log_ts:{val:1755161719775179001, v:0}}, {limited_id:18000001, latest_log_ts:{val:1755157440846556001, v:0}}, {limited_id:60000001, latest_log_ts:{val:1755135657266496001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:46, palf_env_:0x7f99321fa030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-08-14 09:02:06.724988] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=986963] snapshot_traffic_info once [2025-08-14 09:02:05.753287] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=22732] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:02:06.733319] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=5] gc stale ls task succ [2025-08-14 09:02:06.728531] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=17] temporary file module use no blocks [2025-08-14 09:02:06.735911] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=7366] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:06.735921] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:06.735931] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623028168206, flush_io_finished_ret_:0, flush_io_finished_round_:26268, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:03.779007] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=27] blacklist_loop exec finished(cost_time=18, is_enabled=true, send_cnt=0) [2025-08-14 09:02:06.725784] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=958914][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:05.765|207=960.37ms|210=0us|222=1us|total=960.38ms) [2025-08-14 09:02:06.738104] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:06.725727] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] timer task use too much time(end=1755162126725726, start=1755162125754818, delta=970908) [2025-08-14 09:02:05.753257] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9161] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:02:06.727257] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:06.725916] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:06.738198] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12260][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:02:06.738217] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:06.738227] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:06.738253] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:06.738264] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:06.738286] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:06.733444] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:06.724948] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=1951703] [sql nio session](*s={this:0x7f98f733e030, session_id:3221589980, trace_id:YB42AC4D0007-00063C496359B261-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:251}, err:0, last_decode_time:1755161748778070, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1681, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:06.733374] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=5] gc stale ls task succ [2025-08-14 09:02:04.750164] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:243) [512][T1_TimerWK3_Flush][T1][Y0-0000000000000000-0-0] [lt=18474] ====== traversal_flush timer task ====== [2025-08-14 09:02:06.725502] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=12] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3839) [2025-08-14 09:02:06.738513] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=12990][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:06.738691] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:06.738702] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3840) [2025-08-14 09:02:06.738707] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:06.724555] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=34] get wrs ts(ls_id={id:1}, delta=578985882, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [321]OB_MALLOC COST TOO MUCH TIME, cost_time=997896, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=16, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=997859], seq:[0, 1, 2, 3, 4] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=2887051, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=43, WASH_BLOCK_END=1889098, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=997908], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:03.848415] EDIAG eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2887078, owner: ObLog, click_count: 5, time dist:[FORMAT_END=14, ObMalloc#start=0, ObMalloc#end=2887058, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x27a8a8c8 0x97b7e1a 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [393]OB_MALLOC COST TOO MUCH TIME, cost_time=997911, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=997892], seq:[0, 1, 2] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=997922, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=997921], seq:[0, 1] [2025-08-14 09:02:04.753263] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1983678, owner: ObLog, click_count: 7, time dist:[FORMAT_END=7, ObMalloc#start=0, ObMalloc#end=985716, ObMalloc#start=22, ObMalloc#end=997928, ALLOC_END=2, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [717]OB_MALLOC COST TOO MUCH TIME, cost_time=998006, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=24, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=997963], seq:[0, 1, 2, 3, 4] [717]OB_MALLOC COST TOO MUCH TIME, cost_time=998017, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=998015], seq:[0, 1] [2025-08-14 09:02:03.743559] EDIAG [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2991949, owner: ObLog, click_count: 9, time dist:[ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=43, FORMAT_END=971090, ObMalloc#start=1, ObMalloc#end=1022759, ObMalloc#start=25, ObMalloc#end=998023, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0xfd250b6 0xfd247d3 0xfd244ea 0xfd242a2 0x1d19629b 0x1d17baa3 0x1d17ad94 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [401]OB_MALLOC COST TOO MUCH TIME, cost_time=997998, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=997976], seq:[0, 1, 2, 3, 4] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=998009, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=998008], seq:[0, 1] [2025-08-14 09:02:04.754361] EDIAG [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1982757, owner: ObLog, click_count: 7, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=984707, ObMalloc#start=20, ObMalloc#end=998015, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a1f867 0x27a135c3 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [387]OB_MALLOC COST TOO MUCH TIME, cost_time=991254, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=991219], seq:[0, 1, 2, 3, 4, 5, 6] [387]OB_MALLOC COST TOO MUCH TIME, cost_time=991265, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=991265], seq:[0, 1] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=989315, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=989275], seq:[0, 1, 2, 3, 4, 5, 6] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=989326, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=989325], seq:[0, 1] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=988162, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=988125], seq:[0, 1, 2, 3, 4, 5, 6] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=988174, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=988173], seq:[0, 1] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=985723, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=985688], seq:[0, 1, 2, 3, 4, 5, 6] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=985735, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=985734], seq:[0, 1] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=981549, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=981511], seq:[0, 1, 2, 3, 4, 5, 6] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=981561, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=981560], seq:[0, 1] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=955578, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=955540], seq:[0, 1, 2, 3, 4, 5, 6] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=955588, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=955588], seq:[0, 1] [513]OB_MALLOC COST TOO MUCH TIME, cost_time=878942, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=878906], seq:[0, 1, 2, 3, 4, 5, 6] [513]OB_MALLOC COST TOO MUCH TIME, cost_time=878952, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=878951], seq:[0, 1] [2025-08-14 09:02:03.848415] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=17] PNIO [ratelimit] time: 1755162123848414, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007616, add_bytes: 0 [2025-08-14 09:02:06.739895] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=2887078] PNIO [ratelimit] time: 1755162126739894, bytes: 0, bw: 0.000000 MB/s, add_ts: 2891480, add_bytes: 0 [2025-08-14 09:02:04.753263] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:06.739965] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1983678][errcode=0] epoll handler too much time :1986719 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:06.739987] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:03.743559] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=18] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:06.740054] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=2991949] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623025183186, flush_io_finished_ret_:0, flush_io_finished_round_:26293, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:3, normal_idle_loop_cnt_:3, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:04.754361] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=10] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:05.747393] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:06.740187] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=991294][errcode=0] epoll handler too much time :992807 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:05.750583] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:06.740279] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=988200][errcode=0] epoll handler too much time :989703 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:05.757292] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=11] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:05.783346] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=28] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:06.740403] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=955611][errcode=0] epoll handler too much time :957068 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:06.718801] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=960923] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:06.723855] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119812272, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-08-14 09:02:06.743478] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:06.745441] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:06.745504] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:06.746251] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:06.746888] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:06.748286] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:06.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:03.779007] EDIAG [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2952532, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2952531, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1fce4e91 0x1fce455d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:05.740709] EDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=987943][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1011251, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=21, FORMAT_END=1011209, ObMalloc#start=0, ObMalloc#end=3, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:05.749389] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1004452, owner: ObLog, click_count: 5, time dist:[FORMAT_END=18, ObMalloc#start=1, ObMalloc#end=989332, ALLOC_END=15097, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [419]OB_MALLOC COST TOO MUCH TIME, cost_time=2694324, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=47, WASH_BLOCK_END=1668684, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1006030], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:04.059639] EDIAG try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2694359, owner: ObLog, click_count: 7, time dist:[FORMAT_END=12, ObMalloc#start=0, ObMalloc#end=2694340, ALLOC_END=4, APPEND_END=3, ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=2], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x26499505 0x97074a3 0x26498fa9 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:03.772196] EDIAG [ELECT] operator() (election_proposer.cpp:232) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2982336, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2982331, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xfa07f77 0xfa07a14 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:05.753211] EDIAG [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=17002][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1004629, owner: ObLog, click_count: 3, time dist:[FORMAT_END=984266, ALLOC_END=20360, APPEND_END=3], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x2006a4c4 0x997aa00 0x2006ed17 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:04.722126] EDIAG [RS] get_valid_sts_after (ob_tenant_info_loader.cpp:454) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1561224][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2035484, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2035479, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1001d942 0x996d843 0x996d15a 0x996cfd8 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:06.750884] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:06.742940] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:06.752939] WDIAG ev_invoke_pending (ev.c:2136) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9982][errcode=0] epoll handler too much time :10016 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:05.740709] WDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=987943][errcode=0] epoll handler too much time :989447 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:04.753351] WDIAG [COMMON] wait (ob_io_define.cpp:880) [511][T1_TimerWK2_SSTableGC][T1][YB42AC4D0007-00063C495D6A02D9-0-0] [lt=950289][errcode=-4012] fail to wait result condition(ret=-4012, wait_ms=7993, *this={is_inited_:true, is_finished_:true, is_canceled_:false, has_estimated_:false, complete_size_:4096, offset_:1802570, size_:921, timeout_us_:10000000, result_ref_cnt_:2, out_ref_cnt_:2, flag_:{mode:"READ", group_id_:0, func_type_:0, wait_event_id_:3, is_sync_:false, is_unlimited_:false, is_detect_:false, is_write_through_:false, is_sealed_:true, is_time_detect_:false, need_close_dev_and_fd_:false, reserved_:0}, ret_code_:{io_ret_:0, fs_errno_:0}, tenant_id_:1, tenant_io_mgr_:{ptr:0x7f994b36a030}, user_data_buf_:null, buf_:null, io_callback_:0x7f9858859400, time_log:{begin_ts:1755162108745057, enqueue_used:975225, dequeue_used:427, submit_used:1, return_used:46449, callback_enqueue_used:986995, callback_dequeue_used:10387, callback_finish_used:13982444, end_used:-20355}}) [2025-08-14 09:02:06.753524] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1011251] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:06.753540] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:05.776421] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:05.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:06.753901] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=975999][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:05.776|207=977.48ms|210=0us|222=2us|total=977.49ms) [2025-08-14 09:02:06.753919] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] timer task use too much time(end=1755162126753918, start=1755162125776419, delta=977499) [2025-08-14 09:02:06.753931] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=961581, expected_run_ts_us_=1755162125792349, ObClockGenerator::getRealClock()=1755162126753930, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:05.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:06.753947] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:05.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:06.737279] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:191) [1003][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=11] ls blacklist refresh finish(cost_time=55986529) [2025-08-14 09:02:06.725932] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=284768][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:06.440|207=285.20ms|210=0us|222=1us|total=313.80ms) [2025-08-14 09:02:06.754538] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=28562][errcode=-4389] timer task use too much time(end=1755162126754538, start=1755162126440728, delta=313810) [2025-08-14 09:02:06.718346] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=1956202] current monitor number(seq_=-1) [2025-08-14 09:02:06.737927] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12123][errcode=-4389] timer task use too much time(end=1755162126737927, start=1755162125765413, delta=972514) [2025-08-14 09:02:06.754665] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=16705][errcode=0] task scheduled too delay(scheduled_delay=765518, expected_run_ts_us_=1755162125989145, ObClockGenerator::getRealClock()=1755162126754664, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:05.989, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:06.754686] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:05.989, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:06.754704] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] task scheduled too delay(scheduled_delay=665388, expected_run_ts_us_=1755162126089316, ObClockGenerator::getRealClock()=1755162126754704, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:06.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:06.754719] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:06.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:06.754734] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=550437, expected_run_ts_us_=1755162126204297, ObClockGenerator::getRealClock()=1755162126754734, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:06.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:06.754748] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:06.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:05.860123] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a58b8, timer:0x7f9966ee3e30, task:0x7f994f1a3a78, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr15MediumCheckTaskE", scheduled_time:1755161764744054, pushed_time:0, delay:20000000}, running_task_token={this:0x7f994e1a57d8, timer:0x7f9966ee3e30, task:0x7f994f1a3a38, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr14MediumLoopTaskE", scheduled_time:1755161760733696, pushed_time:1755161760733763, delay:120000000}, total_delay=361116068, threashold=1000000, tenant_id=1) [2025-08-14 09:02:05.749389] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:06.755396] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1004452][errcode=0] epoll handler too much time :1006024 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:06.755423] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:06.738123] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12374][errcode=0] task scheduled too delay(scheduled_delay=1004685, expected_run_ts_us_=1755162125733437, ObClockGenerator::getRealClock()=1755162126738123, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7281290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755444] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17293][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7281290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755482] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] task scheduled too delay(scheduled_delay=1013854, expected_run_ts_us_=1755162125741627, ObClockGenerator::getRealClock()=1755162126755482, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891825810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755500] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891825810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755514] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=1011961, expected_run_ts_us_=1755162125743552, ObClockGenerator::getRealClock()=1755162126755513, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891971290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755527] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891971290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755540] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1012384, expected_run_ts_us_=1755162125743155, ObClockGenerator::getRealClock()=1755162126755540, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a0c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755553] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a0c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755569] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] task scheduled too delay(scheduled_delay=998669, expected_run_ts_us_=1755162125756899, ObClockGenerator::getRealClock()=1755162126755569, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fc830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891825210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755582] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fc830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891825210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755595] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=999406, expected_run_ts_us_=1755162125756188, ObClockGenerator::getRealClock()=1755162126755595, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7282910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755607] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7282910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755620] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=999301, expected_run_ts_us_=1755162125756319, ObClockGenerator::getRealClock()=1755162126755620, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a54b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7282a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755635] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a54b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7282a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755649] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=995101, expected_run_ts_us_=1755162125760547, ObClockGenerator::getRealClock()=1755162126755649, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72edc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755661] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72edc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755674] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=990456, expected_run_ts_us_=1755162125765218, ObClockGenerator::getRealClock()=1755162126755674, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72edb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755687] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72edb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755702] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=989579, expected_run_ts_us_=1755162125766122, ObClockGenerator::getRealClock()=1755162126755702, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7398490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755715] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7398490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755728] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=987094, expected_run_ts_us_=1755162125768633, ObClockGenerator::getRealClock()=1755162126755728, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7398310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755740] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7398310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755753] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=994755, expected_run_ts_us_=1755162125760998, ObClockGenerator::getRealClock()=1755162126755753, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7289110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755768] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7289110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755782] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=984947, expected_run_ts_us_=1755162125770834, ObClockGenerator::getRealClock()=1755162126755782, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ac910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755794] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ac910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755807] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=976467, expected_run_ts_us_=1755162125779339, ObClockGenerator::getRealClock()=1755162126755807, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728ac10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755821] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728ac10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755835] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=952929, expected_run_ts_us_=1755162125802905, ObClockGenerator::getRealClock()=1755162126755835, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b5690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.802, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755847] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b5690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.802, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755860] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=954386, expected_run_ts_us_=1755162125801474, ObClockGenerator::getRealClock()=1755162126755860, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891970990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755873] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891970990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755888] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] task scheduled too delay(scheduled_delay=951577, expected_run_ts_us_=1755162125804310, ObClockGenerator::getRealClock()=1755162126755888, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728f9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7292010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755900] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728f9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7292010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755913] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=937373, expected_run_ts_us_=1755162125818540, ObClockGenerator::getRealClock()=1755162126755913, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72eae10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755926] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72eae10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755939] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=933031, expected_run_ts_us_=1755162125822907, ObClockGenerator::getRealClock()=1755162126755939, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739f290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755955] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739f290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755968] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=890610, expected_run_ts_us_=1755162125865358, ObClockGenerator::getRealClock()=1755162126755968, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7289e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755981] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7289e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.755993] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=889220, expected_run_ts_us_=1755162125866773, ObClockGenerator::getRealClock()=1755162126755993, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7288f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.866, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.756006] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7288f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:05.866, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.756028] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=426928, expected_run_ts_us_=1755162126329099, ObClockGenerator::getRealClock()=1755162126756028, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7281b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.756040] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7281b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.756059] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72756b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b5510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.756072] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7283b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.724, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.756085] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a36b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7283c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.756100] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919911b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196cc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.756113] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d40b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ec190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.756126] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728b390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.756138] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728b510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.756151] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.756166] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.756179] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b9210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.744, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.756192] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919aaf90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.756209] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919aba10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.738168] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10895][errcode=0] epoll handler too much time :10921 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:06.738327] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=13356] [sql nio session](*s={this:0x7f98f7332030, session_id:3221590132, trace_id:YB42AC4D0007-00063C496359B265-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:266}, err:0, last_decode_time:1755161768783802, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:2125, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:06.757949] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=11] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:06.740140] WDIAG ev_invoke_pending (ev.c:2136) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=1982757][errcode=0] epoll handler too much time :1985783 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:06.757984] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=17809] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:06.758000] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=10] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:06.735159] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=9165][errcode=-4389] timer task use too much time(end=1755162126735158, start=1755162125847273, delta=887885) [2025-08-14 09:02:04.059639] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=11] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:02:06.758136] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:06.738296] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:06.758408] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19900][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:06.758427] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:06.758442] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:02:06.735361] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts statistics(tenant_id=1002, gts_rpc_cnt=0, get_gts_cache_cnt=15, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:02:06.758554] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=23155] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162123762672]}) [2025-08-14 09:02:06.758570] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1002}) [2025-08-14 09:02:06.758555] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=76][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:06.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:06.758582] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1002}) [2025-08-14 09:02:06.759098] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:0, time_interval:500.00ms, expected_run_time:09:02:07.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:06.759115] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:02:06.759|total=3.00s) [2025-08-14 09:02:06.759131] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] task executed too delay(delay_time=103544337, *p_task_={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:0, time_interval:500.00ms, expected_run_time:09:02:07.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:06.759206] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] task executed too delay(delay_time=102953711, *p_task_={this:0x7f99319fc190, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc470, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:0, time_interval:10.00s, expected_run_time:09:02:08.720, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:06.740511] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16614] get_min_unreplayed_log_info(lsn={lsn:1604088821}, scn={val:1755100948296325001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873479, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:12492179, ret_code_:0}}, next_to_submit_lsn_:{lsn:1604088821}, next_to_submit_scn_:{val:1755100948296325001, v:0}, base_lsn_:{lsn:1409200128}, base_scn_:{val:1753840802540914000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:1604088821}, end_lsn:{lsn:1604088821}, read_buf:{buf_len_:3747840, buf_:0x7f98f6a05000}, block_size:67104768, log_storage_:0x7f990bfebbf0, log_storage_:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:42, read_io_size:147709952, read_disk_cost_ts:1061450, start_lsn:{lsn:1409200128}}}}}}) [2025-08-14 09:02:06.759244] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18688] get wrs ts(ls_id={id:1}, delta=377007539, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:06.759260] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:05.753211] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=17002] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:02:06.760719] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=2035484] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:02:06.760737] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:02:06.760750] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:06.760761] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:06.735752] WDIAG [SHARE] get_unit_groups_by_tenant (ob_unit_table_operator.cpp:890) [784][T1002_TBalance][T1002][YB42AC4D0007-00063C4964C9A63F-0-0] [lt=993041][errcode=-4012] failed to read unit group(ret=-4012, ret="OB_TIMEOUT", sql=SELECT distinct unit_group_id, status from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1002) order by unit_group_id ) [2025-08-14 09:02:06.739366] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14782] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:06.756223] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ad390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.762442] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=6206][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196cb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.762468] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196b810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.763012] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EE9F-0-0] [lt=22946][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571fc8b0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcb90, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:0, time_interval:3.00s, expected_run_time:09:02:07.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:06.763032] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EE9F-0-0] [lt=19][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:02:06.763|total=4.91s) [2025-08-14 09:02:06.763049] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EE9F-0-0] [lt=12][errcode=-4016] task executed too delay(delay_time=1897831, *p_task_={this:0x7f99571fcc90, caller:mds_tenant_service.cpp:mtl_start:117, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_145, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcf70, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:0, time_interval:15.00s, expected_run_time:09:02:19.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:06.764733] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:06.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:06.766033] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [726][T1001_ObXAHbWor][T1001][Y0-0000000000000000-0-0] [lt=27] XA scheduler heartbeat task statistics(avg_time=3252) [2025-08-14 09:02:06.762397] WDIAG [RPC] handle_resp (ob_poc_rpc_proxy.cpp:192) [314][pnio1][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] async_cb handler cost too much time(total_time=1009212, decode_time=-1755162125740686, process_time=0, session_destroy_time=1755162126749898, ret=0, pcode=5395) [2025-08-14 09:02:06.766307] WDIAG stat_cleanup (debug.c:35) [314][pnio1][T0][Y0-0000000000000000-0-0] [lt=3902][errcode=0] PNIO [delay_warn] cost too much time: 1043007us, procedure: pn_pktc_resp_cb [2025-08-14 09:02:06.767482] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ad210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.751, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.768806] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EEA1-0-0] [lt=14] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-08-14 09:02:06.762916] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:06.770537] WDIAG [SHARE] read_units (ob_unit_table_operator.cpp:1282) [742][T1001_DBMSSched][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4012] execute sql failed(sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1002), ret=-4012) [2025-08-14 09:02:06.770561] WDIAG [SHARE] get_units_by_tenant (ob_unit_table_operator.cpp:927) [742][T1001_DBMSSched][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4012] read_units failed(ret=-4012, ret="OB_TIMEOUT", sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1002)) [2025-08-14 09:02:06.773106] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:06.773338] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=14] [sql nio session](*s={this:0x7f98f72b3030, session_id:3221562808, trace_id:YB42AC4D0007-00063C4965DA1F2E-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:267}, err:0, last_decode_time:1755161748766816, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1890, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:06.773363] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=25] [sql nio session](*s={this:0x7f98f72ab030, session_id:3221589435, trace_id:YB42AC4D0007-00063C4965DA1F35-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:246}, err:0, last_decode_time:1755161762721798, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:2456, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:06.773377] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=13] [sql nio session](*s={this:0x7f98f72a9030, session_id:3221590528, trace_id:YB42AC4D0007-00063C4965DA1F34-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:250}, err:0, last_decode_time:1755161762715415, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:205, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:06.773397] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=19] [sql nio session](*s={this:0x7f98f72b5030, session_id:3221590961, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:10, sql_initiative_shutdown:false, reader:{fd:278}, err:5, last_decode_time:1755162075745307, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:06.775410] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:06.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:06.777546] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891825b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.782506] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:06.787608] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919acf10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.787642] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f0790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.793576] INFO handle (memory_dump.cpp:671) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=14] statistics: tenant_cnt: 6, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 168 271 0 1 2 110 1 0 1 3 7 0 0 1 4 1 1 0 1 5 86 6 0 1 6 13 0 0 1 7 22 2 0 1 8 53 1 0 1 12 17 3 0 1 13 10 1 0 1 14 1 1 0 1 15 1 1 0 1 16 5 6 0 1 18 4 3 0 1 20 2 1 0 1 22 1 1 0 1 24 1 0 0 1 25 6 0 0 500 0 558 274 0 500 7 191 6 0 500 8 41 1 0 500 9 2 1 0 500 10 3 2 0 500 16 50 4 0 500 17 52 7 0 500 22 38 64 0 500 23 27 10 0 508 0 2 7 0 508 8 5 1 0 509 0 2 7 0 509 8 5 1 0 1001 0 147 249 0 1001 2 26 1 0 1001 3 5 0 0 1001 4 1 1 0 1001 5 85 6 0 1001 6 11 0 0 1001 7 30 2 0 1001 8 36 1 0 1001 12 17 3 0 1001 13 3 1 0 1001 14 1 1 0 1001 16 6 2 0 1001 18 4 3 0 1001 20 4 1 0 1001 25 8 0 0 1002 0 262 278 0 1002 2 198 1 0 1002 3 4 0 0 1002 4 1 1 0 1002 5 101 6 0 1002 6 14 0 0 1002 7 6 2 0 1002 8 73 1 0 1002 12 18 3 0 1002 13 7 1 0 1002 14 1 2 0 1002 16 8 8 0 1002 18 9 3 0 1002 20 2 1 0 1002 21 1 0 0 1002 24 1 0 0 1002 25 7 0 0 cost_time: 1074255 [2025-08-14 09:02:06.797668] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919abe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.797699] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b5090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.797715] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199fe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919acd90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.797731] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e80b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891973b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.797756] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a31b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ede10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.805595] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:06.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:06.817797] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ab890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.806, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.817830] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196a610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.817843] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7280210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.817863] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7288510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.800, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.817879] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728e330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7290690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.805, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.817889] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f1b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.803, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.820818] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:02:06.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:06.825846] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:06.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:06.837923] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739e990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.824, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.847986] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891969b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.832, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.858245] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=2694359] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:02:06.860821] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:06.878181] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ade10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.864, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.878216] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891921990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.878229] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919adc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.878257] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919adb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.888245] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72865b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7288990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.878, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.908372] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728de30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7290b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.899, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.918444] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7ee210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:06.902, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:06.925911] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=69][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:06.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:06.960880] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:06.990825] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:06.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:06.996167] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:06.989, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.016289] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:07.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.025983] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:06.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.026693] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:07.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.060904] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:07.126046] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:07.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.129652] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7289d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.111, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.160948] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:07.218996] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=581381][errcode=0] EASY SLOW: thread: 7f99542fb300, lastrun: 1755162115.752503 cost: 11.466490 loop:26416, slowcnt: 78 [2025-08-14 09:02:07.226112] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:07.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.238791] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:07.242247] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:07.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.247373] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:07.239, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.252309] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=55][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:07.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.258622] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:07.258691] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=65][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:02:07.258708] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:07.258725] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:07.258751] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:02:07.258774] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:07.258791] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:07.258812] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:07.258835] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:07.258851] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:07.258864] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:07.259034] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:07.259048] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:07.259061] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:02:07.259153] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:07.259168] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162127259149], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:07.259185] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162127259149], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:07.259198] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162127259149]}) [2025-08-14 09:02:07.259211] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=10, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:02:07.259223] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162127259149]}) [2025-08-14 09:02:07.259248] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=21] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:07.259258] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162127259247], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:07.259280] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=20] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162127259247]}) [2025-08-14 09:02:07.259289] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts statistics(tenant_id=1001, gts_rpc_cnt=0, get_gts_cache_cnt=0, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:02:07.259298] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162127259247]}) [2025-08-14 09:02:07.260989] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:07.267464] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:07.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.278016] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:07.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.308144] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:07.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.326179] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:07.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.361052] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:07.426204] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:07.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.461093] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:07.493587] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:07.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.498862] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:07.489, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.518964] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:07.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.526285] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=57][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:07.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.529394] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=77][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:07.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.561153] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:07.599333] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:07.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.626360] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:07.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.643583] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571634a0, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957163780, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:5.00s, expected_run_time:09:02:07.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.643619] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:07.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.661205] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:07.695844] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=30012][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571630c0, caller:mds_tenant_service.cpp:mtl_start:117, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_145, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571633a0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.695887] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:1, time_interval:3.00s, expected_run_time:09:02:07.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.716053] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23019] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162126723947}, version_delta=579977536) [2025-08-14 09:02:07.720196] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:07.721370] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=1553221][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc5a78, timer:0x7f9970f8c420, task:0x564d6e8e31f8, task_type:"N9oceanbase3omt19ObTenantTimezoneMgr15AddTenantTZTaskE", scheduled_time:1755161702763912, pushed_time:0, delay:5000000}, running_task_token={this:0x7f9970fc66b8, timer:0x7f9970f8c420, task:0x564d6e8e3228, task_type:"N9oceanbase3omt19ObTenantTimezoneMgr18UpdateTenantTZTaskE", scheduled_time:1755161702763810, pushed_time:1755161702764275, delay:5000000}, total_delay=424957457, threashold=1000000, tenant_id=500) [2025-08-14 09:02:07.721892] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18765] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:06.766065] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [719][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571f9d70, caller:ob_table_lock_service.cpp:start:276, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE6$_1273, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa090, time_wheel:0x7f99375f7d10, thread_pool:0x7f99375f74f0, is_running:0, time_interval:10.00s, expected_run_time:09:02:12.332, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.726704] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [719][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=959185][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:02:07.726|total=5.23s) [2025-08-14 09:02:07.728344] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:07.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.730391] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:07.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.733376] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4866] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:07.733420] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=4965] gc stale ls task succ [2025-08-14 09:02:07.238825] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:227) [1031][PxTargetMgr0][T1][Y0-0000000000000000-0-0] [lt=29] refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3841) [2025-08-14 09:02:07.734223] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=494647][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:07.734261] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=25] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:07.734283] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=22] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3842) [2025-08-14 09:02:07.734292] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:07.734483] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:07.734496] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=12] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3843) [2025-08-14 09:02:07.734504] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:06.767172] WDIAG [SERVER.OMT] operator() (ob_tenant_timezone_mgr.cpp:50) [272][TimerWK0_TimezoneMgr][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4012] fail to update time zone info(ret=-4012) [2025-08-14 09:02:07.735161] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=16563] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:02:07.735287] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=2535] gc stale ls task succ [2025-08-14 09:02:07.737326] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=22] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:07.737356] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=25] temporary file module use no blocks [2025-08-14 09:02:07.737368] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:07.737390] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=21] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:07.737404] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623029175722, flush_io_finished_ret_:0, flush_io_finished_round_:26269, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:07.737697] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:07.740463] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:07.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.740620] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=981585] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:07.740665] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=21633] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:07.741425] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:07.743859] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:07.745413] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:07.746484] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:07.747211] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:07.747357] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=22] [MEMORY] tenant: 1, limit: 2147483648 hold: 1008979968 cache_hold: 56623104 cache_used: 56623104 cache_item_count: 27 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 444993536 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 230686720 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 1163264 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 483328 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 53714944 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 1458176 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 14204928 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 111149056 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35651584 limit= 429496720 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 8003584 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= PS_CACHE_CTX_ID hold_bytes= 65536 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 8536064 limit= 9223372036854775807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 3481600 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3964928 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 212992 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 884736 limit= 9223372036854775807 [2025-08-14 09:02:07.747807] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=11] PNIO [ratelimit] time: 1755162127747805, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007911, add_bytes: 0 [2025-08-14 09:02:07.747871] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:07.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.747918] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891968390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.747944] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ac790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.747959] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891991430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196ce10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.749934] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:07.749954] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:07.752674] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:06.737294] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [950][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F32-0-0] [lt=996751][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1002, req={req_server:"172.77.0.7:2882", version:{val:1755161758221798000, v:0}, valid_part_count:0, total_part_count:2, generate_timestamp:1755162121718595}, twrs={inited:true, tenant_id:1002, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162124742888}, server_version_for_stat:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162124742888}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:07.754650] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=36166] current monitor number(seq_=-1) [2025-08-14 09:02:07.757934] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:07.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.758713] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919aac90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.758743] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b9510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.758759] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ad090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.758782] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e62b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891972910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.758797] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73940b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7ee690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.760288] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:07.760357] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=14] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:07.760383] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=9] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:07.760744] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:07.760766] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:02:07.760786] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:07.760800] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:07.760819] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:02:07.760837] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:07.735283] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=23] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:07.760596] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:07.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.761301] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:07.761699] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:07.761714] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=14] temporary file module use no blocks [2025-08-14 09:02:07.761723] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:07.761737] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:07.761753] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=16] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623029179849, flush_io_finished_ret_:0, flush_io_finished_round_:26294, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:06.737917] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [979][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F33-0-0] [lt=24][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1002, req={req_server:"172.77.0.7:2882", version:{val:1755161758221798000, v:0}, valid_part_count:0, total_part_count:2, generate_timestamp:1755162121818668}, twrs={inited:true, tenant_id:1002, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162124742888}, server_version_for_stat:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162124742888}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:06.737917] EDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [979][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F33-0-0] [lt=24][errcode=-4341] LOGGER COST TOO MUCH TIME, cost: 1022552, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1022542, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=7], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1c03240b 0x1c031da7 0x1c03844f 0x984d055 0x9280839 0x926c154 0x12c5a0c9 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:07.745848] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:07.762253] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16378][errcode=0] epoll handler too much time :16412 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:07.761005] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=25677] temporary file module use no blocks [2025-08-14 09:02:07.762418] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=1410] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:07.762418] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7584] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:07.762431] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:07.762444] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623029174762, flush_io_finished_ret_:0, flush_io_finished_round_:26305, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:07.760849] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:07.762497] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=1633][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:07.762515] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:07.762531] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:07.762545] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:07.762716] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:07.762731] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:07.762746] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:02:07.718353] INFO [STORAGE] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:123) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26] wead read timestamp is too old(timestamp={val:1755161749741193000, v:0}, gts_scn={val:1755161762721798002, v:0}, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1755161749741193000, v:0}}) [2025-08-14 09:02:07.764086] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=44589] get_max_applied_scn(scn={val:1755161744727824001, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:49, palf_committed_end_lsn_:{lsn:7323686009}, palf_committed_end_scn_:{val:1755161747742105001, v:0}, last_check_scn_:{val:1755161749741193000, v:0}, max_applied_cb_scn_:{val:1755161744727824001, v:0}}) [2025-08-14 09:02:07.764118] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27] get_min_unreplayed_log_info(lsn={lsn:7323685240}, scn={val:1755100946796590002, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873485, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:8946764, ret_code_:0}}, next_to_submit_lsn_:{lsn:7323685240}, next_to_submit_scn_:{val:1755100946796590002, v:0}, base_lsn_:{lsn:7180210176}, base_scn_:{val:1753840803166611000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:7323685240}, end_lsn:{lsn:7323685240}, read_buf:{buf_len_:3747840, buf_:0x7f98f7a05000}, block_size:67104768, log_storage_:0x7f9901131bf0, log_storage_:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:28, read_io_size:98508800, read_disk_cost_ts:645339, start_lsn:{lsn:7180210176}}}}}}) [2025-08-14 09:02:07.764752] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=72] get wrs ts(ls_id={id:1001}, delta=383036642, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:06.770579] WDIAG [SERVER] update_tenant_server_cache (ob_dbms_sched_job_master.cpp:582) [742][T1001_DBMSSched][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4012] failed to get tenant unit(ret=-4012, ret="OB_TIMEOUT", tenant_id_=1001) [2025-08-14 09:02:07.763374] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:288) [732][T1001_ArcFetche][T1001][YB42AC4D0007-00063C495CA99C16-0-0] [lt=960727] ObArchiveFetcher is running(thread_index=0) [2025-08-14 09:02:07.763396] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:233) [827][T1002_RFLWorker][T1002][YB42AC4D0007-00063C495D099C16-0-0] [lt=17006] ObRemoteFetchWorker is running(thread_index=0) [2025-08-14 09:02:07.769441] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7eeb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.769475] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73936b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7eec90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.769489] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f0c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.764769] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:07.777932] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:07.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.779517] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ac010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.779552] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72765b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b5e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.779568] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ac610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.779579] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891969410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.779604] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f0d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.783530] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:07.789590] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919aab10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.774, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.789624] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739ef90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.789638] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a45b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7283390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.789652] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7eee10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.789667] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73931b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a1e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.799637] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ac490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.799670] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196c210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.799685] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7273c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196c090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.799711] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d27b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72eb290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.799726] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d94b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.799751] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7283510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.780, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.799764] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739bbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728b990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.780, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.799778] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739fa10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.799790] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f727dc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f729ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.799794] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] [sql nio session](*s={this:0x7f98f72d0030, session_id:3221589415, trace_id:YB42AC4D0007-00063C4965FA43DC-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:245}, err:0, last_decode_time:1755161751714366, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1662, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:07.799802] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f727d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f729af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.799821] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=26] [sql nio session](*s={this:0x7f98f72d5030, session_id:3221590200, trace_id:YB42AC4D0007-00063C4965FA43E0-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:248}, err:0, last_decode_time:1755161776777167, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1593, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:07.799823] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891921c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.808130] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:07.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.809701] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891814fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7292d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.793, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.809734] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891921b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.819428] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c0b90, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571c0e70, time_wheel:0x7f9959fef990, thread_pool:0x7f9959fef170, is_running:1, time_interval:3.00s, expected_run_time:09:02:07.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.819779] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7facb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7281710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:07.821728] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:02:07.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.828408] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=54][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:07.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.859331] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=18] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:02:07.861337] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:07.928439] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:07.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.959326] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=22825][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:07.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.961409] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:07.990820] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:07.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:07.992627] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=426][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:07.989, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.000813] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199fbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919acc10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:07.995, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:08.012731] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:08.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.028503] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:07.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.029468] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:08.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.059411] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:07.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.061456] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:08.093258] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:08.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.128569] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:08.552, task_priority:1, with_handle_protected:1}) [472]OB_MALLOC COST TOO MUCH TIME, cost_time=593969, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=593962], seq:[0] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=563166, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=563165], seq:[0] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=561442, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=561432], seq:[0] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=508965, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=508959], seq:[0] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=508988, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=508986], seq:[0, 1] [2025-08-14 09:02:08.214025] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=54][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:08.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.723303] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [314][pnio1][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755161989955997", warnings:[]}, tenant_id_=1, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:02:08.723828] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=509026][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:08.214|207=509.80ms|210=0us|222=2us|total=509.84ms) [2025-08-14 09:02:08.723862] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-4389] timer task use too much time(end=1755162128723862, start=1755162128214022, delta=509840) [2025-08-14 09:02:08.723874] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=484727, expected_run_ts_us_=1755162128239145, ObClockGenerator::getRealClock()=1755162128723873, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:08.239, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.723914] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:08.239, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.723935] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=0] task scheduled too delay(scheduled_delay=468447, expected_run_ts_us_=1755162128255487, ObClockGenerator::getRealClock()=1755162128723935, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:08.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.723951] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:08.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.723969] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=134652, expected_run_ts_us_=1755162128589316, ObClockGenerator::getRealClock()=1755162128723969, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:08.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.723987] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:08.589, task_priority:1, with_handle_protected:1}) [770]OB_MALLOC COST TOO MUCH TIME, cost_time=494626, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=494612], seq:[0] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=494864, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=494862], seq:[0, 1] [2025-08-14 09:02:08.228631] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=52][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:08.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.724298] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=494915][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:08.228|207=495.67ms|210=0us|222=2us|total=495.68ms) [2025-08-14 09:02:08.724327] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4389] timer task use too much time(end=1755162128724327, start=1755162128228629, delta=495698) [390]OB_MALLOC COST TOO MUCH TIME, cost_time=492922, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=492917], seq:[0] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=492935, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=492934], seq:[0, 1] [1031]OB_MALLOC COST TOO MUCH TIME, cost_time=489037, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=489031], seq:[0] [1031]OB_MALLOC COST TOO MUCH TIME, cost_time=489048, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=489047], seq:[0, 1] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=481602, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=481597], seq:[0] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=481613, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=481611], seq:[0, 1] [2025-08-14 09:02:08.230673] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] EASY SLOW: thread: 7f995447d300, lastrun: 1755162127.769941 cost: 0.460729 loop:26502, slowcnt: 4 [2025-08-14 09:02:08.234591] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:08.724412] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=489078] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3844) [857]OB_MALLOC COST TOO MUCH TIME, cost_time=955741, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=962147, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=962145], seq:[0] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=962176, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=962170], seq:[0, 1] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=572025, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=561452], seq:[0, 1] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=604856, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=593991], seq:[0, 1] [2025-08-14 09:02:04.722884] EDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4005609, owner: ObLog, click_count: 3, time dist:[FORMAT_END=4005604, ALLOC_END=5, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0xfc0e73d 0x993f560 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [372]OB_MALLOC COST TOO MUCH TIME, cost_time=471390, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=471383], seq:[0] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=471418, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=471415], seq:[0, 1] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=463758, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=463753], seq:[0] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=463771, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=463771], seq:[0, 1] [683]OB_MALLOC COST TOO MUCH TIME, cost_time=279600, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=279594], seq:[0] [683]OB_MALLOC COST TOO MUCH TIME, cost_time=279607, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=279606], seq:[0, 1] [797]OB_MALLOC COST TOO MUCH TIME, cost_time=214196, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=214186, ALLOC_CHUNK_END=4], seq:[0, 1] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=587052, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=563172], seq:[0, 1] [797]OB_MALLOC COST TOO MUCH TIME, cost_time=231216, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=214227, WASH_BLOCK_END=135, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=16848], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:07.737880] EDIAG [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:355) [835][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1021617, owner: ObLog, click_count: 9, time dist:[ALLOC_BLOCK#start=23875, ALLOC_BLOCK#end=1, FORMAT_END=959934, ObMalloc#start=1, ObMalloc#end=37803, ALLOC_END=2, APPEND_END=1, ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=2], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1de5c20c 0x1de57e2a 0x1ded7f7a 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:08.724425] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:08.724450] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=11] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:08.724471] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=21] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3845) [2025-08-14 09:02:08.724480] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:08.724670] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:08.724682] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=11] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3846) [2025-08-14 09:02:08.724689] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:08.242357] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:08.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.724736] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=481647][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:08.242|207=482.38ms|210=1us|222=2us|total=482.40ms) [2025-08-14 09:02:08.724760] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4389] timer task use too much time(end=1755162128724759, start=1755162128242355, delta=482404) [2025-08-14 09:02:08.724772] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=480138, expected_run_ts_us_=1755162128244633, ObClockGenerator::getRealClock()=1755162128724772, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:08.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.724790] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:08.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.727827] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [317][pnio1][T1002][YB42AC4D0007-00063C49612B8F33-0-0] [lt=8][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1002, delta=9989469, last_succ_cluster_heartbeat_tstamp_=1755162118738357) [2025-08-14 09:02:08.727845] WDIAG [RPC] handle_resp (ob_poc_rpc_proxy.cpp:192) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] async_cb handler cost too much time(total_time=959494, decode_time=959473, process_time=18, session_destroy_time=3, ret=0, pcode=1815) [2025-08-14 09:02:08.727859] WDIAG stat_cleanup (debug.c:35) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] PNIO [delay_warn] cost too much time: 959006us, procedure: pn_pktc_resp_cb [2025-08-14 09:02:08.727869] WDIAG eloop_delay_warn (debug.h:93) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] PNIO [delay_warn] eloop handle events delay high: 959006, malloc=0/0 write=0/0 read=0/1 server_process=0/0 client_cb=959006/1 [2025-08-14 09:02:07.769951] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:08.733600] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=962192][errcode=0] epoll handler too much time :963657 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:08.128606] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:08.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.734412] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=604886][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:08.128|207=605.80ms|210=0us|222=2us|total=605.83ms) [2025-08-14 09:02:08.734438] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4389] timer task use too much time(end=1755162128734438, start=1755162128128605, delta=605833) [2025-08-14 09:02:04.722884] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:8607757, max_lsn:1604090331, max_scn:{val:1755161756730847001, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:1556737504}, reuse_lsn:{lsn:1604090331}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:1562147291}}, last_submit_lsn:{lsn:1604090203}, last_submit_end_lsn:{lsn:1604090331}, last_submit_log_id:8607757, last_submit_log_pid:50, max_flushed_lsn:{lsn:1604090203}, max_flushed_end_lsn:{lsn:1604090331}, max_flushed_log_pid:50, committed_end_lsn:{lsn:1604090090}, last_slide_log_id:8607755, last_slide_scn:{val:1755161749741193000, v:0}, last_slide_lsn:{lsn:1604089968}, last_slide_end_lsn:{lsn:1604090090}, last_slide_log_pid:50, last_slide_log_accum_checksum:1446576759, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f990bfe4530}, [StateMgr]={this:0x7f990bfe89b0, self:"172.77.0.7:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"PENDING", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:51}, leader:"0.0.0.0:0", leader_epoch:-1, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:1604090331}, scan_disk_log_finished:true, last_check_start_id:8607756, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161901828039, palf_role_change_cb:0x7f990bfef470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:50, prev:{version:1, config:{config_version:{proposal_id:49, config_seq:45}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:50, config_seq:46}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_mode_pid:50}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_end_lsn:{lsn:1604088821}, prev_mode_pid:50}, reconfig_barrier:{prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_end_lsn:{lsn:1604088821}, prev_mode_pid:50}, persistent_config_version:{proposal_id:50, config_seq:46}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755161738854289, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755161738854289, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f990bfe5c70}, [ModeMgr]={palf_id:1, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:50, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:1543409664}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:51}, log_config_meta:{version:3, proposal_id:50, prev:{version:1, config:{config_version:{proposal_id:49, config_seq:45}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:50, config_seq:46}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_mode_pid:50}, log_snapshot_meta:{version:2, base_lsn:{lsn:1543409664}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:712704}, readable_log_tail:{lsn:712704}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:1610514432}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/meta", dir_fd:225, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66392064, block_header_serialize_buf_:0x7f990bfeabb8, flashback_version:0}, log_storage:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1755161901828095, this:0x7f990bfe90f0}, [Reconfirm]={palf_id:1, self:"172.77.0.7:2882", state:"RECONFIRM_MODE_META", new_proposal_id:51, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"172.77.0.7:2882", majority_max_accept_pid:50, majority_max_lsn:{lsn:1604090331}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:1755161901828373, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f990bfe8b30}) [2025-08-14 09:02:08.734589] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=4005609] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161974728707, last_flushed_end_lsn:{lsn:1604090331}}]) [2025-08-14 09:02:08.734607] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13] current log_task status(palf_id=1, self="172.77.0.7:2882", log_id=8607756, log_task={header:{begin_lsn:{lsn:1604090090}, end_lsn:{lsn:1604090203}, log_id:8607756, min_scn:{val:1755161749741193001, v:0}, max_scn:{val:1755161749741193001, v:0}, data_len:57, proposal_id:50, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:1604090090}, data_checksum:2031283194, accum_checksum:759186326, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1755161750716165, freeze_ts:1755161750716173, submit_ts:1755161750716183, flushed_ts:1755161853765770, gen_to_freeze cost time:8, gen_to_submit cost time:18, submit_to_flush cost time:103049587}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161974728707, last_flushed_end_lsn:{lsn:1604090331}}]) [2025-08-14 09:02:08.262918] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:08.735082] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=471443][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:02:08.735102] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:08.735110] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:08.735128] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:02:08.735142] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:08.735152] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:08.735168] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:08.735179] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:08.735199] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:08.735209] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:08.735395] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:08.735407] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:08.735417] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:02:08.271021] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:08.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.735532] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=463806][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:08.271|207=464.51ms|210=1us|222=2us|total=464.52ms) [2025-08-14 09:02:08.735545] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task use too much time(end=1755162128735545, start=1755162128271019, delta=464526) [2025-08-14 09:02:08.735554] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=0] task scheduled too delay(scheduled_delay=443204, expected_run_ts_us_=1755162128292349, ObClockGenerator::getRealClock()=1755162128735554, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:08.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.735568] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:08.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.735603] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=26] gc stale ls task succ [2025-08-14 09:02:08.455555] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fa190, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa470, time_wheel:0x7f99341cdbf0, thread_pool:0x7f99341cd3d0, is_running:1, time_interval:2.00s, expected_run_time:09:02:08.334, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.735648] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=279668][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:08.455|207=280.09ms|210=0us|222=1us|total=280.11ms) [2025-08-14 09:02:08.735660] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task use too much time(end=1755162128735659, start=1755162128455551, delta=280108) [2025-08-14 09:02:08.741395] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:08.743772] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:08.745293] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:02:08.745375] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:08.747209] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:08.747249] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=12] gc stale ls task succ [2025-08-14 09:02:08.747319] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:08.747353] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:08.159474] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:08.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.747466] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=587100][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:08.159|207=587.99ms|210=0us|222=1us|total=588.01ms) [2025-08-14 09:02:08.747489] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4389] timer task use too much time(end=1755162128747488, start=1755162128159472, delta=588016) [2025-08-14 09:02:08.747524] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8030, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8310, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:10.00s, expected_run_time:09:02:08.730, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.747545] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:08.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.749913] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:08.752309] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EEA3-0-0] [lt=17][errcode=0] cost too much time:mds_tenant_service.cpp:int oceanbase::storage::mds::ObTenantMdsTimer::try_gc_mds_table_(oceanbase::storage::ObTablet &), (this=|threshold=5.00ms|start at 09:02:08.736|394=0us|total=16.09ms) [2025-08-14 09:02:08.752330] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EEA3-0-0] [lt=19][errcode=0] cost too much time:mds_tenant_service.cpp:int oceanbase::storage::mds::ObTenantMdsTimer::process_with_tablet_(oceanbase::storage::ObTablet &), (this=|threshold=10.00ms|start at 09:02:08.736|315=0us|317=1us|319=1us|total=16.11ms) [2025-08-14 09:02:08.752461] WDIAG [RPC] handle_resp (ob_poc_rpc_proxy.cpp:192) [314][pnio1][T0][Y0-0000000000000000-0-0] [lt=541][errcode=0] async_cb handler cost too much time(total_time=1986136, decode_time=-1755162126766323, process_time=0, session_destroy_time=1755162128752459, ret=0, pcode=1815) [2025-08-14 09:02:08.752476] WDIAG stat_cleanup (debug.c:35) [314][pnio1][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] PNIO [delay_warn] cost too much time: 1986013us, procedure: pn_pktc_resp_cb [2025-08-14 09:02:08.752485] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [314][pnio1][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559440, sock=0x7f995f412a58, code=-47 [2025-08-14 09:02:08.754845] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EEA3-0-0] [lt=18] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-08-14 09:02:08.751735] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=5008759] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:08.751746] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=5009831] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:08.757586] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:08.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.759378] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=12] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:08.755325] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:08.752491] WDIAG delay_warn (debug.h:103) [314][pnio1][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] PNIO [delay_warn] pktc_flush_cb delay high: 3988027, start_us=1755162124763889 [2025-08-14 09:02:08.760408] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [314][pnio1][T1002][Y0-0000000000000000-0-0] [lt=7883][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1002, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755161990359407", warnings:[]}, tenant_id_=1002, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:02:08.760433] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [314][pnio1][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559450, sock=0x7f995f412a58, code=-47 [2025-08-14 09:02:08.760442] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [314][pnio1][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755161991047636", warnings:[]}, tenant_id_=1, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:02:08.760453] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [314][pnio1][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559458, sock=0x7f995f412a58, code=-47 [2025-08-14 09:02:08.760459] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [314][pnio1][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755161992722735", warnings:[]}, tenant_id_=1, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:02:08.760467] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [314][pnio1][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559466, sock=0x7f995f412a58, code=-47 [2025-08-14 09:02:08.760474] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [314][pnio1][T1002][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1002, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755161993929568", warnings:[]}, tenant_id_=1002, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:02:08.760482] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [314][pnio1][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559474, sock=0x7f995f412a58, code=-47 [2025-08-14 09:02:08.760489] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [314][pnio1][T1002][Y0-0000000000000000-0-0] [lt=4][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1002, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755161994331772", warnings:[]}, tenant_id_=1002, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:02:08.760516] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=15] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:08.760552] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=3186] tx gc loop thread is running(MTL_ID()=1001) [2025-08-14 09:02:08.760564] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=11] try gc retain ctx [2025-08-14 09:02:08.760569] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=5] try to retry start_working [2025-08-14 09:02:08.528978] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fe410, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe6f0, time_wheel:0x7f9922df9d40, thread_pool:0x7f9922df9520, is_running:1, time_interval:2.00s, expected_run_time:09:02:08.432, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.760605] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=231273][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:08.528|207=231.63ms|210=0us|222=2us|total=231.65ms) [2025-08-14 09:02:08.760622] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] timer task use too much time(end=1755162128760621, start=1755162128528975, delta=231646) [2025-08-14 09:02:08.721852] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:08.751120] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=4338] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:02:08.760652] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=38740][errcode=0] epoll handler too much time :38809 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:08.754702] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=4451] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:02:07.762008] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:08.759762] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=981011] [COMPACTION DAG STATUS] (start_time=1755162116772697, end_time=1755162128754923, dag_status={}) [2025-08-14 09:02:08.760683] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=997165] get number(ret=-4038, service_type_=0, range=1, base_id=1755162127761191000, start_id=0, end_id=140295968341544) [2025-08-14 09:02:08.760702] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:08.760497] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [314][pnio1][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559490, sock=0x7f995f412a58, code=-47 [2025-08-14 09:02:08.760714] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [314][pnio1][T1002][Y0-0000000000000000-0-0] [lt=214][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1002, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755161995244347", warnings:[]}, tenant_id_=1002, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:02:08.760726] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [314][pnio1][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559513, sock=0x7f995f412a58, code=-47 [2025-08-14 09:02:08.760719] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162127259499], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:08.760734] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [314][pnio1][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559497, sock=0x7f995f412a58, code=-47 [2025-08-14 09:02:08.760735] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162127259499], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:08.760740] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [314][pnio1][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755161999746075", warnings:[]}, tenant_id_=1, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:02:08.760749] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [314][pnio1][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559503, sock=0x7f995f412a58, code=-47 [2025-08-14 09:02:08.760748] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162127259499]}) [2025-08-14 09:02:08.760755] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [314][pnio1][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162001976784", warnings:[]}, tenant_id_=1, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:02:08.760759] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1002}) [2025-08-14 09:02:08.760765] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [314][pnio1][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559547, sock=0x7f995f412a58, code=-47 [2025-08-14 09:02:08.760768] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1002}) [2025-08-14 09:02:07.737880] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:355) [835][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=14] finish check and clear obj lock(ls->get_ls_id()={id:1001}) [2025-08-14 09:02:08.761126] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [835][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1021617][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99319fe030, caller:ob_table_lock_service.cpp:start:276, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE6$_1273, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe310, time_wheel:0x7f99535f3990, thread_pool:0x7f99535f3170, is_running:0, time_interval:10.00s, expected_run_time:09:02:16.426, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.761166] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [835][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:02:08.761|total=2.17s) [2025-08-14 09:02:08.720589] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:08.723420] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3122] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=579980665, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=253042, succ_cluster_heartbeat_count=2186, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161000606550000, v:0}, local_cluster_delta=1127112631, force_self_check=false, weak_read_refresh_interval=100000) [768]OB_MALLOC COST TOO MUCH TIME, cost_time=121454, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=121449], seq:[0] [768]OB_MALLOC COST TOO MUCH TIME, cost_time=121468, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=121467], seq:[0, 1] [2025-08-14 09:02:08.161468] EDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] LOGGER COST TOO MUCH TIME, cost: 1551934, owner: ObLog, click_count: 5, time dist:[FORMAT_END=14, ObMalloc#start=1, ObMalloc#end=572037, ALLOC_END=979881, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b52e837 0x996d04c 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:08.644094] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1070173, owner: ObLog, click_count: 5, time dist:[FORMAT_END=32, ObMalloc#start=2, ObMalloc#end=121475, ALLOC_END=948663, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [381]OB_MALLOC COST TOO MUCH TIME, cost_time=460150, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=41, WASH_BLOCK_END=460108], seq:[0, 1, 2] [1031]OB_MALLOC COST TOO MUCH TIME, cost_time=509790, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=64, WASH_BLOCK_END=493554, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=16168], seq:[0, 1, 2, 3, 4] [784]OB_MALLOC COST TOO MUCH TIME, cost_time=975638, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=975630], seq:[0] [2025-08-14 09:02:08.761639] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=3183] tx gc loop thread is running(MTL_ID()=1) [2025-08-14 09:02:08.761649] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] try gc retain ctx [2025-08-14 09:02:08.760317] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=35] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 444993536 used= 389633824 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 70 washed_blocks= 287 washed_size= 70221824 [MEMORY] request_cached_chunk_cnt= 4 [MEMORY] hold= 389633824 used= 379387395 count= 32872 avg_used= 11541 mod=SUMMARY [2025-08-14 09:02:08.762199] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=1859] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 230686720 used= 228884480 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 block_cnt= 110 chunk_cnt= 110 mod=Memstore [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:02:08.762211] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= EXECUTE_CTX_ID hold= 1163264 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 15 washed_size= 13516800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:08.727248] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=5199] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119812272, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-08-14 09:02:08.762557] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=35236] get wrs ts(ls_id={id:1}, delta=379020024, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:08.762573] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:08.762744] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=22] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:08.762765] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] temporary file module use no blocks [2025-08-14 09:02:08.762773] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:08.762785] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:08.762798] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623030202234, flush_io_finished_ret_:0, flush_io_finished_round_:26306, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:08.763183] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25] get wrs ts(ls_id={id:1001}, delta=384035008, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:08.763194] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:08.727390] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728bb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:08.718, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:08.765659] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=38206][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180fbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739fe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:08.719, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:08.765686] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b9b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:08.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:08.765698] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891973990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:08.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:08.765708] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728b090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:08.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:08.765717] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918105b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a0490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:08.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:08.763215] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=370541415, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=251638, succ_cluster_heartbeat_count=1480, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161006924036004, v:0}, local_cluster_delta=1121839177, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:02:08.765854] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=2620] get wrs ts(ls_id={id:1}, delta=379021639, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:08.765875] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:08.766401] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1001}, delta=384035008, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:08.766410] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:08.766429] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8] get wrs ts(ls_id={id:1}, delta=379021639, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:08.744296] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc190, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc470, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:10.00s, expected_run_time:09:02:08.720, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.745406] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=20] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:08.768125] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=23789][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:08.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.768150] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:08.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.745461] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:08.751022] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=27] PNIO [ratelimit] time: 1755162128751020, bytes: 0, bw: 0.000000 MB/s, add_ts: 1003215, add_bytes: 0 [2025-08-14 09:02:08.751623] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:08.768363] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16713][errcode=0] epoll handler too much time :16751 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:08.723244] WDIAG [SERVER] scheduler (ob_dbms_sched_job_master.cpp:175) [742][T1001_DBMSSched][T1001][Y0-0000000000000000-0-0] [lt=995955][errcode=0] fail to update server(tmp_ret=-4012) [2025-08-14 09:02:08.753354] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:08.769350] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15971][errcode=0] epoll handler too much time :16005 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:08.754082] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [409][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] [sql nio session](*s={this:0x7f98f7334030, session_id:3221557375, trace_id:YB42AC4D0007-00063C4962CA3040-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:262}, err:0, last_decode_time:1755161121788142, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:2389, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:08.769380] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [409][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15267] [sql nio session](*s={this:0x7f98f7333030, session_id:3221589989, trace_id:YB42AC4D0007-00063C4962CA3068-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:273}, err:0, last_decode_time:1755161766744050, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1903, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:08.769397] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [409][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] [sql nio session](*s={this:0x7f98f7339030, session_id:3221590023, trace_id:YB42AC4D0007-00063C4962CA3069-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:254}, err:0, last_decode_time:1755161784744265, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1683, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:08.770341] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:08.775747] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891972310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:08.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:08.775771] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ed990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:08.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:08.775781] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891815730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7292a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:08.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:08.785795] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72eca90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:08.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:08.785819] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e94b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739e810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:08.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:08.785829] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739e690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:08.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:08.785838] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180ea30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739f410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:08.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:08.795851] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ea510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:08.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:08.795875] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891825990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:08.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:08.795885] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196be10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:08.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:08.795893] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728d430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7290f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:08.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:08.805921] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a1690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:08.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:08.805943] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73922b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a1510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:08.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:08.805952] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a1390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:08.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:08.805961] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f73a1210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:08.799, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:08.805970] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7289890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:08.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:08.815990] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891973e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:08.809, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:08.816013] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eaa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7ef410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:08.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:08.846173] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73945b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f1c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:08.837, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:08.926720] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f1390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:08.911, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:08.934530] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:08.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.947635] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:08.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.999611] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:08.989, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.019733] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:09.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.034590] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:08.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.047697] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:08.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.090210] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:09.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.134611] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:09.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.147770] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=79][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:09.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.210892] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:09.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.234643] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:09.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.241073] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:09.239, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.247834] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:09.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.261249] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:09.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.334709] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:09.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.347898] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:09.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.434756] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:09.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.447983] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:09.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.492649] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:09.489, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.534817] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:09.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.548055] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:09.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.634881] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:09.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.648122] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:09.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.761654] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=4] try to retry start_working [2025-08-14 09:02:08.759567] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=983409] [COMPACTION DAG STATUS] (start_time=1755162115726171, end_time=1755162128754959, dag_status={}) [2025-08-14 09:02:08.760290] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1004629] [COMPACTION DAG STATUS] (start_time=1755162116776349, end_time=1755162128754972, dag_status={}) [2025-08-14 09:02:08.762323] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=38080] get wrs ts(ls_id={id:1}, delta=581022701, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:08.763868] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:08.161468] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:09.715871] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1551934][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:08.644094] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:08.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.716043] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] get_fetch_log_dst failed or invalid(ret=0, palf_id=1001, self="172.77.0.7:2882", fetch_log_dst="172.77.0.7:2882") [2025-08-14 09:02:08.747377] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:09.716064] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=967223][errcode=0] epoll handler too much time :968693 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:08.769407] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [409][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] [sql nio session](*s={this:0x7f98f733d030, session_id:3221590940, trace_id:YB42AC4D0007-00063C4962CA306E-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:247}, err:0, last_decode_time:1755162096770219, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1286, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:08.788293] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:09.716180] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=926478][errcode=0] epoll handler too much time :927904 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:08.824402] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:08.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.859953] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=17] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:02:08.992009] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:08.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.235574] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:08.762222] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:02:09.716533] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=952849] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 53714944 used= 47487424 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 84 washed_blocks= 334 washed_size= 126640128 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 47487424 used= 43098388 count= 3994 avg_used= 10790 mod=SUMMARY [2025-08-14 09:02:09.716548] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= WORK_AREA hold= 1458176 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 13 washed_blocks= 66 washed_size= 25804800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:09.716562] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 14204928 used= 10970224 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 277 washed_size= 31932416 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10970224 used= 6078924 count= 18573 avg_used= 327 mod=SUMMARY [2025-08-14 09:02:09.716576] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 111149056 used= 108896256 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 block_cnt= 211 chunk_cnt= 53 mod=CoStack [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 mod=SUMMARY [2025-08-14 09:02:08.746292] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:08.772360] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [872][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E799C16-0-0] [lt=942005] ObArchiveSender is running(thread_index=1) [2025-08-14 09:02:08.822245] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:02:08.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.307969] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=959050][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162128.788275 cost: 0.519691 loop:26473, slowcnt: 5 [2025-08-14 09:02:09.700091] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [556][T1_OBJLockGC][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fafd0, caller:ob_table_lock_service.cpp:start:276, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE6$_1273, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fb2b0, time_wheel:0x7f994b19db50, thread_pool:0x7f994b19d330, is_running:1, time_interval:10.00s, expected_run_time:09:02:09.696, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.715677] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=951909] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:09.715910] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=1070173][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:08.644|207=1.07s|210=0us|222=2us|total=1.07s) [2025-08-14 09:02:09.718912] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=2996][errcode=-4389] timer task use too much time(end=1755162129718911, start=1755162128644091, delta=1074820) [2025-08-14 09:02:09.716201] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=890449][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:08.824|207=891.80ms|210=0us|222=1us|total=891.81ms) [2025-08-14 09:02:09.718953] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=2748][errcode=-4389] timer task use too much time(end=1755162129718953, start=1755162128824400, delta=894553) [2025-08-14 09:02:09.716252] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=723146][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:08.992|207=724.24ms|210=0us|222=2us|total=724.27ms) [2025-08-14 09:02:09.719064] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=2806][errcode=-4389] timer task use too much time(end=1755162129719063, start=1755162128992006, delta=727057) [2025-08-14 09:02:09.719078] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=474443, expected_run_ts_us_=1755162129244633, ObClockGenerator::getRealClock()=1755162129719077, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:09.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.719107] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:09.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.720473] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2722] get wrs ts(ls_id={id:1}, delta=581976671, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:09.720496] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:09.720825] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B923E-0-0] [lt=27][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162129719805) [2025-08-14 09:02:09.720850] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B923E-0-0] [lt=18][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162129719805}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162129719856}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162129719856}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:09.715707] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=950399][errcode=0] epoll handler too much time :951846 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:09.721489] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=581976671, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:09.721505] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:09.718056] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=894453][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:08.822|207=895.81ms|210=0us|222=2us|total=895.84ms) [2025-08-14 09:02:09.716609] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=22] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 56623104 used= 56623104 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 56623104 used= 0 count= 27 avg_used= 0 mod=SUMMARY [2025-08-14 09:02:09.512755] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:09.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.724160] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7523] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 34365376 limit= 429496720 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 34365376 used= 31086608 count= 475 avg_used= 65445 mod=SUMMARY [2025-08-14 09:02:09.724158] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=211079][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:09.512|207=211.40ms|210=0us|222=1us|total=211.42ms) [2025-08-14 09:02:09.724173] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] timer task use too much time(end=1755162129724172, start=1755162129512753, delta=211419) [2025-08-14 09:02:09.724176] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 8003584 used= 4947392 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 10 washed_blocks= 23 washed_size= 12967936 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4947392 used= 4793344 count= 604 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:09.724189] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:09.724182] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=134865, expected_run_ts_us_=1755162129589316, ObClockGenerator::getRealClock()=1755162129724182, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:09.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.724196] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:09.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.724201] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 65536 used= 31024 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 2031616 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 31024 used= 27058 count= 20 avg_used= 1352 mod=SUMMARY [2025-08-14 09:02:09.724213] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= RPC_CTX_ID hold= 8536064 used= 2187264 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2187264 used= 1438080 count= 93 avg_used= 15463 mod=SUMMARY [2025-08-14 09:02:09.724228] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 3481600 used= 2391232 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 10 washed_size= 4907008 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2391232 used= 2367040 count= 36 avg_used= 65751 mod=SUMMARY [2025-08-14 09:02:09.724790] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:02:09.725737] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [316][pnio1][T1][YB42AC4D0007-00063C49607B923E-0-0] [lt=58][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1, delta=10996728, last_succ_cluster_heartbeat_tstamp_=1755162118729009) [2025-08-14 09:02:09.724256] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=19] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 3964928 used= 3080192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 3 washed_size= 229376 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3080192 used= 2983936 count= 376 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:09.726710] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=2433] [MEMORY] tenant_id= 1 ctx_id= SCHEMA_SERVICE hold= 212992 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1884160 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:09.726725] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:09.726736] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= MERGE_NORMAL_CTX_ID hold= 884736 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 6 washed_blocks= 9 washed_size= 11698176 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:09.724210] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:09.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.726502] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:08.775822] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:08.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.735031] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=957756][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:08.775|207=959.21ms|210=0us|222=1us|total=959.22ms) [2025-08-14 09:02:09.735043] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task use too much time(end=1755162129735043, start=1755162128775820, delta=959223) [2025-08-14 09:02:09.735051] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=0] task scheduled too delay(scheduled_delay=942701, expected_run_ts_us_=1755162128792349, ObClockGenerator::getRealClock()=1755162129735050, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:08.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.735063] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:08.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.735073] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=932079, expected_run_ts_us_=1755162128802994, ObClockGenerator::getRealClock()=1755162129735073, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40b7f0, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40bad0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:10.00s, expected_run_time:09:02:08.802, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.735083] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40b7f0, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40bad0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:10.00s, expected_run_time:09:02:08.802, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.224774] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:09.724052] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=5985][errcode=-4389] timer task use too much time(end=1755162129724051, start=1755162128822243, delta=901808) [2025-08-14 09:02:09.735121] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1001, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:8571658, max_lsn:7323687369, max_scn:{val:1755161749741193002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:7278610047}, reuse_lsn:{lsn:7323686131}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:7281744329}}, last_submit_lsn:{lsn:7323686131}, last_submit_end_lsn:{lsn:7323687369}, last_submit_log_id:8571658, last_submit_log_pid:49, max_flushed_lsn:{lsn:7323686009}, max_flushed_end_lsn:{lsn:7323686131}, max_flushed_log_pid:49, committed_end_lsn:{lsn:7323686009}, last_slide_log_id:8571656, last_slide_scn:{val:1755161747742105001, v:0}, last_slide_lsn:{lsn:7323685887}, last_slide_end_lsn:{lsn:7323686009}, last_slide_log_pid:49, last_slide_log_accum_checksum:2505458373, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f990112a530}, [StateMgr]={this:0x7f990112e9b0, self:"172.77.0.7:2882", palf_id:1001, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"PENDING", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:49}, leader:"0.0.0.0:0", leader_epoch:-1, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:7323687369}, scan_disk_log_finished:true, last_check_start_id:8571657, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161767744317, palf_role_change_cb:0x7f9901135470, allow_vote:true}, [ConfigMgr]={palf_id:1001, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:49, prev:{version:1, config:{config_version:{proposal_id:48, config_seq:47}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:49, config_seq:48}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_mode_pid:49}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_end_lsn:{lsn:7323685240}, prev_mode_pid:49}, reconfig_barrier:{prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_end_lsn:{lsn:7323685240}, prev_mode_pid:49}, persistent_config_version:{proposal_id:49, config_seq:48}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755161739755312, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755161739755312, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f990112bc70}, [ModeMgr]={palf_id:1001, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, accepted_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1001, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1753069671378640001, v:0}, base_lsn_for_block_gc:{lsn:7247314944}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:49}, log_config_meta:{version:3, proposal_id:49, prev:{version:1, config:{config_version:{proposal_id:48, config_seq:47}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:49, config_seq:48}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_mode_pid:49}, log_snapshot_meta:{version:2, base_lsn:{lsn:7247314944}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1753069671378640000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}}, log_meta_storage:{log_tail:{lsn:765952}, readable_log_tail:{lsn:765952}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7381524480}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/meta", dir_fd:232, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66338816, block_header_serialize_buf_:0x7f9901130bb8, flashback_version:0}, log_storage:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, palf_epoch:2, last_purge_throttling_ts:1755161769777679, this:0x7f990112f0f0}, [Reconfirm]={palf_id:1001, self:"172.77.0.7:2882", state:"WAITING_LOG_FLUSHED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:1755161769777679, this:0x7f990112eb30}) [2025-08-14 09:02:09.726992] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f1e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.718, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:09.735645] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant: 500, limit: 9223372036854775807 hold: 703815680 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 336588800 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 35078144 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 85983232 limit= 9223372036854775807 [MEMORY] ctx_id= LIBEASY hold_bytes= 3145728 limit= 9223372036854775807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 25165824 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 2711552 limit= 9223372036854775807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 37232640 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 78925824 limit= 9223372036854775807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 98983936 limit= 9223372036854775807 [2025-08-14 09:02:09.735697] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=13] gc stale ls task succ [2025-08-14 09:02:09.737054] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8607][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f739fd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.726, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:09.737079] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a67b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7397290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.728, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:09.737597] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:09.737613] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] temporary file module use no blocks [2025-08-14 09:02:09.737619] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=5] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:09.735598] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=470] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [2025-08-14 09:02:09.738633] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=3025] current log_task status(palf_id=1001, self="172.77.0.7:2882", log_id=8571657, log_task={header:{begin_lsn:{lsn:7323686009}, end_lsn:{lsn:7323686131}, log_id:8571657, min_scn:{val:1755161749741193000, v:0}, max_scn:{val:1755161749741193000, v:0}, data_len:66, proposal_id:49, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:7323686009}, data_checksum:2222604892, accum_checksum:1531236791, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1755161749745909, freeze_ts:1755161749745914, submit_ts:1755161749745940, flushed_ts:1755161785758706, gen_to_freeze cost time:5, gen_to_submit cost time:31, submit_to_flush cost time:36012766}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [2025-08-14 09:02:09.738804] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99319fc190, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc470, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:0, time_interval:10.00s, expected_run_time:09:02:18.720, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.738828] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:02:09.738|total=2.98s) [2025-08-14 09:02:09.738840] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] task executed too delay(delay_time=105006844, *p_task_={this:0x7f99319fd110, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd3f0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:0, time_interval:10.00s, expected_run_time:09:02:14.726, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.738984] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=-4016] task executed too delay(delay_time=8740398, *p_task_={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:0, time_interval:250.00ms, expected_run_time:09:02:09.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:08.766436] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:09.737629] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:09.739826] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=193] LogLoopThread round_cost_time(us)(round_cost_time=5979352) [2025-08-14 09:02:09.739850] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=0] get_fetch_log_dst failed or invalid(ret=0, palf_id=1, self="172.77.0.7:2882", fetch_log_dst="172.77.0.7:2882") [2025-08-14 09:02:09.739858] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=7] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:02:09.741070] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=971201] get wrs ts(ls_id={id:1001}, delta=385008946, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:09.741089] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:09.741145] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119812272, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-08-14 09:02:09.741169] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=379995577, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:09.741176] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:09.741743] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8] get wrs ts(ls_id={id:1001}, delta=385008946, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:09.741757] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:09.735681] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=14] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 336588800 used= 300764144 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 495 washed_blocks= 2222 washed_size= 915300352 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 300764144 used= 293377892 count= 12630 avg_used= 23228 mod=SUMMARY [2025-08-14 09:02:09.744903] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9184] [MEMORY] tenant_id= 500 ctx_id= GLIBC hold= 39108608 used= 21963632 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 178 washed_blocks= 809 washed_size= 350961664 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 21963632 used= 10364997 count= 131700 avg_used= 78 mod=SUMMARY [2025-08-14 09:02:09.744921] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 500 ctx_id= CO_STACK hold= 85983232 used= 83091456 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 83091456 used= 82936896 count= 161 avg_used= 515136 block_cnt= 161 chunk_cnt= 41 mod=CoStack [MEMORY] hold= 83091456 used= 82936896 count= 161 avg_used= 515136 mod=SUMMARY [2025-08-14 09:02:09.744933] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 500 ctx_id= LIBEASY hold= 3145728 used= 3070816 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1048576 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3070816 used= 2859784 count= 223 avg_used= 12824 mod=SUMMARY [2025-08-14 09:02:09.744945] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 500 ctx_id= LOGGER_CTX_ID hold= 18874368 used= 6242304 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 6242304 used= 6239232 count= 3 avg_used= 2079744 mod=SUMMARY [2025-08-14 09:02:09.744972] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 500 ctx_id= RPC_CTX_ID hold= 4849664 used= 1286144 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 50 washed_blocks= 63 washed_size= 102105088 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1286144 used= 849280 count= 53 avg_used= 16024 mod=SUMMARY [2025-08-14 09:02:09.746384] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:09.738526] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:09.748145] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=5034][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:09.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.748472] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919d9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b4610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:09.748508] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7277c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72eac90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:09.741877] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:09.744696] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:09.745612] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:09.744984] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 500 ctx_id= PKT_NIO hold= 37232640 used= 35792384 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 42 washed_blocks= 63 washed_size= 82591744 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 35792384 used= 34871392 count= 116 avg_used= 300615 mod=SUMMARY [2025-08-14 09:02:09.748886] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=3867] [MEMORY] tenant_id= 500 ctx_id= SCHEMA_SERVICE hold= 78925824 used= 68904576 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 22 washed_size= 5767168 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 68904576 used= 67642943 count= 4856 avg_used= 13929 mod=SUMMARY [2025-08-14 09:02:09.748902] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 500 ctx_id= UNEXPECTED_IN_500 hold= 98983936 used= 97130832 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 97130832 used= 86748768 count= 3007 avg_used= 28848 mod=SUMMARY [2025-08-14 09:02:09.748917] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 508, limit: 1073741824 hold: 15613952 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 5128192 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10485760 limit= 9223372036854775807 [2025-08-14 09:02:09.748929] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 508 ctx_id= DEFAULT_CTX_ID hold= 5128192 used= 3765328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3765328 used= 3723912 count= 84 avg_used= 44332 mod=SUMMARY [2025-08-14 09:02:09.748951] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 508 ctx_id= CO_STACK hold= 10485760 used= 10321920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10321920 used= 10302720 count= 20 avg_used= 515136 mod=SUMMARY [2025-08-14 09:02:09.748995] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 509, limit: 1073741824 hold: 15613952 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 5128192 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10485760 limit= 9223372036854775807 [784]OB_MALLOC COST TOO MUCH TIME, cost_time=1024490, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=37878, WASH_BLOCK_END=166, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=975656], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:07.761929] EDIAG [RS] get_primary_zone_unit_array (ob_ls_service_helper.cpp:383) [784][T1002_TBalance][T1002][YB42AC4D0007-00063C4964C9A63F-0-0] [lt=24989][errcode=-4012] LOGGER COST TOO MUCH TIME, cost: 1984398, owner: ObLog, click_count: 5, time dist:[FORMAT_END=959893, ObMalloc#start=0, ObMalloc#end=1024503, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x104eaae8 0x104ea576 0x10539f82 0x10531684 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:07.770008] EDIAG [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6175) [14780][T1001_TimerWK4_SSTableGC][T1001][YB42AC4D0007-00063C4974099C4E-0-0] [lt=21][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1976467, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1976466, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x1bb90396 0x975a989 0x975972e 0x9758a1f 0x1aee8055 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:07.770008] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6175) [14780][T1001_TimerWK4_SSTableGC][T1001][YB42AC4D0007-00063C4974099C4E-0-0] [lt=21] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:60349}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1755136957320163000}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1755159231184436000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1755159231184436000) [2025-08-14 09:02:09.725713] INFO [STORAGE.TRANS] refresh_elr_tenant_config_ (ob_tx_elr_util.cpp:42) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=7] refresh tenant config success(tenant_id=1001, *this={last_refresh_ts:1755162129715187, can_tenant_elr:true}) [2025-08-14 09:02:09.749506] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=23747] get_max_applied_scn(scn={val:1755161719775179002, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:45, palf_committed_end_lsn_:{lsn:8461098908}, palf_committed_end_scn_:{val:1755161719775179002, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161719775179002, v:0}}) [2025-08-14 09:02:09.749525] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=15] get_min_unreplayed_log_info(lsn={lsn:8461098908}, scn={val:1755100947600130001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161920778323, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:23084275, ret_code_:0}}, next_to_submit_lsn_:{lsn:8461098908}, next_to_submit_scn_:{val:1755100947600130001, v:0}, base_lsn_:{lsn:7515734016}, base_scn_:{val:1753840867171901045, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6e05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:8461098908}, end_lsn:{lsn:8461098908}, read_buf:{buf_len_:3747840, buf_:0x7f98f6e05000}, block_size:67104768, log_storage_:0x7f99297a3bf0, log_storage_:{log_tail:{lsn:8461098908}, readable_log_tail:{lsn:8461098908}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8455200768}, min_scn:{val:1755160384851871002, v:0}, curr_block_id:126, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:215, min_block_id:93, max_block_id:127, curr_writable_block_id:126}, logical_block_size_:67104768, curr_block_writable_size_:61206628, block_header_serialize_buf_:0x7f99297a48b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:198, read_io_size:699822080, read_disk_cost_ts:4178035, start_lsn:{lsn:7515734016}}}}}}) [2025-08-14 09:02:09.749005] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 509 ctx_id= DEFAULT_CTX_ID hold= 5128192 used= 3765328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3765328 used= 3723912 count= 84 avg_used= 44332 mod=SUMMARY [2025-08-14 09:02:09.751370] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=2336] [MEMORY] tenant_id= 509 ctx_id= CO_STACK hold= 10485760 used= 10321920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10321920 used= 10302720 count= 20 avg_used= 515136 mod=SUMMARY [2025-08-14 09:02:09.751424] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant: 1001, limit: 1073741824 hold: 606650368 cache_hold: 44040192 cache_used: 44040192 cache_item_count: 21 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 302309376 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 54525952 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 3080192 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 483328 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 45514752 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 5521408 limit= 157286400 [MEMORY] ctx_id= GLIBC hold_bytes= 15499264 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 73424896 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35651584 limit= 214748360 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 3252224 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 12582912 limit= 1073741800 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 5570560 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3866624 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 1179648 limit= 9223372036854775807 [2025-08-14 09:02:09.751449] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1001 ctx_id= DEFAULT_CTX_ID hold= 302309376 used= 268986064 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 81 washed_blocks= 406 washed_size= 113410048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 268986064 used= 262866630 count= 3793 avg_used= 69303 mod=SUMMARY [2025-08-14 09:02:09.751476] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=21] [MEMORY] tenant_id= 1001 ctx_id= MEMSTORE_CTX_ID hold= 54525952 used= 54099968 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 54099968 used= 54075216 count= 26 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:02:09.751483] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= EXECUTE_CTX_ID hold= 3080192 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 8 washed_size= 7405568 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:09.751491] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1001 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:02:09.751499] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= PLAN_CACHE_CTX_ID hold= 45514752 used= 36449984 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 77 washed_blocks= 201 washed_size= 132743168 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 36449984 used= 32916258 count= 3312 avg_used= 9938 mod=SUMMARY [2025-08-14 09:02:09.751507] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= WORK_AREA hold= 5521408 used= 0 limit= 157286400 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 9 washed_blocks= 40 washed_size= 17547264 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:09.751515] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= GLIBC hold= 15499264 used= 10889728 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 25 washed_blocks= 164 washed_size= 47415296 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10889728 used= 6020141 count= 18313 avg_used= 328 mod=SUMMARY [2025-08-14 09:02:09.751524] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= CO_STACK hold= 73424896 used= 71221248 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 32 washed_blocks= 34 washed_size= 2072576 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 71221248 used= 71088768 count= 138 avg_used= 515136 block_cnt= 138 chunk_cnt= 36 mod=CoStack [MEMORY] hold= 71221248 used= 71088768 count= 138 avg_used= 515136 mod=SUMMARY [2025-08-14 09:02:09.751536] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1001 ctx_id= KVSTORE_CACHE_ID hold= 44040192 used= 44040192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 44040192 used= 0 count= 21 avg_used= 0 mod=SUMMARY [2025-08-14 09:02:09.751544] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 33906624 limit= 214748360 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33906624 used= 30629648 count= 468 avg_used= 65447 mod=SUMMARY [2025-08-14 09:02:09.751551] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1001 ctx_id= TX_CALLBACK_CTX_ID hold= 3252224 used= 2342592 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 3 washed_blocks= 8 washed_size= 3039232 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2342592 used= 2269696 count= 286 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:09.751559] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:09.751568] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= RPC_CTX_ID hold= 12582912 used= 393216 limit= 1073741800 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 393216 used= 260096 count= 16 avg_used= 16256 mod=SUMMARY [2025-08-14 09:02:09.751578] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= TX_DATA_TABLE hold= 5570560 used= 2342784 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 8 washed_size= 2818048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2342784 used= 2319424 count= 30 avg_used= 77314 mod=SUMMARY [2025-08-14 09:02:09.751587] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= MDS_DATA_ID hold= 3866624 used= 2785280 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 5 washed_size= 4521984 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2785280 used= 2698240 count= 340 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:09.751602] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1001 ctx_id= MERGE_NORMAL_CTX_ID hold= 1179648 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 8 washed_blocks= 24 washed_size= 15597568 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:09.751618] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant: 1002, limit: 8589934592 hold: 1943941120 cache_hold: 48234496 cache_used: 48234496 cache_item_count: 23 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 1007513600 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 415236096 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 1114112 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 950272 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 211812352 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 1441792 limit= 429496725 [MEMORY] ctx_id= GLIBC hold_bytes= 12582912 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 149913600 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 37748736 limit= 1717986900 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 7766016 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 5980160 limit= 8589934500 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 5718016 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3047424 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 1032192 limit= 9223372036854775807 [2025-08-14 09:02:09.751645] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= DEFAULT_CTX_ID hold= 1007513600 used= 916354480 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 13 [MEMORY] hold= 185860096 used= 185534336 count= 114 avg_used= 1627494 block_cnt= 114 chunk_cnt= 104 mod=MysqlRequesReco [MEMORY] hold= 167792640 used= 167772192 count= 1 avg_used= 167772192 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 83927040 used= 83886080 count= 2 avg_used= 41943040 block_cnt= 2 chunk_cnt= 2 mod=LogGroupBuffer [MEMORY] hold= 67190784 used= 67109344 count= 4 avg_used= 16777336 block_cnt= 4 chunk_cnt= 4 mod=MrgMemPoolChk [MEMORY] hold= 916354480 used= 899157225 count= 9436 avg_used= 95290 mod=SUMMARY [2025-08-14 09:02:09.751656] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= MEMSTORE_CTX_ID hold= 415236096 used= 411992064 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 411992064 used= 411803568 count= 198 avg_used= 2079816 block_cnt= 198 chunk_cnt= 198 mod=Memstore [MEMORY] hold= 411992064 used= 411803568 count= 198 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:02:09.751663] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= EXECUTE_CTX_ID hold= 1114112 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 7 washed_size= 7274496 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:09.751670] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= TRANS_CTX_MGR_ID hold= 950272 used= 933888 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1146880 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 933888 used= 921984 count= 2 avg_used= 460992 mod=SUMMARY [2025-08-14 09:02:09.751679] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= PLAN_CACHE_CTX_ID hold= 211812352 used= 201995008 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 176930880 used= 160228962 count= 13450 avg_used= 11912 block_cnt= 13450 chunk_cnt= 101 mod=SqlPhyPlan [MEMORY] hold= 201995008 used= 183048143 count= 17906 avg_used= 10222 mod=SUMMARY [2025-08-14 09:02:09.751685] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1002 ctx_id= WORK_AREA hold= 1441792 used= 0 limit= 429496725 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 14 washed_blocks= 54 washed_size= 27918336 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:09.751693] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= GLIBC hold= 12582912 used= 11425280 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 11425280 used= 6322556 count= 19216 avg_used= 329 mod=SUMMARY [2025-08-14 09:02:09.751701] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1002 ctx_id= CO_STACK hold= 149913600 used= 147087360 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 68 washed_blocks= 70 washed_size= 3178496 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 147087360 used= 146813760 count= 285 avg_used= 515136 block_cnt= 285 chunk_cnt= 73 mod=CoStack [MEMORY] hold= 147087360 used= 146813760 count= 285 avg_used= 515136 mod=SUMMARY [2025-08-14 09:02:09.751713] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1002 ctx_id= KVSTORE_CACHE_ID hold= 48234496 used= 48234496 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 48234496 used= 0 count= 23 avg_used= 0 mod=SUMMARY [2025-08-14 09:02:09.751720] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1002 ctx_id= META_OBJ_CTX_ID hold= 37748736 used= 35741632 limit= 1717986900 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 35741632 used= 32457488 count= 496 avg_used= 65438 mod=SUMMARY [2025-08-14 09:02:09.751728] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= TX_CALLBACK_CTX_ID hold= 7766016 used= 6934400 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 17 washed_size= 6914048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 6934400 used= 6729728 count= 848 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:09.751736] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= LOB_CTX_ID hold= 147456 used= 16384 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 16384 used= 15872 count= 2 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:09.751746] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= RPC_CTX_ID hold= 5980160 used= 2097152 limit= 8589934500 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 6 washed_blocks= 6 washed_size= 10797056 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2097152 used= 1372672 count= 92 avg_used= 14920 mod=SUMMARY [2025-08-14 09:02:09.733171] WDIAG [RPC] handle_resp (ob_poc_rpc_proxy.cpp:192) [314][pnio1][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] async_cb handler cost too much time(total_time=972401, decode_time=-1755162128760769, process_time=0, session_destroy_time=1755162129733170, ret=0, pcode=5385) [2025-08-14 09:02:09.752998] WDIAG stat_cleanup (debug.c:35) [314][pnio1][T0][Y0-0000000000000000-0-0] [lt=19794][errcode=0] PNIO [delay_warn] cost too much time: 992006us, procedure: pn_pktc_resp_cb [2025-08-14 09:02:09.753010] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [314][pnio1][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559519, sock=0x7f995f412a58, code=-47 [2025-08-14 09:02:09.753016] WDIAG delay_warn (debug.h:103) [314][pnio1][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] PNIO [delay_warn] pktc_flush_cb delay high: 4029027, start_us=1755162125722896 [2025-08-14 09:02:09.753029] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [314][pnio1][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162006156846", warnings:[]}, tenant_id_=1, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:02:09.753050] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [314][pnio1][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559527, sock=0x7f995f412a58, code=-47 [2025-08-14 09:02:09.753057] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [314][pnio1][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162006762903", warnings:[]}, tenant_id_=1, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:02:09.751755] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1002 ctx_id= TX_DATA_TABLE hold= 5718016 used= 4521856 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 9 washed_blocks= 16 washed_size= 13156352 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4521856 used= 4480128 count= 40 avg_used= 112003 mod=SUMMARY [2025-08-14 09:02:09.753143] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=1376] [MEMORY] tenant_id= 1002 ctx_id= MDS_DATA_ID hold= 3047424 used= 1998848 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 4 washed_size= 1146880 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1998848 used= 1936384 count= 244 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:09.753151] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1002 ctx_id= MDS_CTX_ID hold= 147456 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:09.753173] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:09.753181] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= MERGE_NORMAL_CTX_ID hold= 1032192 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 12 washed_size= 13647872 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:09.739675] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=2042] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623031177283, flush_io_finished_ret_:0, flush_io_finished_round_:26271, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:09.715361] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:125) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=952263] try to adjust log cb pool [2025-08-14 09:02:09.718026] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=970264][errcode=0] epoll handler too much time :971750 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:09.754146] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=36051] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:08.762366] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:09.754245] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=990377] temporary file module use no blocks [2025-08-14 09:02:09.754251] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=5] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:09.754259] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=7] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:09.754268] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=9] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623030201550, flush_io_finished_ret_:0, flush_io_finished_round_:26295, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:08.761474] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=40822][errcode=0] epoll handler too much time :40899 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:09.754332] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=991340] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:09.746269] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:02:09.753067] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [314][pnio1][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559536, sock=0x7f995f412a58, code=-47 [2025-08-14 09:02:09.756713] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:09.756789] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:09.757352] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:09.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.757378] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:09.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.757426] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=17291] PNIO [ratelimit] time: 1755162129757425, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006405, add_bytes: 0 [2025-08-14 09:02:09.757992] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=32] current monitor number(seq_=-1) [2025-08-14 09:02:09.758715] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ad990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:09.758742] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919ad810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:09.761370] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:09.761434] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:09.761451] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5655] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:09.761750] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=13] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [416]OB_MALLOC COST TOO MUCH TIME, cost_time=965358, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=11969], seq:[0, 1] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=981719, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=981710], seq:[0] [2025-08-14 09:02:09.761776] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=10] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:09.768588] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=22700] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:09.768926] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:09.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.768973] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7277230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ea690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:09.768985] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7272330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919a5090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:09.768994] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919822b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891970390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:09.769003] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891969a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:09.769012] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d71b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891969890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:09.769022] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196b390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:09.769031] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a27b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ed810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:09.769040] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a40b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7283690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:09.769048] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7faf30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7281590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:09.770196] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:09.779018] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:09.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.779070] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7352330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196fc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:09.785271] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:09.789120] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919aca90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:09.789144] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f734fc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891918610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:09.799527] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891969110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:09.799552] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196c510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:09.809613] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:09.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:09.821496] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=581998254, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:09.821518] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:09.846569] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=911742][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:02:09.916415] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=854983] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:02:09.921536] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=581998254, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:09.921553] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:09.999802] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:09.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.020877] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:10.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.021597] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=581998254, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:10.021611] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:10.121685] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=581998254, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:10.121701] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:10.135193] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:10.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.221756] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=581998254, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:10.221771] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:10.221822] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [622][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9243-0-0] [lt=8024][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162130221223) [2025-08-14 09:02:10.221840] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [622][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9243-0-0] [lt=13][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162130221223}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162130221257}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162130221257}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:10.222425] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=581998254, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:10.222433] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:10.235269] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:10.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.241345] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:10.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.251410] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:10.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.272893] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:10.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.303083] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:10.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.322511] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1}, delta=581998254, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:10.322528] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:10.335340] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:10.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.422602] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=581998254, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:10.422621] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:10.435403] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:10.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.493349] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:10.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.522637] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=581998254, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:10.522653] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:10.524608] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:10.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.563358] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fe410, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe6f0, time_wheel:0x7f9922df9d40, thread_pool:0x7f9922df9520, is_running:1, time_interval:2.00s, expected_run_time:09:02:10.432, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.619413] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:10.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.622646] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=581998254, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:10.622669] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:10.711354] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [316][pnio1][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1002, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162121718605", warnings:[]}, tenant_id_=1002, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:02:10.162283] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:10.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.720416] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=557283][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:10.162|207=558.13ms|210=0us|222=2us|total=558.16ms) [2025-08-14 09:02:10.720445] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=-4389] timer task use too much time(end=1755162130720444, start=1755162130162280, delta=558164) [2025-08-14 09:02:09.809651] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72eb110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.794, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:09.990696] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:09.989, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.329010] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=492964][errcode=0] EASY SLOW: thread: 7f99542fb300, lastrun: 1755162115.752503 cost: 14.576505 loop:26416, slowcnt: 104 [2025-08-14 09:02:10.535456] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:10.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.721445] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=185705][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:10.535|207=185.99ms|210=1us|222=1us|total=186.00ms) [2025-08-14 09:02:10.721459] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task use too much time(end=1755162130721458, start=1755162130535454, delta=186004) [2025-08-14 09:02:10.726757] WDIAG [STORAGE] reset (ob_access_service.cpp:42) [871][T1002_TSnapSvc][T1001][YB42AC4D0007-00063C4962B99DC4-0-0] [lt=13][errcode=-4389] guard used too much time(guard_used_us=194011337, ls_id={id:1}, lbt()="0x26a1806d 0x1c3e21a6 0x940a506 0x9407229 0x939ba72 0x9399e3d 0x9399084 0x93973b0 0x93942f7 0x9725f91 0x124dfd77 0x96a5a4d 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x9acea95 0x1df14b07 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3") [2025-08-14 09:02:10.720363] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:10.722261] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6820] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:07.763314] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4000] [IO STATUS TENANT](tenant_id=1, ref_cnt=7, io_config={group_configs_cnt:3, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, param_config_:{memory_limit:2147483648, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=1126688, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:02:10.721354] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=910320][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:09.809|207=911.70ms|210=0us|222=2us|total=911.72ms) [2025-08-14 09:02:10.732413] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11041][errcode=-4389] timer task use too much time(end=1755162130732412, start=1755162129809650, delta=922762) [2025-08-14 09:02:10.732424] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=927886, expected_run_ts_us_=1755162129804536, ObClockGenerator::getRealClock()=1755162130732423, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196b690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.732444] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f989196b690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.721385] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=729581][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:09.990|207=730.69ms|210=1us|222=1us|total=730.70ms) [2025-08-14 09:02:10.697597] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:1, time_interval:3.00s, expected_run_time:09:02:10.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.732471] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11067][errcode=-4389] timer task use too much time(end=1755162130732470, start=1755162129990694, delta=741776) [2025-08-14 09:02:10.734884] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=2405][errcode=0] task scheduled too delay(scheduled_delay=729396, expected_run_ts_us_=1755162130005487, ObClockGenerator::getRealClock()=1755162130734884, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:10.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.734908] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:10.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.734923] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=645606, expected_run_ts_us_=1755162130089316, ObClockGenerator::getRealClock()=1755162130734923, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:10.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.732456] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=925397, expected_run_ts_us_=1755162129807059, ObClockGenerator::getRealClock()=1755162130732456, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7282d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.735763] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=7] gc stale ls task succ [2025-08-14 09:02:10.730925] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10544][errcode=0] epoll handler too much time :10573 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:09.724087] EDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=299][errcode=-4038] LOGGER COST TOO MUCH TIME, cost: 1012959, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=46, FORMAT_END=1012908, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1b5043ed 0x1b501579 0x1b5009d1 0x1b50387c 0x1b8905de 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [372]OB_MALLOC COST TOO MUCH TIME, cost_time=1013410, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=29, WASH_BLOCK_END=14932, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=981740], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:09.716356] EDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=480048][errcode=-4016] LOGGER COST TOO MUCH TIME, cost: 1036813, owner: ObLog, click_count: 11, time dist:[ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=116, ALLOC_BLOCK#start=12898, ALLOC_BLOCK#end=65, ALLOC_BLOCK#start=10283, ALLOC_BLOCK#end=1, FORMAT_END=13, ObMalloc#start=1, ObMalloc#end=1013427, ALLOC_END=4, APPEND_END=1], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x13b06ffb 0x958a39e 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [621]OB_MALLOC COST TOO MUCH TIME, cost_time=632725, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=41, WASH_BLOCK_END=603891, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=28788], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:10.732003] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=1980006][errcode=-4015] clock out of order(cur_ts=1755162129758427, cur_ts_=1755162129736770, delta=21657) [2025-08-14 09:02:10.738249] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7880] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:09.724087] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=299][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162129261077], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:09.835564] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=11431][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd070, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fd350, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:5.00s, expected_run_time:09:02:09.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.738615] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1012959] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162129261077]}) [2025-08-14 09:02:10.738627] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162129261077]}) [2025-08-14 09:02:10.738647] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:10.738655] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162130738644], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:10.738665] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162130738644]}) [2025-08-14 09:02:10.738671] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162130738644]}) [2025-08-14 09:02:09.768626] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=22673] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:09.815933] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:02:10.738800] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=921472] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:02:10.738810] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:10.738818] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:09.819004] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:09.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.738846] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=918450][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:09.819|207=919.84ms|210=0us|222=1us|total=919.85ms) [2025-08-14 09:02:10.738857] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task use too much time(end=1755162130738857, start=1755162129819002, delta=919855) [2025-08-14 09:02:10.738854] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:10.738872] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:10.738883] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162130738850], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:10.738893] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162130738850]}) [2025-08-14 09:02:10.738900] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162130738850]}) [2025-08-14 09:02:10.734953] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=2492][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7282d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.739657] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=4692][errcode=0] task scheduled too delay(scheduled_delay=932499, expected_run_ts_us_=1755162129807156, ObClockGenerator::getRealClock()=1755162130739656, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7289710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.739682] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7289710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.739696] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=938541, expected_run_ts_us_=1755162129801154, ObClockGenerator::getRealClock()=1755162130739695, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7397b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.734933] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:10.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.740098] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=581998254, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:10.740118] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:10.747340] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:10.747384] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:10.750028] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:10.750283] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:10.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.750318] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:10.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.739726] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=4781][errcode=0] task scheduled too delay(scheduled_delay=535428, expected_run_ts_us_=1755162130204297, ObClockGenerator::getRealClock()=1755162130739726, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:10.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.751017] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11272][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:10.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.751099] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=62][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:10.739, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.739708] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7397b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.751172] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11443][errcode=0] task scheduled too delay(scheduled_delay=949452, expected_run_ts_us_=1755162129801719, ObClockGenerator::getRealClock()=1755162130751171, ret=0, ret="OB_SUCCESS", *this={this:0x7f98918299b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891916210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.754302] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:10.754361] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=17] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:10.754376] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] temporary file module use no blocks [2025-08-14 09:02:10.754382] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=5] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:10.754400] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=16] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:09.716356] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=480048][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:02:10.754800] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=1036813][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:10.754814] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:10.754840] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:02:10.754855] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:10.754864] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:10.754982] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:10.754995] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:10.755007] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:10.755016] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=763, timeout ts=09:02:10.603, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:40.603], idx=763, next_allow_print_ts="09:02:40.603") [2025-08-14 09:02:10.755024] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:10.755364] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:10.755378] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:10.755392] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:02:10.755505] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13530] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:10.755535] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27] temporary file module use no blocks [2025-08-14 09:02:10.755541] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:10.755550] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=7] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:10.755559] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623032192995, flush_io_finished_ret_:0, flush_io_finished_round_:26272, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:10.121932] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9242-0-0] [lt=25][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162130121136) [2025-08-14 09:02:10.755674] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9242-0-0] [lt=632775][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162130121136}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162130739009}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162130739009}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:10.742350] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6797] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:10.755701] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13330][errcode=0] epoll handler too much time :13360 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:09.733190] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1022852, owner: ObLog, click_count: 9, time dist:[ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=103, ALLOC_BLOCK#start=10024, ALLOC_BLOCK#end=1, FORMAT_END=1012711, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:09.735381] EDIAG [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=509834][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1020671, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=79, FORMAT_END=1020584, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1825892b 0x1825769a 0x1823b8a5 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:10.755995] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161101770923, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755161572801635) [2025-08-14 09:02:10.756032] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [316][pnio1][T1][YB42AC4D0007-00063C49607B9242-0-0] [lt=20][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1, delta=534115, last_succ_cluster_heartbeat_tstamp_=1755162130221916) [2025-08-14 09:02:10.756047] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=43] get_max_applied_scn(scn={val:1755161547738516001, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:50, palf_committed_end_lsn_:{lsn:5282737655}, palf_committed_end_scn_:{val:1755161547738516001, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161547738516001, v:0}}) [2025-08-14 09:02:10.754805] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=8432] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:02:10.754411] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623032194054, flush_io_finished_ret_:0, flush_io_finished_round_:26296, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:10.756087] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] get_min_unreplayed_log_info(lsn={lsn:5282737655}, scn={val:1755100947825033001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161624848121, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:13063726, ret_code_:0}}, next_to_submit_lsn_:{lsn:5282737655}, next_to_submit_scn_:{val:1755100947825033001, v:0}, base_lsn_:{lsn:4831543296}, base_scn_:{val:1755063445733337001, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7605000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:5282737655}, end_lsn:{lsn:5282737655}, read_buf:{buf_len_:3747840, buf_:0x7f98f7605000}, block_size:67104768, log_storage_:0x7f993e529bf0, log_storage_:{log_tail:{lsn:5282737655}, readable_log_tail:{lsn:5282737655}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:5234171904}, min_scn:{val:1755152244802535004, v:0}, curr_block_id:78, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:206, min_block_id:42, max_block_id:79, curr_writable_block_id:78}, logical_block_size_:67104768, curr_block_writable_size_:18539017, block_header_serialize_buf_:0x7f993e52a8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:85, read_io_size:301367296, read_disk_cost_ts:2364604, start_lsn:{lsn:4831543296}}}}}}) [2025-08-14 09:02:10.756826] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=90] get wrs ts(ls_id={id:1}, delta=583017131, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:10.756843] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:10.756875] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162130756038}, version_delta=583018359) [2025-08-14 09:02:09.733190] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:09.735381] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=509834] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3847) [2025-08-14 09:02:10.757622] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1022852][errcode=0] epoll handler too much time :1024442 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:10.757633] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=1020671][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:10.757653] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:10.757656] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:10.757668] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=11] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3848) [2025-08-14 09:02:10.757675] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:10.757842] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:10.757861] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=18] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3849) [2025-08-14 09:02:10.757866] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:10.758611] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=13] current monitor number(seq_=-1) [2025-08-14 09:02:07.763314] EDIAG [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4000][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2964198, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1988026, ObMalloc#start=1, ObMalloc#end=965389, ALLOC_END=10781, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0xfad91f4 0x99f635e 0xfad88de 0xfad867a 0x1f9ebaba 0x97c8ff1 0x1f9819be 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:09.761400] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=8] gc stale ls task succ [2025-08-14 09:02:10.761643] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=7] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:10.761850] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=8] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:10.751187] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918299b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891916210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.762051] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10843][errcode=0] task scheduled too delay(scheduled_delay=936190, expected_run_ts_us_=1755162129825860, ObClockGenerator::getRealClock()=1755162130762051, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ec910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.825, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.762082] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ec910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.825, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.762110] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] task scheduled too delay(scheduled_delay=936039, expected_run_ts_us_=1755162129826071, ObClockGenerator::getRealClock()=1755162130762110, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ec790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.826, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.762129] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f72ec790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.826, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.762142] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=941368, expected_run_ts_us_=1755162129820773, ObClockGenerator::getRealClock()=1755162130762141, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7282f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.820, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.762161] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7282f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.820, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.762174] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=933835, expected_run_ts_us_=1755162129828338, ObClockGenerator::getRealClock()=1755162130762173, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72879b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728a010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.828, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.762192] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72879b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728a010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.828, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.762208] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] task scheduled too delay(scheduled_delay=915798, expected_run_ts_us_=1755162129846410, ObClockGenerator::getRealClock()=1755162130762208, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7289590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.846, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.762245] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7289590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:09.846, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.762278] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72760b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98919b5b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.719, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.762308] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7283810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.762336] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728b690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.762354] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739be30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f728b810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.762369] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7281e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.762384] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7edc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f0910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.762412] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ebe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f994b7f0a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.762424] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891971110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.762443] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f9891968c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.750, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.762458] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7283990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.762467] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=22] PNIO [ratelimit] time: 1755162130762465, bytes: 0, bw: 0.000000 MB/s, add_ts: 1005040, add_bytes: 0 [2025-08-14 09:02:10.762474] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728d1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:415, func_shared_ptr_.ptr:0x7f98f7291110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.762502] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:10.762545] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:10.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.762937] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:10.762958] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=18] temporary file module use no blocks [2025-08-14 09:02:10.762967] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:10.762979] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:10.762992] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623032202641, flush_io_finished_ret_:0, flush_io_finished_round_:26308, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:09.754775] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [314][pnio1][T1002][Y0-0000000000000000-0-0] [lt=1696][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1002, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162008720783", warnings:[]}, tenant_id_=1002, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:02:10.436686] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fa190, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa470, time_wheel:0x7f99341cdbf0, thread_pool:0x7f99341cd3d0, is_running:1, time_interval:2.00s, expected_run_time:09:02:10.334, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.765086] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=327902][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:10.436|207=328.40ms|210=0us|222=1us|total=328.41ms) [2025-08-14 09:02:10.765102] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] timer task use too much time(end=1755162130765102, start=1755162130436684, delta=328418) [2025-08-14 09:02:10.761620] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=373] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:09.841251] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=379995577, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:10.765339] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=922689] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:10.767157] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1001}, delta=386038844, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:10.767183] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:10.767210] INFO [STORAGE.TRANS] amend (ob_tenant_weak_read_server_version_mgr.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] [WRS] tenant has no valid partition on current server, generate weak read snapshot version based on current timestamp(tenant_id=1002, epoch_tstamp_=1755162129841221, total_part_count_=2, valid_inner_part_count_=0, valid_user_part_count_=0, version_={val:1755161758221798000, v:0}, base_version_when_no_valid_partition={val:1755161758221798000, v:0}) [2025-08-14 09:02:10.767263] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get_max_applied_scn(scn={val:1755161749741193000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:50, palf_committed_end_lsn_:{lsn:1604090090}, palf_committed_end_scn_:{val:1755161749741193000, v:0}, last_check_scn_:{val:1755161749741193001, v:0}, max_applied_cb_scn_:{val:1755161749741193000, v:0}}) [2025-08-14 09:02:10.767281] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=381025475, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:10.767306] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:10.767800] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1001}, delta=386038844, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:10.767810] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:10.720403] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=38683] finish update min start scn(max_decided_scn={val:1755161547738516001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1755161493768447002, v:0}, keep_alive_scn_:{val:1755161493768447002, v:0}, update_ts_:1755162129736770}) [2025-08-14 09:02:10.768428] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [14780][T1001_TimerWK4_SSTableGC][T1001][YB42AC4D0007-00063C4974099C4E-0-0] [lt=1976467][errcode=0] cost too much time:mds_table_base.cpp:int oceanbase::storage::mds::MdsTableBase::advance_state_to(oceanbase::storage::mds::MdsTableBase::State) const, (this=|threshold=1.00ms|start at 09:02:10.757|total=10.63ms) [2025-08-14 09:02:10.742378] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4087] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:10.768456] WDIAG ev_invoke_pending (ev.c:2136) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26038][errcode=0] epoll handler too much time :26084 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:10.721314] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=61] finish update min start scn(max_decided_scn={val:1755161719775179002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1755161287778691002, v:0}, keep_alive_scn_:{val:1755161287778691002, v:0}, update_ts_:1755162129736770}) [2025-08-14 09:02:10.768877] INFO [OCCAM] ObOccamTimerTaskRAIIHandle (ob_occam_timer.h:363) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=47] task handle constructed(*this={this:0x7f9917455510, task:NULL, is_inited:false, is_running:false}) [2025-08-14 09:02:10.768929] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:0, time_interval:250.00ms, expected_run_time:09:02:10.989, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.768946] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:02:10.768|total=1.03s) [2025-08-14 09:02:10.768971] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] task executed too delay(delay_time=6028004, *p_task_={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:0, time_interval:250.00ms, expected_run_time:09:02:11.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.769038] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=-4016] task executed too delay(delay_time=3672387, *p_task_={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:0, time_interval:500.00ms, expected_run_time:09:02:11.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.769103] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4016] task executed too delay(delay_time=3551915, *p_task_={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:0, time_interval:500.00ms, expected_run_time:09:02:11.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.769267] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=13] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:10.770227] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:10.772061] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:10.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.772384] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:416, func_shared_ptr_.ptr:0x7f98919b4790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.772410] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919afe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:416, func_shared_ptr_.ptr:0x7f98919b9990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.772421] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d62b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:416, func_shared_ptr_.ptr:0x7f9891968f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.772436] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:416, func_shared_ptr_.ptr:0x7f9891970210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.782415] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:416, func_shared_ptr_.ptr:0x7f9891968b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.782445] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:416, func_shared_ptr_.ptr:0x7f989196b210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.782459] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:416, func_shared_ptr_.ptr:0x7f98f7280990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.782472] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:416, func_shared_ptr_.ptr:0x7f98f7397a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.782892] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:10.792476] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:416, func_shared_ptr_.ptr:0x7f98f7280690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.792506] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:416, func_shared_ptr_.ptr:0x7f994b7efe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.792520] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:416, func_shared_ptr_.ptr:0x7f994b7f0010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.802276] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:10.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.802536] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:416, func_shared_ptr_.ptr:0x7f98f7282c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.812558] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:416, func_shared_ptr_.ptr:0x7f98f728b210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.802, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.821567] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c0b90, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571c0e70, time_wheel:0x7f9959fef990, thread_pool:0x7f9959fef170, is_running:1, time_interval:3.00s, expected_run_time:09:02:10.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.822618] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:416, func_shared_ptr_.ptr:0x7f98f728a190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.814, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.822648] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:416, func_shared_ptr_.ptr:0x7f98f7282010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.817, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.822663] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:416, func_shared_ptr_.ptr:0x7f994b7ee810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.822676] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:416, func_shared_ptr_.ptr:0x7f994b7ee990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.819, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.822690] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:416, func_shared_ptr_.ptr:0x7f98f73a0a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.813, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.832656] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919831b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:416, func_shared_ptr_.ptr:0x7f9891970c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.828, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.838907] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:10.838929] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:10.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.840630] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=1905294][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:02:10.840691] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [571][T1_ArcSender][T1][YB42AC4D0007-00063C495BA99C16-0-0] [lt=1905325][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:02:10.842329] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=6] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:02:10.842359] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=24] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:02:10.842716] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:416, func_shared_ptr_.ptr:0x7f9891973c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.835, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.851677] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EEA5-0-0] [lt=9] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-08-14 09:02:10.852775] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:416, func_shared_ptr_.ptr:0x7f9891968210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.847, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.852809] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:416, func_shared_ptr_.ptr:0x7f9891970b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.844, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.852820] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:416, func_shared_ptr_.ptr:0x7f98f7282190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:10.844, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:10.856555] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=583117250, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:10.856576] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:10.856621] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9249-0-0] [lt=25][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162130856004) [2025-08-14 09:02:10.856641] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9249-0-0] [lt=16][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162130856004}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162130856025}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162130856025}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:10.857230] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=583118317, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:10.857259] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:10.867331] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=381125189, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:10.917095] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=12] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:02:10.938948] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:10.938997] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:10.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:10.957303] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=583217873, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:10.957324] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:10.992206] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:10.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:11.023761] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:11.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:11.038982] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:11.039060] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:10.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:11.120698] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:11.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:11.139044] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:11.139100] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:11.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:11.220759] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:11.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:11.239100] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:11.239106] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=7] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:11.239129] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162131239102], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:11.239152] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162131239102], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:11.239179] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=24] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162131239102]}) [2025-08-14 09:02:11.239168] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:11.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:11.239192] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:11.239203] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:11.239215] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:11.239223] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162131239213], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:11.239247] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=22] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162131239213]}) [2025-08-14 09:02:11.239447] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:11.239465] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:11.239479] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162131239442], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:11.239492] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162131239442]}) [2025-08-14 09:02:11.239503] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts statistics(tenant_id=1002, gts_rpc_cnt=0, get_gts_cache_cnt=21, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:02:11.243708] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=55][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:11.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:11.253771] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:11.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:11.275493] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:11.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:11.305666] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:11.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:11.320832] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:11.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:11.330680] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=584213][errcode=0] EASY SLOW: thread: 7f99543f7300, lastrun: 1755162130.744743 cost: 0.585935 loop:26483, slowcnt: 5 [2025-08-14 09:02:11.339111] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:11.339218] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:11.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:11.420882] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:11.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:11.439181] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:11.439261] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:11.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:11.495187] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:11.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:11.520943] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:11.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:11.526840] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:11.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:11.539220] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:11.539325] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:11.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:11.620724] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:11.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:11.620998] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:11.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:11.622095] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ebbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:416, func_shared_ptr_.ptr:0x7f994b7ee510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:11.615, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:11.639280] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:11.639387] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:11.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:11.721047] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6998] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:11.729008] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:11.693, task_priority:1, with_handle_protected:1}) [567]OB_MALLOC COST TOO MUCH TIME, cost_time=675649, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:02:11.256278] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:11.257952] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:11.730942] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=472272] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3850) [2025-08-14 09:02:11.730961] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:11.730984] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:11.730997] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:227) [1031][PxTargetMgr0][T1001][Y0-0000000000000000-0-0] [lt=10] refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3851) [2025-08-14 09:02:11.731006] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:11.731183] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:11.731196] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=12] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3852) [2025-08-14 09:02:11.731204] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:10.867363] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=29] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:11.723384] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9707] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:11.732518] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:11.735808] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=862687] get wrs ts(ls_id={id:1001}, delta=387001697, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:11.735846] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=33] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:11.735925] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=381988328, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:11.735939] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:10.732395] EDIAG [BALANCE] gather_stat_primary_zone_num_and_units (ob_tenant_balance_service.cpp:179) [784][T1002_TBalance][T1002][YB42AC4D0007-00063C4964C9A63F-0-0] [lt=1984398][errcode=-4012] LOGGER COST TOO MUCH TIME, cost: 1002656, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1002648, ObMalloc#start=0, ObMalloc#end=3, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1053a872 0x1053a038 0x10531684 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:11.736450] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=10] gc stale ls task succ [2025-08-14 09:02:10.732395] WDIAG [BALANCE] gather_stat_primary_zone_num_and_units (ob_tenant_balance_service.cpp:179) [784][T1002_TBalance][T1002][YB42AC4D0007-00063C4964C9A63F-0-0] [lt=1984398][errcode=-4012] failed to get primary zone unit array(ret=-4012, ret="OB_TIMEOUT", tenant_schema={tenant_id:1002, schema_version:1753069684171464, tenant_name:"mdsk", zone_list:[cnt:1, "zone1"], primary_zone:"RANDOM", charset_type:0, locked:false, comment:"", name_case_mode:-1, read_only:false, locality_str:"FULL{1}@zone1", zone_replica_attr_array:[cnt:1, {replica_attr_set:{full_replica_attr_array:[cnt:1, {num:1, memstore_percent:100}], logonly_replica_attr_array:[cnt:0], readonly_replica_attr_array:[cnt:0], encryption_logonly_replica_attr_array:[cnt:0], columnstore_replica_attr_array:[cnt:0]}, zone_set:[cnt:1, "zone1"], zone:"zone1"}], primary_zone_array:[], previous_locality_str:"", default_tablegroup_id:18446744073709551615, default_tablegroup_name:"", compatibility_mode:0, drop_tenant_time:-1, status:0, in_recyclebin:false, arbitration_service_status:{status:3}}) [2025-08-14 09:02:11.737717] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=583991005, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:11.737740] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:11.738222] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=583991005, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:11.738349] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:11.738653] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=387001697, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:11.738743] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:11.739363] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=968665] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:11.739451] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:11.739475] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:11.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:11.741350] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:11.742254] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:11.742279] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4046] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:11.747198] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:11.747346] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:11.747950] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=13] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:11.747983] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=30][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162131747945], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:11.748002] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162131747945], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:11.748020] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162131747945]}) [2025-08-14 09:02:11.748034] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=11] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:11.748042] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162131748033], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:11.748054] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162131748033]}) [2025-08-14 09:02:11.748223] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:11.748244] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:11.748259] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162131748219], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:11.748272] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162131748219]}) [2025-08-14 09:02:11.748521] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:416, func_shared_ptr_.ptr:0x7f98919abd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:11.721, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:11.748559] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:416, func_shared_ptr_.ptr:0x7f98919abb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:11.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:11.748575] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:416, func_shared_ptr_.ptr:0x7f98919b4f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:11.726, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:11.748586] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fdc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:416, func_shared_ptr_.ptr:0x7f98f7282310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:11.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:11.748604] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:416, func_shared_ptr_.ptr:0x7f98f7282490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:11.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:11.748619] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:416, func_shared_ptr_.ptr:0x7f98f728be10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:11.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:11.748689] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:11.733, task_priority:1, with_handle_protected:1}) [391]OB_MALLOC COST TOO MUCH TIME, cost_time=1004075, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=39, WASH_BLOCK_END=1004033, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:10.744754] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3185][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1004114, owner: ObLog, click_count: 5, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=1004099, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:09.766338] EDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [15320][T1001_TimerWK9_TabletStatRpt][T1001][Y0-0000000000000000-0-0] [lt=2986183][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1981764, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1981757, ObMalloc#start=0, ObMalloc#end=3, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x974f472 0x974e159 0x974e084 0x948fab3 0x1b5700f9 0x96edf7e 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [399]OB_MALLOC COST TOO MUCH TIME, cost_time=1001458, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=46, WASH_BLOCK_END=1001412], seq:[0, 1, 2] [2025-08-14 09:02:10.748966] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1001509, owner: ObLog, click_count: 5, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=1001469, ALLOC_END=26, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:10.769227] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=997736] gc stale ls task succ [2025-08-14 09:02:10.744754] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3185] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:11.750410] WDIAG ev_invoke_pending (ev.c:2136) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1004114][errcode=0] epoll handler too much time :1005665 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:11.750438] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:11.729909] WDIAG [RPC] handle_resp (ob_poc_rpc_proxy.cpp:192) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=77][errcode=0] async_cb handler cost too much time(total_time=960768, decode_time=5, process_time=0, session_destroy_time=960763, ret=0, pcode=5384) [2025-08-14 09:02:11.750797] WDIAG stat_cleanup (debug.c:35) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=20855][errcode=0] PNIO [delay_warn] cost too much time: 982007us, procedure: pn_pktc_resp_cb [2025-08-14 09:02:11.750808] WDIAG eloop_delay_warn (debug.h:93) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] PNIO [delay_warn] eloop handle events delay high: 982007, malloc=0/0 write=0/0 read=0/1 server_process=0/0 client_cb=982007/1 [2025-08-14 09:02:11.750836] WDIAG pktc_handle_req_queue (pktc_post.h:138) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] PNIO [delay_warn] delay high: 883006 [2025-08-14 09:02:09.766338] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [15320][T1001_TimerWK9_TabletStatRpt][T1001][Y0-0000000000000000-0-0] [lt=2986183][errcode=0] long time to hold one guard(schema_mgr=0x7f988fae9820, tenant_id=1001, version=1753074886385104, cur_timestamp=1755162038755203, ref_timestamp=1755161928728963, lbt()="0x26a1806d 0x974f3e5 0x974e159 0x974e084 0x948fab3 0x1b5700f9 0x96edf7e 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3") [383]OB_MALLOC COST TOO MUCH TIME, cost_time=1000270, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=43, WASH_BLOCK_END=1000227], seq:[0, 1, 2] [2025-08-14 09:02:10.750907] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1000326, owner: ObLog, click_count: 5, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=1000289, ALLOC_END=27, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:11.751445] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [14780][T1001_TimerWK4_SSTableGC][T1001][YB42AC4D0007-00063C4974099C4E-0-0] [lt=24][errcode=0] cost too much time:mds_row.ipp:int oceanbase::storage::mds::MdsRow::get_latest_committed(READ_OP &&) const [K = oceanbase::storage::mds::DummyKey, V = oceanbase::..., (this=|threshold=5.00ms|start at 09:02:10.768|541=0us|total=983.01ms) [2025-08-14 09:02:11.751480] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [14780][T1001_TimerWK4_SSTableGC][T1001][YB42AC4D0007-00063C4974099C4E-0-0] [lt=25][errcode=0] cost too much time:mds_unit.ipp:int oceanbase::storage::mds::MdsUnit::get_latest_committed(OP &&) const [K = oceanbase::storage::mds::DummyKey, V = oceanbase::sto..., (this=|threshold=10.00ms|start at 09:02:10.768|910=0us|911=0us|total=983.03ms) [2025-08-14 09:02:10.748966] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:11.752006] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1001509][errcode=0] epoll handler too much time :1003051 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:11.752024] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:10.755346] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [723][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=22] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=16, total_destructed_detector=16, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f99333562b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-08-14 09:02:11.752121] WDIAG [DETECT] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [723][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=995266][errcode=0] cost too much time:ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t), (this=|threshold=100.00ms|start at 09:02:11.752|total=996.79ms) [2025-08-14 09:02:10.750907] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:11.752766] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1000326][errcode=0] epoll handler too much time :1001869 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:11.752783] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:11.755643] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=19] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:11.755662] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] temporary file module use no blocks [2025-08-14 09:02:11.755668] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=5] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:11.755678] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=8] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:11.755688] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623033195346, flush_io_finished_ret_:0, flush_io_finished_round_:26273, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:11.756210] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=1299] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:02:11.760469] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:11.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:11.718675] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=954468] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:11.769278] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [14780][T1001_TimerWK4_SSTableGC][T1001][YB42AC4D0007-00063C4974099C4E-0-0] [lt=12][errcode=0] cost too much time:mds_table_impl.ipp:virtual int oceanbase::storage::mds::MdsTableImpl, oceanbase::st..., (this=|threshold=5.00ms|start at 09:02:10.751|663=0us|total=1.02s) [2025-08-14 09:02:11.769335] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=979620] gc stale ls task succ [2025-08-14 09:02:11.750890] WDIAG [RPC] create (ob_poc_rpc_server.cpp:140) [317][pnio1][T0][YB42AC4D0007-00063C49612B8F3A-0-0] [lt=8][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1815, fly_ts=883578, send_timestamp=1755162130867311, peer="172.77.0.7:42280", sz=265) [2025-08-14 09:02:11.769804] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [979][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F3A-0-0] [lt=1022552][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1002, svr="172.77.0.7:2882", version={val:1755161758221798000, v:0}, valid_part_count=0, total_part_count=2, generate_timestamp=1755162130867302) [2025-08-14 09:02:11.769837] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [979][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F3A-0-0] [lt=24][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1002, req={req_server:"172.77.0.7:2882", version:{val:1755161758221798000, v:0}, valid_part_count:0, total_part_count:2, generate_timestamp:1755162130867302}, twrs={inited:true, tenant_id:1002, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162131735911}, server_version_for_stat:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162131735911}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:11.770117] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=19] PNIO [ratelimit] time: 1755162131770115, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007650, add_bytes: 0 [2025-08-14 09:02:11.770193] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [317][pnio1][T1002][YB42AC4D0007-00063C49612B8F3A-0-0] [lt=18845][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1002, delta=3042366, last_succ_cluster_heartbeat_tstamp_=1755162128727826) [2025-08-14 09:02:11.770252] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119812272, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-08-14 09:02:11.770290] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get_min_unreplayed_log_info(lsn={lsn:1604088821}, scn={val:1755100948296325001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873479, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:12492179, ret_code_:0}}, next_to_submit_lsn_:{lsn:1604088821}, next_to_submit_scn_:{val:1755100948296325001, v:0}, base_lsn_:{lsn:1409200128}, base_scn_:{val:1753840802540914000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:1604088821}, end_lsn:{lsn:1604088821}, read_buf:{buf_len_:3747840, buf_:0x7f98f6a05000}, block_size:67104768, log_storage_:0x7f990bfebbf0, log_storage_:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:42, read_io_size:147709952, read_disk_cost_ts:1061450, start_lsn:{lsn:1409200128}}}}}}) [2025-08-14 09:02:11.770366] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=73] get wrs ts(ls_id={id:1}, delta=382028072, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:11.770379] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:11.770395] INFO [STORAGE] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:123) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] wead read timestamp is too old(timestamp={val:1755161749741193000, v:0}, gts_scn={val:1755161762721798002, v:0}, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1755161749741193000, v:0}}) [2025-08-14 09:02:11.770876] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1001}, delta=387042619, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:11.770886] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:11.771514] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=10] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:11.771570] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:11.773782] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:11.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:11.786946] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:11.803974] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:11.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:11.840266] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:11.840308] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=50][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:11.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:11.842916] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=29] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:02:11.842943] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:02:11.870399] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=382128901, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:11.870427] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:11.870955] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1001}, delta=387142270, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:11.870976] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:11.930468] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=23] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:02:11.940365] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:11.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:11.941261] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:11.970385] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=382228774, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:11.970417] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:11.970930] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1001}, delta=387242143, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:11.970941] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:11.993671] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:11.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:12.025491] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:12.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:12.040414] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:11.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:12.042244] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:12.070467] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=382327281, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:12.070492] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:12.071034] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1001}, delta=387340650, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:12.071051] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:12.140435] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:12.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:12.142315] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:12.170471] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=382428767, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:12.170495] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:12.171083] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1001}, delta=387443196, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:12.171100] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:12.231266] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:12.231293] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=25] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3853) [2025-08-14 09:02:12.231320] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:12.231336] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:12.231345] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3854) [2025-08-14 09:02:12.240472] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:12.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:12.242377] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:12.245262] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:12.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:12.248503] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:12.248527] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162132248497], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:12.248549] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162132248497], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:12.248577] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=23] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162132248497]}) [2025-08-14 09:02:12.248593] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:12.248607] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:12.248619] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=8] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:12.248630] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162132248618], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:12.248646] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162132248618]}) [2025-08-14 09:02:12.248821] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:12.248838] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:12.248850] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162132248817], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:12.248873] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162132248817]}) [2025-08-14 09:02:12.255321] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:12.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:12.277012] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:12.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:12.340506] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:12.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:12.342423] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:12.358576] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] EASY SLOW: thread: 7f995407d300, lastrun: 1755162131.755266 cost: 0.603307 loop:26504, slowcnt: 6 [2025-08-14 09:02:12.440568] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:12.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:12.442482] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:12.466275] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fa190, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa470, time_wheel:0x7f99341cdbf0, thread_pool:0x7f99341cd3d0, is_running:1, time_interval:2.00s, expected_run_time:09:02:12.334, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:12.496679] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:12.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:12.540652] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:12.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:12.542542] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:12.564444] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fe410, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe6f0, time_wheel:0x7f9922df9d40, thread_pool:0x7f9922df9520, is_running:1, time_interval:2.00s, expected_run_time:09:02:12.432, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:12.630864] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=51][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571634a0, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957163780, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:5.00s, expected_run_time:09:02:12.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:12.630898] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:12.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:12.640723] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:12.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:12.642583] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [385]OB_MALLOC COST TOO MUCH TIME, cost_time=967713, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=16369], seq:[0, 1] [2025-08-14 09:02:11.751511] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:12.720738] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=967761][errcode=0] epoll handler too much time :969237 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:11.751127] INFO [STORAGE] runTimerTask (ob_tenant_tablet_stat_mgr.cpp:1211) [15320][T1001_TimerWK9_TabletStatRpt][T1001][Y0-0000000000000000-0-0] [lt=1981764] TenantTabletStatMgr refresh all tablet stream(MTL_ID()=1001, interval_step=3, global_iter_pool={is_inited:false, is_washing:false, is_disabled:false, get_cnt:0, tenant_id:1001, bucket_cnt:0, tenant_mem_user_limit:0, tenant_mem_user_hold:0, cached_node_array:0x7f99333dcff0}) [2025-08-14 09:02:12.722371] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8776] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [272]OB_MALLOC COST TOO MUCH TIME, cost_time=952674, tenant_id=500, label=CommSysVarFac, ctx_id=0, prio=0, size=126944, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:02:11.766117] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [726][T1001_ObXAHbWor][T1001][Y0-0000000000000000-0-0] [lt=21] XA scheduler heartbeat task statistics(avg_time=2) [2025-08-14 09:02:11.768855] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=50102] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [372]OB_MALLOC COST TOO MUCH TIME, cost_time=976667, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=976666], seq:[0] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=976693, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=976681], seq:[0, 1] [2025-08-14 09:02:11.769257] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=994085] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:02:11.730221] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=473222][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:02:12.730269] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=998522][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:12.730289] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:12.730303] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:02:12.730330] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:12.730343] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:12.730362] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:12.730378] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:12.730394] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:12.231354] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:12.730408] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:12.730588] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=498302] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:12.730608] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=19] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3855) [2025-08-14 09:02:12.307196] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:12.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:12.730618] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:12.730628] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=422788][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:12.307|207=423.43ms|210=1us|222=1us|total=423.45ms) [2025-08-14 09:02:12.730619] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:12.730646] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] timer task use too much time(end=1755162132730645, start=1755162132307194, delta=423451) [2025-08-14 09:02:12.730652] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:12.730667] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:02:12.730663] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=220630, expected_run_ts_us_=1755162132510031, ObClockGenerator::getRealClock()=1755162132730662, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:12.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:12.730701] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:12.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:12.732594] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:12.736703] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=15] gc stale ls task succ [2025-08-14 09:02:12.730126] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=959817] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:12.743868] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13720] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:12.743880] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:12.743888] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [857]OB_MALLOC COST TOO MUCH TIME, cost_time=477243, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1], seq:[0, 1] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=910408, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:02:12.748334] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:12.748368] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:12.742263] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:12.741902] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:12.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:12.749785] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:12.750159] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=585011369, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:12.750178] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:12.750207] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=585011688, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=253058, succ_cluster_heartbeat_count=2190, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161000606550000, v:0}, local_cluster_delta=1132143654, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:02:12.750726] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=38] get wrs ts(ls_id={id:1}, delta=585011369, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:12.750738] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [396]OB_MALLOC COST TOO MUCH TIME, cost_time=985032, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=31, WASH_BLOCK_END=985001], seq:[0, 1, 2] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=983558, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=983518], seq:[0, 1, 2] [2025-08-14 09:02:11.764717] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:12.751297] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=985087][errcode=0] epoll handler too much time :986599 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:11.766279] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=18] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:12.746177] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:12.748444] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [953][T1002_L0_G0][T1002][YB42AC4D0007-00063C4965DA1F35-0-0] [lt=39][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:12.748667] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=383006482, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:12.740410] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:12.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:12.752323] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=11889][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:12.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:12.741836] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1}]) [477]OB_MALLOC COST TOO MUCH TIME, cost_time=989781, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=26, WASH_BLOCK_END=989735], seq:[0, 1, 2] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=1005765, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=33, WASH_BLOCK_END=1005731], seq:[0, 1, 2] [552]OB_MALLOC COST TOO MUCH TIME, cost_time=999805, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=29, WASH_BLOCK_END=999776], seq:[0, 1, 2] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=1011559, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=35, WASH_BLOCK_END=1011524], seq:[0, 1, 2] [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=1011414, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=32, WASH_BLOCK_END=1011382], seq:[0, 1, 2] [2025-08-14 09:02:12.738184] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:12.760304] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22085][errcode=0] epoll handler too much time :22131 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:12.748276] WDIAG [BALANCE] gather_stat_ (ob_tenant_balance_service.cpp:198) [784][T1002_TBalance][T1002][YB42AC4D0007-00063C4964C9A63F-0-0] [lt=1002656][errcode=-4012] fail to execute gather_stat_primary_zone_num_and_units(ret=-4012, ret="OB_TIMEOUT", tenant_id_=1002) [2025-08-14 09:02:12.767467] WDIAG [BALANCE] do_work (ob_tenant_balance_service.cpp:87) [784][T1002_TBalance][T1002][YB42AC4D0007-00063C4964C9A63F-0-0] [lt=12464][errcode=-4012] failed to gather stat(ret=-4012, ret="OB_TIMEOUT") [2025-08-14 09:02:12.749096] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:12.767501] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=18374][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162132749092], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:12.767531] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162132749092], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:12.767546] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162132749092]}) [2025-08-14 09:02:12.767567] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=47, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:02:12.767588] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=11] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:12.767594] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162132767584], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:12.767604] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162132767584]}) [2025-08-14 09:02:12.767611] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6] gts statistics(tenant_id=1001, gts_rpc_cnt=0, get_gts_cache_cnt=1, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:02:12.771574] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:12.770476] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:12.767794] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:12.778727] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=10915] get number(ret=-4038, service_type_=0, range=1, base_id=1755162132767412000, start_id=0, end_id=140295968341544) [2025-08-14 09:02:12.778744] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:12.778760] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162132767789], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:12.778777] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162132767789]}) [2025-08-14 09:02:12.772325] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=11] gc stale ls task succ [2025-08-14 09:02:12.782256] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:12.755825] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [14780][T1001_TimerWK4_SSTableGC][T1001][YB42AC4D0007-00063C4974099C4E-0-0] [lt=20][errcode=0] cost too much time:ob_i_tablet_mds_interface.ipp:int oceanbase::storage::ObITabletMdsInterface::get_latest_committed(OP &&) const [T = oceanbase::storage::ObTabletCreateDeleteMdsUserData, OP = oceanbase::storage::MdsDefaultDeepCopyOperationget_ls_id()={id:1001}) [2025-08-14 09:02:16.751188] WDIAG [SQL.SESSION] set_session_state_ (ob_basic_session_info.cpp:6475) [840][T1002_MysqlQueu][T1002][Y0-00063C4965599C46-0-0] [lt=2999][errcode=-5066] session is killed(ret=-5066, sessid_=3221590961, proxy_sessid_=0, state=1) [2025-08-14 09:02:16.758606] WDIAG [SERVER] verify_identify (obmp_connect.cpp:2333) [840][T1002_MysqlQueu][T1002][Y0-00063C4965599C46-0-0] [lt=7404][errcode=-5066] fail to set session state(ret=-5066) [2025-08-14 09:02:16.758619] WDIAG [SERVER] process (obmp_connect.cpp:352) [840][T1002_MysqlQueu][T1002][Y0-00063C4965599C46-0-0] [lt=11][errcode=-5066] fail to verify_identify(ret=-5066) [2025-08-14 09:02:16.758679] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1421) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=28] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1001, is_allowed_broadcast=true) [2025-08-14 09:02:16.758812] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] task executed too delay(delay_time=997172, *p_task_={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:0, time_interval:250.00ms, expected_run_time:09:02:16.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:16.758828] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] task executed too delay(delay_time=979831, *p_task_={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:0, time_interval:250.00ms, expected_run_time:09:02:16.768, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:16.758897] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=14] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:16.761040] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98f739f890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:16.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:16.761135] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=20] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:16.739758] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:943) [342][LSMetaTblUp0][T1][YB42AC4D0007-00063C495B39A91B-0-0] [lt=964098][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:-1}, session={this:0x7f9781604038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f98b5dfc700}) [2025-08-14 09:02:16.759174] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [15320][T1001_TimerWK9][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4389] timer task too much delay in priority queue(token={this:0x7f9931bb2578, timer:0x7f9934119ee0, task:0x7f9934405028, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr17RefreshConfigTaskE", scheduled_time:1755161881725525, pushed_time:1755162043719350, delay:10000000}, delay_in_priority_queue=161993825, threashold=1000000) [2025-08-14 09:02:16.762398] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [15320][T1001_TimerWK9][T1001][Y0-0000000000000000-0-0] [lt=3218][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f9931bb2578, timer:0x7f9934119ee0, task:0x7f9934405028, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr17RefreshConfigTaskE", scheduled_time:1755161881725525, pushed_time:1755162043719350, delay:10000000}, delay_in_thread_pool=93018518, threashold=500000) [2025-08-14 09:02:12.760345] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1531) [871][T1002_TSnapSvc][T1001][YB42AC4D0007-00063C4962B99DC4-0-0] [lt=20] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f98b4f6cc60, tx_id:{txid:0}, state:1, addr:"172.77.0.7:2882", tenant_id:1001, session_id:3221590797, assoc_session_id:3221590797, client_sid:3221590797, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1755161719732459, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180067074, seq_base:1755161719730670, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], conflict_txs:[]}, rollback=true, need_rollback=true, session={this:0x7f990074c1f8, id:3221590797, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1001, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f98b4f6cc60}, exec_ctx.get_errcode()=-4038) [2025-08-14 09:02:16.763068] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:647) [342][LSMetaTblUp0][T1][YB42AC4D0007-00063C495B39A91B-0-0] [lt=22601][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs, exec_ctx)(ret=-4283, session_id=1, *tx_desc={this:0x7f98b5dfc700, tx_id:{txid:21195871}, state:1, addr:"172.77.0.7:2882", tenant_id:1, session_id:1, assoc_session_id:1, client_sid:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1755162132729460, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:28963886, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1753069619, cluster_version:17180067074, seq_base:1755162131769264, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:2}) [2025-08-14 09:02:16.764616] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=14] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:16.764652] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [830][T1002_GCCollect][T1001][YB42AC4D0007-00063C496069AFFA-0-0] [lt=794][errcode=0] cost too much time:mds_row.ipp:int oceanbase::storage::mds::MdsRow::get_latest_committed(READ_OP &&) const [K = oceanbase::storage::mds::DummyKey, V = oceanbase::..., (this=|threshold=5.00ms|start at 09:02:16.750|541=1us|total=14.40ms) [2025-08-14 09:02:16.764684] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [830][T1002_GCCollect][T1001][YB42AC4D0007-00063C496069AFFA-0-0] [lt=25][errcode=0] cost too much time:mds_unit.ipp:int oceanbase::storage::mds::MdsUnit::get_latest_committed(OP &&) const [K = oceanbase::storage::mds::DummyKey, V = oceanbase::sto..., (this=|threshold=10.00ms|start at 09:02:16.750|910=1us|911=0us|total=14.43ms) [2025-08-14 09:02:16.764702] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [830][T1002_GCCollect][T1001][YB42AC4D0007-00063C496069AFFA-0-0] [lt=10][errcode=0] cost too much time:mds_table_impl.ipp:virtual int oceanbase::storage::mds::MdsTableImpl, oceanbase::st..., (this=|threshold=5.00ms|start at 09:02:16.738|663=0us|total=26.06ms) [2025-08-14 09:02:16.764720] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [830][T1002_GCCollect][T1001][YB42AC4D0007-00063C496069AFFA-0-0] [lt=10][errcode=0] cost too much time:ob_i_tablet_mds_interface.ipp:int oceanbase::storage::ObITabletMdsInterface::get_latest_committed(OP &&) const [T = oceanbase::storage::ObTabletCreateDeleteMdsUserData, OP = oceanbase::storage::MdsDefaultDeepCopyOperation &, oceanbase::common::ObIAllocator &) [K = oceanbase::compaction::ObMediumCompactionInfoKey, V = oceanbase::compac..., (this=|threshold=1.00ms|start at 09:02:18.747|219=974.66ms|238=8.78ms|total=983.47ms) [2025-08-14 09:02:19.731273] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:19.731309] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:02:19.731332] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:19.731363] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:19.731392] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:02:19.731409] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:19.731420] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:19.731448] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:19.731460] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:19.731475] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:19.731487] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:19.731511] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:19.731525] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:19.731695] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:19.731709] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:19.731726] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:02:19.732734] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=389980369, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:19.732759] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:19.733367] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1001}, delta=394993738, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:19.733380] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:19.734345] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:19.735157] WDIAG [SHARE] read_unit_groups (ob_unit_table_operator.cpp:1298) [784][T1002_TBalance][T1002][YB42AC4D0007-00063C4964C9A640-0-0] [lt=5799][errcode=-4012] execute sql failed(sql=SELECT distinct unit_group_id, status from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1002) order by unit_group_id , ret=-4012) [2025-08-14 09:02:19.735183] WDIAG [SHARE] get_unit_groups_by_tenant (ob_unit_table_operator.cpp:890) [784][T1002_TBalance][T1002][YB42AC4D0007-00063C4964C9A640-0-0] [lt=23][errcode=-4012] failed to read unit group(ret=-4012, ret="OB_TIMEOUT", sql=SELECT distinct unit_group_id, status from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1002) order by unit_group_id ) [2025-08-14 09:02:19.739265] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:19.733799] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6175) [14780][T1001_TimerWK4_SSTableGC][T1001][YB42AC4D0007-00063C4974099C4E-0-0] [lt=38] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:517}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1755136957320163000}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1755159231184436000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1755159231184436000) [2025-08-14 09:02:19.742294] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:19.742336] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:19.736932] WDIAG [BALANCE] gather_stat_primary_zone_num_and_units (ob_tenant_balance_service.cpp:179) [784][T1002_TBalance][T1002][YB42AC4D0007-00063C4964C9A640-0-0] [lt=14][errcode=-4012] failed to get primary zone unit array(ret=-4012, ret="OB_TIMEOUT", tenant_schema={tenant_id:1002, schema_version:1753069684171464, tenant_name:"mdsk", zone_list:[cnt:1, "zone1"], primary_zone:"RANDOM", charset_type:0, locked:false, comment:"", name_case_mode:-1, read_only:false, locality_str:"FULL{1}@zone1", zone_replica_attr_array:[cnt:1, {replica_attr_set:{full_replica_attr_array:[cnt:1, {num:1, memstore_percent:100}], logonly_replica_attr_array:[cnt:0], readonly_replica_attr_array:[cnt:0], encryption_logonly_replica_attr_array:[cnt:0], columnstore_replica_attr_array:[cnt:0]}, zone_set:[cnt:1, "zone1"], zone:"zone1"}], primary_zone_array:[], previous_locality_str:"", default_tablegroup_id:18446744073709551615, default_tablegroup_name:"", compatibility_mode:0, drop_tenant_time:-1, status:0, in_recyclebin:false, arbitration_service_status:{status:3}}) [2025-08-14 09:02:19.744776] WDIAG [BALANCE] gather_stat_ (ob_tenant_balance_service.cpp:198) [784][T1002_TBalance][T1002][YB42AC4D0007-00063C4964C9A640-0-0] [lt=7820][errcode=-4012] fail to execute gather_stat_primary_zone_num_and_units(ret=-4012, ret="OB_TIMEOUT", tenant_id_=1002) [2025-08-14 09:02:19.744801] WDIAG [BALANCE] do_work (ob_tenant_balance_service.cpp:87) [784][T1002_TBalance][T1002][YB42AC4D0007-00063C4964C9A640-0-0] [lt=23][errcode=-4012] failed to gather stat(ret=-4012, ret="OB_TIMEOUT") [2025-08-14 09:02:19.745261] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:19.747044] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=30] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:19.747253] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:19.747530] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7270030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98919a3b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:19.735, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:19.747600] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=68][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ecb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98919ac310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:19.739, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:19.747615] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7353730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98f7396990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:19.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:19.747651] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f994b7efd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:19.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:19.747710] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=83][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:19.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:19.750293] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:19.750329] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:19.750370] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:02:19.750386] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:19.747122] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=22] gc stale ls task succ [2025-08-14 09:02:19.750902] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=6201][errcode=-4015] clock out of order(cur_ts=1755162139750896, cur_ts_=1755162139721562, delta=29334) [2025-08-14 09:02:19.750346] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:02:19.752041] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:19.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:19.752103] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=17] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:19.751253] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:19.752602] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2713] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:19.756381] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:19.757599] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ea30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98919ac190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:19.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:19.758541] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:19.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:19.758838] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:19.759895] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=46] LogLoopThread round_cost_time(us)(round_cost_time=17) [2025-08-14 09:02:19.760366] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=24] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:19.761186] INFO [SQL.PC] need_check_schema_version (ob_plan_cache_value.cpp:2334) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE3-0-0] [lt=23723] need check schema(new_schema_version=1755156894100160, cached_tenant_schema_version=1755156894100160, is_contain_synonym()=false, contain_sys_name_table_=true, is_contain_tmp_tbl()=false, is_contain_sys_pl_object()=false, need_check=true, constructed_sql_=select * from __all_balance_task where parent_list is null or parent_list = ?) [2025-08-14 09:02:19.763922] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 08:59:22.723173] EDIAG [STORAGE] refresh_queuing_mode (ob_tenant_tablet_stat_mgr.cpp:1161) [14853][T1_TimerWK5_TabletStatRpt][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1027198, owner: ObLog, click_count: 5, time dist:[FORMAT_END=22, ObMalloc#start=1, ObMalloc#end=1027171, ALLOC_END=3, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x1b57202e 0x1b5700ed 0x96edf7e 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:19.764147] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=4137] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:19.764562] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=20] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:19.764584] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162139764557], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:19.764603] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162139764557], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:19.764620] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162139764557]}) [2025-08-14 09:02:19.764636] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=12] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:19.764645] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162139764635], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:19.764657] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162139764635]}) [2025-08-14 09:02:19.764832] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:19.764844] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:19.764869] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162139764828], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:19.764881] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162139764828]}) [2025-08-14 09:02:19.767112] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:02:19.767168] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:02:19.767212] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:02:19.767318] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:02:19.767340] INFO [COMMON] print_io_status (ob_io_manager.cpp:2639) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4090, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:962159, total:962163) [2025-08-14 09:02:19.767382] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, param_config_:{memory_limit:536870912, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=865568, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:02:19.767438] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2826) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=52] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4090, [delay/us]: prepare:0, schedule:0, submit:600000000, device:962159, total:962163) [2025-08-14 09:02:19.767461] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:02:19.767483] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:19.767498] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:19.767520] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SERVER] net_in=4kB/s, net_out=4kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=1280000kB/s [2025-08-14 09:02:19.767891] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=97][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919afbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98919b9810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:19.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:19.767922] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919854b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f9891972190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:19.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:19.767935] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f9891972010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:19.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:19.767948] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f9891971e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:19.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:19.767960] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d45b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98f72ec490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:19.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:19.768287] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=35] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:19.744820] INFO [BALANCE] do_work (ob_tenant_balance_service.cpp:146) [784][T1002_TBalance][T1002][YB42AC4D0007-00063C4964C9A640-0-0] [lt=10] [TENANT_BALANCE] finish one round(ret=-4012, ret="OB_TIMEOUT", tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1002, job_cnt=0, primary_zone_num_=0, unit_group_array_=[], ls_array_=[], idle_time_us=100000, last_partition_balance_time=1755161743785057, last_statistic_bg_stat_time=-1, last_statistic_schema_version=-1, last_statistic_max_transfer_task_id={id:-1}, enable_rebalance=true, enable_transfer=true) [2025-08-14 09:02:19.770034] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=419993517, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:19.770060] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:19.770759] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:19.773475] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=27] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:19.773542] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:19.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:19.780420] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=57][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:19.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:19.781482] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72747b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98919b4c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:19.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:19.781516] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98f7280510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:19.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:19.781535] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f9891825390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:19.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:19.781555] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98919ad510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:19.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:19.781575] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f9891969d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:19.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:19.781594] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f994b7f1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:19.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:19.783294] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:19.785627] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=30] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:19.785659] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=24] temporary file module use no blocks [2025-08-14 09:02:19.785676] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=17] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:19.785690] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:19.785704] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623041225336, flush_io_finished_ret_:0, flush_io_finished_round_:26314, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:19.789998] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=23] gc stale ls task succ [2025-08-14 09:02:19.790291] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:19.790323] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28] temporary file module use no blocks [2025-08-14 09:02:19.790334] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:19.790350] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:19.790375] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=24] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623041228438, flush_io_finished_ret_:0, flush_io_finished_round_:26281, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:19.790488] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=20] current monitor number(seq_=-1) [2025-08-14 09:02:19.791263] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=18630] [sql nio session](*s={this:0x7f98f733f030, session_id:3221589417, trace_id:YB42AC4D0007-00063C496359B268-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:252}, err:0, last_decode_time:1755161813752329, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1778, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:19.791305] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=41] [sql nio session](*s={this:0x7f98f733e030, session_id:3221589980, trace_id:YB42AC4D0007-00063C496359B261-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:251}, err:0, last_decode_time:1755161748778070, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1681, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:19.791320] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=14] [sql nio session](*s={this:0x7f98f7332030, session_id:3221590132, trace_id:YB42AC4D0007-00063C496359B265-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:266}, err:0, last_decode_time:1755161768783802, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:2125, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:19.792312] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98919b5c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:19.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:19.792353] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98f72ec310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:19.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:19.801322] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:19.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:19.816406] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919836b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f9891970f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:19.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:19.831006] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=22] PNIO [ratelimit] time: 1755162139831003, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007611, add_bytes: 0 [2025-08-14 09:02:19.842419] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f9891970e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:19.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:19.842452] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98f72eb590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:19.831, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:19.850444] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:19.852105] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:19.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:19.854309] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=8] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:19.856452] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=22] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:19.871593] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=420095244, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:19.871625] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:19.872323] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c0b90, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571c0e70, time_wheel:0x7f9959fef990, thread_pool:0x7f9959fef170, is_running:1, time_interval:3.00s, expected_run_time:09:02:19.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:19.872535] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98f7280390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:19.862, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:19.950507] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:19.952185] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:19.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:19.955267] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=15] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:19.956524] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:19.971626] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=420195730, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:19.971648] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:19.993337] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=72][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:19.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:20.050571] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:20.052246] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=63][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:19.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:20.056275] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=17] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:20.056550] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:20.071070] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119797572, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755162138745740) [2025-08-14 09:02:20.071693] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=420295587, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:20.071718] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:20.150632] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:20.152302] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:20.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:20.156303] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=16] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:20.156612] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=14] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:20.171734] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=420395593, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:20.171755] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:20.231911] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:20.231946] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:02:20.245100] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=52][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:20.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:20.250693] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:20.255160] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:20.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:20.350756] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:20.450830] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:20.461557] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] EASY SLOW: thread: 7f99542fb300, lastrun: 1755162115.752503 cost: 24.709049 loop:26416, slowcnt: 195 [2025-08-14 09:02:20.490990] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:20.483, task_priority:1, with_handle_protected:1}) [419]OB_MALLOC COST TOO MUCH TIME, cost_time=691864, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=691856], seq:[0] [381]OB_MALLOC COST TOO MUCH TIME, cost_time=478200, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=478194], seq:[0] [381]OB_MALLOC COST TOO MUCH TIME, cost_time=478209, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=478207], seq:[0, 1] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=719774, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=719772], seq:[0] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=707160, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=707159], seq:[0] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=917948, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [857]OB_MALLOC COST TOO MUCH TIME, cost_time=911806, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [729]OB_MALLOC COST TOO MUCH TIME, cost_time=474207, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1], seq:[0, 1] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=697023, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=691886], seq:[0, 1] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=712602, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=707167], seq:[0, 1] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=725514, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=67, WASH_BLOCK_END=153, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=719789], seq:[0, 1, 2, 3, 4] [2025-08-14 09:00:49.752458] EDIAG [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:983) [407][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18125][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1003171, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=1, FORMAT_END=1003166, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a5b252 0x94db124 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [1031]OB_MALLOC COST TOO MUCH TIME, cost_time=538089, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=538081], seq:[0] [2025-08-14 09:02:20.550894] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:20.650934] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:20.664770] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:20.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:20.743512] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:20.231962] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:20.743712] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=510964][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:20.743732] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:02:20.743747] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:20.743757] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:20.252395] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:20.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:20.256329] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=16] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:20.256674] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:20.265097] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:20.744055] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=478230][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162140265094], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:20.744072] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162140265094], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:20.744087] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162140265094]}) [2025-08-14 09:02:20.744098] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:20.744107] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:20.744126] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=6] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:20.744133] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162140744122], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:20.744142] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162140744122]}) [2025-08-14 09:02:20.744330] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:20.744340] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:20.744352] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162140744324], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:20.744372] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162140744324]}) [2025-08-14 09:02:20.744380] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162140744324]}) [2025-08-14 09:02:20.743599] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:20.743580] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:20.743918] WDIAG [STORAGE.TRANS] reset (ob_trans_ctx_lock.cpp:220) [955][T1002_L0_G0][T1002][YB42AC4D0007-00063C4965DA1F2E-0-0] [lt=24][errcode=0] [slow ctx lock](request_used=1, hold_used=245010699, lbt()="0x26a1806d 0x1b543ef5 0x96d03e7 0x95a09c5 0x959f435 0x959e66c 0x94c6522 0x92950b2 0x9285f13 0x927efad 0x926c154 0x12c5a0c9 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3") [2025-08-14 09:02:20.743855] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=486434] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [683]OB_MALLOC COST TOO MUCH TIME, cost_time=339205, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=339197], seq:[0] [683]OB_MALLOC COST TOO MUCH TIME, cost_time=339223, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=339219], seq:[0, 1] [2025-08-14 09:00:49.764298] EDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [359][MetaLocAsyncUp0][T0][YB42AC4D0007-00063C495C199FF0-0-0] [lt=981493][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2984809, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2984808, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x974f472 0x974e159 0x974e084 0x948f9df 0x97257c0 0x97256f4 0x9725691 0x1012b7e7 0x1eeb5700 0x1eeb4f81 0x201f4a4f 0x9a1944d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [797]OB_MALLOC COST TOO MUCH TIME, cost_time=206792, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=206178, ALLOC_CHUNK_END=596], seq:[0, 1] [1031]OB_MALLOC COST TOO MUCH TIME, cost_time=550280, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=538103], seq:[0, 1] [797]OB_MALLOC COST TOO MUCH TIME, cost_time=223324, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=206820, WASH_BLOCK_END=4910, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=11588], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:19.759961] EDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1025915, owner: ObLog, click_count: 5, time dist:[FORMAT_END=987747, ObMalloc#start=1, ObMalloc#end=38140, ALLOC_END=27, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0xfc0e73d 0x993f560 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:20.743791] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=490648][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:20.252|207=491.39ms|210=0us|222=2us|total=491.41ms) [2025-08-14 09:02:20.744995] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1202][errcode=-4389] timer task use too much time(end=1755162140744994, start=1755162140252393, delta=492601) [2025-08-14 09:02:20.743831] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=486747] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:20.743773] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:20.745128] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=1347][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:01:49.731, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:01:42.751], idx=659, next_allow_print_ts="09:02:19.731") [2025-08-14 09:02:20.745145] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:20.745160] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:20.747342] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:20.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:20.051070] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=25] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:02:20.745173] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:20.749249] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=4065][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:20.749265] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:20.749441] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:20.749453] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:20.749464] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:02:20.749956] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=420974481, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:20.036325] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:19.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:20.750072] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=712664][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:20.363|207=713.75ms|210=1us|222=1us|total=713.76ms) [2025-08-14 09:02:20.750088] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] timer task use too much time(end=1755162140750087, start=1755162140036323, delta=713764) [2025-08-14 09:02:20.023445] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:20.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:20.750125] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=725580][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:20.234|207=726.68ms|210=0us|222=1us|total=726.69ms) [2025-08-14 09:02:20.750135] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task use too much time(end=1755162140750135, start=1755162140023442, delta=726693) [2025-08-14 09:02:20.745860] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30] get wrs ts(ls_id={id:1}, delta=593005705, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:20.750469] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:20.750999] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:20.747390] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=3732] gc stale ls task succ [2025-08-14 09:02:20.751310] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=600] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:02:20.749973] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=2, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:20.754664] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=2911] get wrs ts(ls_id={id:1}, delta=420975862, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:20.754684] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:20.760566] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=25] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:20.760861] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2747] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:20.750147] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=5][errcode=0] task scheduled too delay(scheduled_delay=457797, expected_run_ts_us_=1755162140292349, ObClockGenerator::getRealClock()=1755162140750146, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:20.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:20.760911] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=10746][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:20.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:20.750174] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4304] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=7, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:20.761134] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:20.762091] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:20.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:20.762797] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10769] get wrs ts(ls_id={id:1}, delta=593022315, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:20.762812] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:20.762985] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:20.763320] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EEAB-0-0] [lt=22] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-08-14 09:02:20.763341] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EEAB-0-0] [lt=13][errcode=-4016] task executed too delay(delay_time=873403, *p_task_={this:0x7f99571fc8b0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcb90, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:0, time_interval:3.00s, expected_run_time:09:02:22.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:20.423665] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fa190, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa470, time_wheel:0x7f99341cdbf0, thread_pool:0x7f99341cd3d0, is_running:1, time_interval:2.00s, expected_run_time:09:02:20.334, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:20.763460] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=339279][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:20.423|207=339.80ms|210=0us|222=2us|total=339.84ms) [2025-08-14 09:02:20.763507] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=45][errcode=-4389] timer task use too much time(end=1755162140763506, start=1755162140423662, delta=339844) [2025-08-14 09:02:20.766310] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=17] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:20.768341] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:02:20.768393] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=27] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:20.768405] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:02:20.768445] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:02:20.768486] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:02:20.768502] INFO [COMMON] print_io_status (ob_io_manager.cpp:2639) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4091, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:950127, total:950133) [2025-08-14 09:02:20.222051] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:20.773209] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=550321] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3886) [2025-08-14 09:02:20.773220] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:20.773251] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:20.773261] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3887) [2025-08-14 09:02:20.773266] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:20.773476] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=37] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:20.773485] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3888) [2025-08-14 09:02:20.773490] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:20.763373] WDIAG [OCCAM] dump_statistics (ob_vtable_event_recycle_buffer.h:144) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EEAC-0-0] [lt=21][errcode=-4006] not init(ret=-4006) [2025-08-14 09:02:20.771674] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:20.771010] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:20.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:20.768522] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, param_config_:{memory_limit:536870912, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=865568, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:02:20.775292] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2826) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6752] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4064, [delay/us]: prepare:0, schedule:0, submit:600000000, device:950127, total:950133) [2025-08-14 09:02:20.775315] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:02:20.775328] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:20.775339] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:20.775349] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SERVER] net_in=0kB/s, net_out=0kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=1280000kB/s [2025-08-14 09:02:20.743906] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:172) [14905][T1001_TimerWK8_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=21] Eliminate task evict sql audit(request_manager_->get_tenant_id()=1001, queue_size=524288, config_mem_limit_=32212254, request_manager_->get_size_used()=0, evict_high_size_level=471859, evict_low_size_level=419430, allocator->allocated()=8318976, evict_high_mem_level=16106127, evict_low_mem_level=0) [2025-08-14 09:02:19.871885] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [761][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB9096-0-0] [lt=1994535][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1001, svr="172.77.0.7:2882", version={val:1755161719775179002, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162139870903) [2025-08-14 09:02:20.777826] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [761][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB9096-0-0] [lt=904562][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1001, req={req_server:"172.77.0.7:2882", version:{val:1755161719775179002, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162139870903}, twrs={inited:true, tenant_id:1001, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162140752905}, server_version_for_stat:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162140752905}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:20.750452] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4049] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:20.778025] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27528][errcode=0] epoll handler too much time :27580 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:20.777740] INFO [MDS] dump_map_holding_item (mds_tenant_service.cpp:462) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EEAC-0-0] [lt=10130] finish scan map holding items(scan_cnt=0) [2025-08-14 09:02:20.779792] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:20.780110] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [317][pnio1][T1001][YB42AC4D0007-00063C4960CB9096-0-0] [lt=12][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1001, delta=1210884, last_succ_cluster_heartbeat_tstamp_=1755162139569224) [2025-08-14 09:02:20.783320] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:20.779720] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [412][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=954665] [sql nio session](*s={this:0x7f98f733a030, session_id:3221590255, trace_id:YB42AC4D0007-00063C49651A3232-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:257}, err:0, last_decode_time:1755161952736711, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1716, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:20.559708] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fe410, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe6f0, time_wheel:0x7f9922df9d40, thread_pool:0x7f9922df9520, is_running:1, time_interval:2.00s, expected_run_time:09:02:20.432, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:20.783430] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=223379][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:20.559|207=223.72ms|210=0us|222=2us|total=223.75ms) [2025-08-14 09:02:20.783459] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4389] timer task use too much time(end=1755162140783458, start=1755162140559706, delta=223752) [2025-08-14 09:02:19.873612] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:19.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:20.784315] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=909325][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:19.873|207=910.70ms|210=0us|222=1us|total=910.72ms) [2025-08-14 09:02:20.784327] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task use too much time(end=1755162140784327, start=1755162139873609, delta=910718) [2025-08-14 09:02:20.784706] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [14905][T1001_TimerWK8_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=33805] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-08-14 09:02:20.784721] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [14905][T1001_TimerWK8][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] timer task too much delay in priority queue(token={this:0x7f9931bb2d58, timer:0x7f9966a57e50, task:0x7f993335ebf0, task_type:"N9oceanbase7obmysql15ObEliminateTaskE", scheduled_time:1755161984280870, pushed_time:1755162046760808, delay:500000}, delay_in_priority_queue=62479938, threashold=1000000) [2025-08-14 09:02:20.784736] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [14905][T1001_TimerWK8][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f9931bb2d58, timer:0x7f9966a57e50, task:0x7f993335ebf0, task_type:"N9oceanbase7obmysql15ObEliminateTaskE", scheduled_time:1755161984280870, pushed_time:1755162046760808, delay:500000}, delay_in_thread_pool=92956557, threashold=500000) [2025-08-14 09:02:20.784748] WDIAG alarm_if_necessary (ob_timer_service.cpp:143) [14905][T1001_TimerWK8][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task's elapsed_time is greater than period(token={this:0x7f9931bb2d58, timer:0x7f9966a57e50, task:0x7f993335ebf0, task_type:"N9oceanbase7obmysql15ObEliminateTaskE", scheduled_time:1755161984280870, pushed_time:1755162046760808, delay:500000}, elapsed_time=1067355, period=500000) [2025-08-14 09:02:20.784782] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [14905][T1001_TimerWK8][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f9931bb21f8, timer:0x7f99341cd260, task:0x7f993339eb60, task_type:"N9oceanbase3sql27ObSqlPlanMonitorRecycleTaskE", scheduled_time:1755162047760490, pushed_time:1755162047762870, delay:1000000}, delay_in_thread_pool=93021900, threashold=500000) [2025-08-14 09:02:20.736350] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:20.786434] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=50007][errcode=0] epoll handler too much time :50095 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:20.786445] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:20.786466] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] temporary file module use no blocks [2025-08-14 09:02:20.786475] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:20.786489] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:20.736935] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:20.786507] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=18] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623042225499, flush_io_finished_ret_:0, flush_io_finished_round_:26315, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:20.786537] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=49522] temporary file module use no blocks [2025-08-14 09:02:20.786545] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:20.786555] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:20.786578] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=22] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623042168330, flush_io_finished_ret_:0, flush_io_finished_round_:26304, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:20.736553] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=942564] blacklist_loop exec finished(cost_time=18, is_enabled=true, send_cnt=0) [2025-08-14 09:02:20.745887] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:20.786698] WDIAG ev_invoke_pending (ev.c:2136) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=40747][errcode=0] epoll handler too much time :40819 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:20.745978] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=391003028, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:20.786769] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=40724] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:20.746691] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:20.786844] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=40092][errcode=0] epoll handler too much time :40165 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:20.747270] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=29] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:20.786912] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=39581][errcode=0] epoll handler too much time :39683 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:20.747417] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:20.786978] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=39501][errcode=0] epoll handler too much time :39570 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:20.747422] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [998][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F54-0-0] [lt=35][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1002, svr="172.77.0.7:2882", version={val:1755161758221798000, v:0}, valid_part_count=0, total_part_count=2, generate_timestamp=1755162139832745) [2025-08-14 09:02:20.787043] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [998][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F54-0-0] [lt=39559][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1002, req={req_server:"172.77.0.7:2882", version:{val:1755161758221798000, v:0}, valid_part_count:0, total_part_count:2, generate_timestamp:1755162139832745}, twrs={inited:true, tenant_id:1002, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162139732717}, server_version_for_stat:{version:{val:18446744073709551615, v:3}, total_part_count:1, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162140745966}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:20.747536] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f989196a790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:20.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:20.787146] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=39545][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918114b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98f73a0d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:20.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:20.787165] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98f728aa90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:20.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:20.787176] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f994b7f0490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:20.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:20.787189] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f994b7f0610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:20.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:20.787199] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98f7281290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:20.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:20.787210] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f9891825810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:20.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:20.787220] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f9891971290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:20.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:20.787230] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98f73a0c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:20.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:20.787252] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fc830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f9891825210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:20.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:20.787264] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98f7282910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:20.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:20.787274] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a54b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98f7282a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:20.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:20.787285] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98f72edc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:20.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:20.787295] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98f72edb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:20.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:20.787306] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98f7398490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:20.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:20.787317] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98f7398310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:20.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:20.787327] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98f7289110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:20.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:20.787337] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98919ac910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:20.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:20.787347] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98f728ac10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:20.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:19.759961] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1001, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:8571658, max_lsn:7323687369, max_scn:{val:1755161749741193002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:7278610047}, reuse_lsn:{lsn:7323686131}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:7281744329}}, last_submit_lsn:{lsn:7323686131}, last_submit_end_lsn:{lsn:7323687369}, last_submit_log_id:8571658, last_submit_log_pid:49, max_flushed_lsn:{lsn:7323686009}, max_flushed_end_lsn:{lsn:7323686131}, max_flushed_log_pid:49, committed_end_lsn:{lsn:7323686009}, last_slide_log_id:8571656, last_slide_scn:{val:1755161747742105001, v:0}, last_slide_lsn:{lsn:7323685887}, last_slide_end_lsn:{lsn:7323686009}, last_slide_log_pid:49, last_slide_log_accum_checksum:2505458373, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f990112a530}, [StateMgr]={this:0x7f990112e9b0, self:"172.77.0.7:2882", palf_id:1001, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"PENDING", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:49}, leader:"0.0.0.0:0", leader_epoch:-1, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:7323687369}, scan_disk_log_finished:true, last_check_start_id:8571657, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161767744317, palf_role_change_cb:0x7f9901135470, allow_vote:true}, [ConfigMgr]={palf_id:1001, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:49, prev:{version:1, config:{config_version:{proposal_id:48, config_seq:47}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:49, config_seq:48}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_mode_pid:49}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_end_lsn:{lsn:7323685240}, prev_mode_pid:49}, reconfig_barrier:{prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_end_lsn:{lsn:7323685240}, prev_mode_pid:49}, persistent_config_version:{proposal_id:49, config_seq:48}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755161739755312, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755161739755312, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f990112bc70}, [ModeMgr]={palf_id:1001, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, accepted_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1001, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1753069671378640001, v:0}, base_lsn_for_block_gc:{lsn:7247314944}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:49}, log_config_meta:{version:3, proposal_id:49, prev:{version:1, config:{config_version:{proposal_id:48, config_seq:47}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:49, config_seq:48}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_mode_pid:49}, log_snapshot_meta:{version:2, base_lsn:{lsn:7247314944}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1753069671378640000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}}, log_meta_storage:{log_tail:{lsn:765952}, readable_log_tail:{lsn:765952}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7381524480}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/meta", dir_fd:232, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66338816, block_header_serialize_buf_:0x7f9901130bb8, flashback_version:0}, log_storage:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, palf_epoch:2, last_purge_throttling_ts:1755161769777679, this:0x7f990112f0f0}, [Reconfirm]={palf_id:1001, self:"172.77.0.7:2882", state:"WAITING_LOG_FLUSHED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:1755161769777679, this:0x7f990112eb30}) [2025-08-14 09:02:20.787828] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [316][pnio1][T1002][YB42AC4D0007-00063C49612B8F54-0-0] [lt=25][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1002, delta=1455049, last_succ_cluster_heartbeat_tstamp_=1755162139332779) [2025-08-14 09:02:20.788523] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1001}, delta=396060200, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:20.788544] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:20.724996] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:20.789373] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=64278][errcode=0] epoll handler too much time :64391 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:20.734138] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=30] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:20.789406] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=55185][errcode=0] epoll handler too much time :55279 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:20.790130] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:02:20.790553] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=91] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:20.790581] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=24] temporary file module use no blocks [2025-08-14 09:02:20.790589] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:20.790609] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=19] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:20.790621] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623042230235, flush_io_finished_ret_:0, flush_io_finished_round_:26282, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:20.790656] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=42] current monitor number(seq_=-1) [2025-08-14 09:02:20.792132] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:147) [14905][T1001_TimerWK8_][T1001][Y0-0000000000000000-0-0] [lt=8] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-08-14 09:02:20.792158] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [14905][T1001_TimerWK8_][T1001][Y0-0000000000000000-0-0] [lt=21] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-08-14 09:02:20.793132] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get_max_applied_scn(scn={val:1755161719775179002, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:45, palf_committed_end_lsn_:{lsn:8461098908}, palf_committed_end_scn_:{val:1755161719775179002, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161719775179002, v:0}}) [2025-08-14 09:02:20.793163] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] get_min_unreplayed_log_info(lsn={lsn:8461098908}, scn={val:1755100947600130001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161920778323, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:23084275, ret_code_:0}}, next_to_submit_lsn_:{lsn:8461098908}, next_to_submit_scn_:{val:1755100947600130001, v:0}, base_lsn_:{lsn:7515734016}, base_scn_:{val:1753840867171901045, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6e05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:8461098908}, end_lsn:{lsn:8461098908}, read_buf:{buf_len_:3747840, buf_:0x7f98f6e05000}, block_size:67104768, log_storage_:0x7f99297a3bf0, log_storage_:{log_tail:{lsn:8461098908}, readable_log_tail:{lsn:8461098908}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8455200768}, min_scn:{val:1755160384851871002, v:0}, curr_block_id:126, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:215, min_block_id:93, max_block_id:127, curr_writable_block_id:126}, logical_block_size_:67104768, curr_block_writable_size_:61206628, block_header_serialize_buf_:0x7f99297a48b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:198, read_io_size:699822080, read_disk_cost_ts:4178035, start_lsn:{lsn:7515734016}}}}}}) [2025-08-14 09:02:20.793823] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=91] get wrs ts(ls_id={id:1}, delta=421018339, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:20.793838] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:20.793910] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EEAD-0-0] [lt=304][errcode=0] cost too much time:mds_tenant_service.cpp:int oceanbase::storage::mds::ObTenantMdsTimer::try_gc_mds_table_(oceanbase::storage::ObTablet &), (this=|threshold=5.00ms|start at 09:02:20.782|394=11.67ms|total=11.69ms) [2025-08-14 09:02:20.793936] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EEAD-0-0] [lt=24][errcode=0] cost too much time:mds_tenant_service.cpp:int oceanbase::storage::mds::ObTenantMdsTimer::process_with_tablet_(oceanbase::storage::ObTablet &), (this=|threshold=10.00ms|start at 09:02:20.782|315=0us|317=1us|319=1us|total=11.71ms) [2025-08-14 09:02:20.794038] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EEAD-0-0] [lt=11] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-08-14 09:02:20.805277] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=2537][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:20.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:20.819266] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98919b5690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:20.802, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:20.819314] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f9891970990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:20.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:20.819330] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728f9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98f7292010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:20.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:20.829327] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98f72eae10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:20.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:20.838622] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=22] PNIO [ratelimit] time: 1755162140838620, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007617, add_bytes: 0 [2025-08-14 09:02:20.839414] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98f739f290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:20.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:20.843892] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=1187] tx gc loop thread is running(MTL_ID()=1001) [2025-08-14 09:02:20.843917] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=24] try gc retain ctx [2025-08-14 09:02:20.843923] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=6] try to retry start_working [2025-08-14 09:02:20.844243] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1126] tx gc loop thread is running(MTL_ID()=1) [2025-08-14 09:02:20.844267] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=24] try gc retain ctx [2025-08-14 09:02:20.844277] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] try to retry start_working [2025-08-14 09:02:20.845075] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:20.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:20.851056] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:20.875377] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=825344][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:02:20.879664] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98f7289e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:20.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:20.879693] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98f7288f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:20.866, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:20.891255] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:20.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:20.945144] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:20.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:20.951074] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:20.991148] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:20.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:20.991317] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:20.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:21.028517] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:21.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:21.040959] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=15][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:02:21.042361] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [571][T1_ArcSender][T1][YB42AC4D0007-00063C495BA99C16-0-0] [lt=13][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:02:21.046315] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:20.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:21.051081] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:21.146402] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:21.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:21.149578] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=697056] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:02:21.151140] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:21.242705] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:21.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:21.244611] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:21.244631] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162141244607], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:21.244647] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162141244607], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:21.244672] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=22] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162141244607]}) [2025-08-14 09:02:21.244682] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162141244607]}) [2025-08-14 09:02:21.244695] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:21.244700] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162141244694], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:21.244709] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162141244694]}) [2025-08-14 09:02:21.244715] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162141244694]}) [2025-08-14 09:02:21.244893] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:21.244902] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:21.244913] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162141244890], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:21.244922] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162141244890]}) [2025-08-14 09:02:21.246473] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:21.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:21.249639] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:21.249661] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:02:21.249677] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:21.249686] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:21.249696] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:02:21.249711] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:21.249720] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:21.249737] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:21.249748] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:01:49.731, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:01:42.751], idx=659, next_allow_print_ts="09:02:19.731") [2025-08-14 09:02:21.249759] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:21.249773] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:21.249783] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:21.249793] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:21.249803] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:21.249968] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:21.249980] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:21.249990] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:02:21.251211] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:21.252764] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:21.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:21.270105] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:21.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:21.273568] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:21.273587] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=17] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3889) [2025-08-14 09:02:21.273594] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:21.273605] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=6] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:21.273612] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=6] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3890) [2025-08-14 09:02:21.300323] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:21.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:21.346744] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:21.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:21.351244] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:21.446810] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:21.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:21.451322] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:21.494264] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:21.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:21.528487] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:21.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:21.546871] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:21.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:21.551396] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:21.646943] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:21.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:21.651464] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:21.669458] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:21.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:20.787466] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1025915] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [2025-08-14 09:02:21.714744] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=925866] current log_task status(palf_id=1001, self="172.77.0.7:2882", log_id=8571657, log_task={header:{begin_lsn:{lsn:7323686009}, end_lsn:{lsn:7323686131}, log_id:8571657, min_scn:{val:1755161749741193000, v:0}, max_scn:{val:1755161749741193000, v:0}, data_len:66, proposal_id:49, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:7323686009}, data_checksum:2222604892, accum_checksum:1531236791, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1755161749745909, freeze_ts:1755161749745914, submit_ts:1755161749745940, flushed_ts:1755161785758706, gen_to_freeze cost time:5, gen_to_submit cost time:31, submit_to_flush cost time:36012766}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [2025-08-14 09:02:21.715466] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=41] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:02:21.715484] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:02:21.719256] INFO [STORAGE.TRANS] amend (ob_tenant_weak_read_server_version_mgr.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [WRS] tenant has no valid partition on current server, generate weak read snapshot version based on current timestamp(tenant_id=1002, epoch_tstamp_=1755162140745966, total_part_count_=2, valid_inner_part_count_=0, valid_user_part_count_=0, version_={val:1755161758221798000, v:0}, base_version_when_no_valid_partition={val:1755161758221798000, v:0}) [2025-08-14 09:02:21.719326] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119812272, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-08-14 09:02:21.719354] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get_max_applied_scn(scn={val:1755161749741193000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:50, palf_committed_end_lsn_:{lsn:1604090090}, palf_committed_end_scn_:{val:1755161749741193000, v:0}, last_check_scn_:{val:1755161749741193001, v:0}, max_applied_cb_scn_:{val:1755161749741193000, v:0}}) [2025-08-14 09:02:21.719386] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=391976413, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:21.719399] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:21.720025] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=396989782, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:21.720036] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:21.720059] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=391976413, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:21.720069] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:21.720660] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1001}, delta=396989782, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:21.721471] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [992][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F55-0-0] [lt=36][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1002, svr="172.77.0.7:2882", version={val:1755161758221798000, v:0}, valid_part_count=0, total_part_count=2, generate_timestamp=1755162141719288) [2025-08-14 09:02:21.721492] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [992][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F55-0-0] [lt=20][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1002, req={req_server:"172.77.0.7:2882", version:{val:1755161758221798000, v:0}, valid_part_count:0, total_part_count:2, generate_timestamp:1755162141719288}, twrs={inited:true, tenant_id:1002, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162141719339}, server_version_for_stat:{version:{val:18446744073709551615, v:3}, total_part_count:1, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162141720054}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:21.721593] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [316][pnio1][T1002][YB42AC4D0007-00063C49612B8F55-0-0] [lt=26][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1002, delta=933764, last_succ_cluster_heartbeat_tstamp_=1755162140787828) [2025-08-14 09:02:21.091404] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:20.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:21.722912] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=630548][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:21.914|207=631.50ms|210=0us|222=2us|total=631.53ms) [2025-08-14 09:02:21.722937] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=-4389] timer task use too much time(end=1755162141722937, start=1755162141091402, delta=631535) [2025-08-14 09:02:21.469477] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] EASY SLOW: thread: 7f99542fb300, lastrun: 1755162115.752503 cost: 25.716965 loop:26416, slowcnt: 205 [2025-08-14 09:02:21.725051] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:21.333291] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98f7281b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:21.725602] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=391713][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:21.333|207=392.31ms|210=0us|222=2us|total=392.33ms) [2025-08-14 09:02:21.011246] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:21.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:21.731640] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=719301][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:21.112|207=720.39ms|210=0us|222=2us|total=720.42ms) [2025-08-14 09:02:21.731680] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4389] timer task use too much time(end=1755162141731680, start=1755162141011243, delta=720437) [2025-08-14 09:02:21.731694] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=726205, expected_run_ts_us_=1755162141005487, ObClockGenerator::getRealClock()=1755162141731693, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:21.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:21.731714] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:21.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:21.720670] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:21.731729] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=642412, expected_run_ts_us_=1755162141089316, ObClockGenerator::getRealClock()=1755162141731729, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:21.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:21.731752] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=527455, expected_run_ts_us_=1755162141204297, ObClockGenerator::getRealClock()=1755162141731752, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:21.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:21.720852] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:21.732128] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11256][errcode=0] epoll handler too much time :11284 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:21.732230] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [359][MetaLocAsyncUp0][T0][YB42AC4D0007-00063C495C199FF0-0-0] [lt=2984809][errcode=-4012] get all ls info by persistent_ls_ failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1001) [567]OB_MALLOC COST TOO MUCH TIME, cost_time=869475, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:02:21.733104] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get_max_applied_scn(scn={val:1755161547738516001, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:50, palf_committed_end_lsn_:{lsn:5282737655}, palf_committed_end_scn_:{val:1755161547738516001, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161547738516001, v:0}}) [2025-08-14 09:02:21.733136] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27] get_min_unreplayed_log_info(lsn={lsn:5282737655}, scn={val:1755100947825033001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161624848121, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:13063726, ret_code_:0}}, next_to_submit_lsn_:{lsn:5282737655}, next_to_submit_scn_:{val:1755100947825033001, v:0}, base_lsn_:{lsn:4831543296}, base_scn_:{val:1755063445733337001, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7605000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:5282737655}, end_lsn:{lsn:5282737655}, read_buf:{buf_len_:3747840, buf_:0x7f98f7605000}, block_size:67104768, log_storage_:0x7f993e529bf0, log_storage_:{log_tail:{lsn:5282737655}, readable_log_tail:{lsn:5282737655}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:5234171904}, min_scn:{val:1755152244802535004, v:0}, curr_block_id:78, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:206, min_block_id:42, max_block_id:79, curr_writable_block_id:78}, logical_block_size_:67104768, curr_block_writable_size_:18539017, block_header_serialize_buf_:0x7f993e52a8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:85, read_io_size:301367296, read_disk_cost_ts:2364604, start_lsn:{lsn:4831543296}}}}}}) [2025-08-14 09:02:21.733805] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=77] get wrs ts(ls_id={id:1}, delta=593993069, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:21.733828] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:21.725627] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4389] timer task use too much time(end=1755162141725627, start=1755162141333289, delta=392338) [2025-08-14 09:02:21.734544] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8893][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72756b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98919b5510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:21.734580] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98f7283b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.724, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:21.734595] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a36b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:417, func_shared_ptr_.ptr:0x7f98f7283c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:21.734640] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:155) [407][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=1003171] unlock session of tenant(conn.sessid_=3221590956, proxy_sessid=0, conn.tenant_id_=1002) [2025-08-14 09:02:21.732522] INFO [STORAGE.TRANS] refresh_elr_tenant_config_ (ob_tx_elr_util.cpp:42) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=4] refresh tenant config success(tenant_id=1001, *this={last_refresh_ts:1755162141731585, can_tenant_elr:true}) [2025-08-14 09:02:21.737590] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:20.792176] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:275) [14905][T1001_TimerWK8_][T1001][Y0-0000000000000000-0-0] [lt=14] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=13) [2025-08-14 09:02:21.737698] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [14905][T1001_TimerWK8][T1001][Y0-0000000000000000-0-0] [lt=944089][errcode=-4389] timer task too much delay in priority queue(token={this:0x7f9931bb3298, timer:0x7f99333a1708, task:0x7f99333a1730, task_type:"N9oceanbase7storage10checkpoint17ObTabletGCService18ObTabletChangeTaskE", scheduled_time:1755162051760578, pushed_time:1755162052836249, delay:5000000}, delay_in_priority_queue=1075671, threashold=1000000) [2025-08-14 09:02:21.737716] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [14905][T1001_TimerWK8][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f9931bb3298, timer:0x7f99333a1708, task:0x7f99333a1730, task_type:"N9oceanbase7storage10checkpoint17ObTabletGCService18ObTabletChangeTaskE", scheduled_time:1755162051760578, pushed_time:1755162052836249, delay:5000000}, delay_in_thread_pool=87948542, threashold=500000) [2025-08-14 09:02:21.737750] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:243) [14905][T1001_TimerWK8_Flush][T1001][Y0-0000000000000000-0-0] [lt=12] ====== traversal_flush timer task ====== [2025-08-14 09:02:21.273617] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:21.737982] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=463493] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:21.738002] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:227) [1031][PxTargetMgr0][T1002][Y0-0000000000000000-0-0] [lt=15] refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3891) [2025-08-14 09:02:21.738013] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:21.738225] INFO [OCCAM] ObOccamTimerTaskRAIIHandle (ob_occam_timer.h:363) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19] task handle constructed(*this={this:0x7f9917455510, task:NULL, is_inited:false, is_running:false}) [2025-08-14 09:02:21.738332] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] task executed too delay(delay_time=976432, *p_task_={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:0, time_interval:250.00ms, expected_run_time:09:02:21.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:21.719018] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] [WRS] update tenant weak read server version(tenant_id=1001, server_version={version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162140793123}, version_delta=421943838) [2025-08-14 09:02:21.740330] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20634] get wrs ts(ls_id={id:1}, delta=421956406, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:21.740348] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:21.740497] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [758][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB909B-0-0] [lt=27][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1001, svr="172.77.0.7:2882", version={val:1755161719775179002, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162141739686) [2025-08-14 09:02:21.740522] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [758][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB909B-0-0] [lt=19][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1001, req={req_server:"172.77.0.7:2882", version:{val:1755161719775179002, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162141739686}, twrs={inited:true, tenant_id:1001, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162141739713}, server_version_for_stat:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162141739713}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:21.740618] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [317][pnio1][T1001][YB42AC4D0007-00063C4960CB909B-0-0] [lt=11][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1001, delta=960510, last_succ_cluster_heartbeat_tstamp_=1755162140780108) [2025-08-14 09:02:21.741935] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=987] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:21.742443] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=954] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:21.745705] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:21.745758] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=6] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:21.745774] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162141745738], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:21.745790] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162141745738], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:21.745805] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162141745738]}) [2025-08-14 09:02:21.745819] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=11] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:21.745827] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162141745818], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:21.745840] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162141745818]}) [2025-08-14 09:02:21.746026] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:21.746273] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:21.746403] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919911b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196cc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:21.746429] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d40b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:21.746444] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:21.746457] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:21.746470] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:21.746484] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:21.746788] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] get wrs ts(ls_id={id:1}, delta=593993069, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:21.746808] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:21.746839] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:21.746864] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:21.747166] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=421956406, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:21.747183] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:21.747666] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=3889] gc stale ls task succ [2025-08-14 09:02:21.735406] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:21.747873] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12447][errcode=0] epoll handler too much time :12488 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:21.748178] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11044] get wrs ts(ls_id={id:1}, delta=392006026, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:21.748244] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=51] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:21.748868] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=397019395, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:21.748879] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:21.749259] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:21.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:21.739276] WDIAG [SQL] open_plan (ob_result_set.cpp:104) [368][VTblLocAsyncUp0][T1][YB42AC4D0007-00063C4968299C62-0-0] [lt=993233][errcode=-4012] query is timeout(ret=-4012, timeout_ts=1755162139769916, start_time=1755162139723815) [2025-08-14 09:02:21.750013] WDIAG [SQL] open (ob_result_set.cpp:137) [368][VTblLocAsyncUp0][T1][YB42AC4D0007-00063C4968299C62-0-0] [lt=10718][errcode=-4012] execute plan failed(ret=-4012) [2025-08-14 09:02:21.750106] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [368][VTblLocAsyncUp0][T1][YB42AC4D0007-00063C4968299C62-0-0] [lt=90][errcode=-4012] open result set failed(ret=-4012) [2025-08-14 09:02:21.750619] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:21.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:21.750659] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:21.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:21.751162] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:21.751226] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:21.751288] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:02:21.751320] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:21.751333] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:21.751346] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:02:21.751384] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:21.751397] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:21.751417] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:21.751430] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:01:49.731, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:01:42.751], idx=659, next_allow_print_ts="09:02:19.731") [2025-08-14 09:02:21.751443] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:21.751459] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:21.751472] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:21.751485] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:21.751498] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:21.751668] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:21.751684] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:21.751697] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:02:21.751792] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:21.753139] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:02:21.753181] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:21.753891] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:21.756285] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:274) [272][TimerWK0_TimezoneMgr][T1002][YB42AC4D0007-00063C4960D9F14C-0-0] [lt=966476][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=1) [2025-08-14 09:02:21.756321] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:953) [272][TimerWK0_TimezoneMgr][T1002][YB42AC4D0007-00063C4960D9F14C-0-0] [lt=28][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2025-08-14 09:02:21.756336] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [272][TimerWK0_TimezoneMgr][T1002][YB42AC4D0007-00063C4960D9F14C-0-0] [lt=20][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:21.756409] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:274) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE3-0-0] [lt=71][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=487) [2025-08-14 09:02:21.756423] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:953) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE3-0-0] [lt=13][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2025-08-14 09:02:21.756430] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE3-0-0] [lt=7][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:21.756479] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:274) [784][T1002_TBalance][T1][YB42AC4D0007-00063C4964C9A641-0-0] [lt=24163][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=491) [2025-08-14 09:02:21.756504] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:953) [784][T1002_TBalance][T1][YB42AC4D0007-00063C4964C9A641-0-0] [lt=25][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2025-08-14 09:02:21.756492] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.744, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:21.756516] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [784][T1002_TBalance][T1][YB42AC4D0007-00063C4964C9A641-0-0] [lt=7] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:21.756521] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aaf90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:21.756534] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [784][T1002_TBalance][T1][YB42AC4D0007-00063C4964C9A641-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:21.756535] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aba10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:21.756547] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [784][T1002_TBalance][T1][YB42AC4D0007-00063C4964C9A641-0-0] [lt=13][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:21.756548] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:21.756562] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196cb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:21.756573] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:21.757700] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [784][T1002_TBalance][T1][YB42AC4D0007-00063C4964C9A641-0-0] [lt=12] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:21.757720] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [784][T1002_TBalance][T1][YB42AC4D0007-00063C4964C9A641-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:21.757733] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [784][T1002_TBalance][T1][YB42AC4D0007-00063C4964C9A641-0-0] [lt=12][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:21.746906] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] finish update min start scn(max_decided_scn={val:1755161547738516001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1755161493768447002, v:0}, keep_alive_scn_:{val:1755161493768447002, v:0}, update_ts_:1755162141731585}) [2025-08-14 09:02:21.759329] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [784][T1002_TBalance][T1][YB42AC4D0007-00063C4964C9A641-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:21.759353] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [784][T1002_TBalance][T1][YB42AC4D0007-00063C4964C9A641-0-0] [lt=24][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:21.759366] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [784][T1002_TBalance][T1][YB42AC4D0007-00063C4964C9A641-0-0] [lt=12][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:21.741258] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=2881] finish update min start scn(max_decided_scn={val:1755161719775179002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1755161287778691002, v:0}, keep_alive_scn_:{val:1755161287778691002, v:0}, update_ts_:1755162141731585}) [2025-08-14 09:02:21.732267] WDIAG [SHARE.LOCATION] renew_location_for_tenant (ob_ls_location_service.cpp:815) [359][MetaLocAsyncUp0][T0][YB42AC4D0007-00063C495C199FF0-0-0] [lt=33][errcode=-4012] fail to get all ls info(ret=-4012, ret="OB_TIMEOUT", tenant_id=1001, ls_infos=[]) [2025-08-14 09:02:21.763637] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=17] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:21.756470] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE3-0-0] [lt=39] get number(ret=-4038, service_type_=0, range=1, base_id=1755162141756052000, start_id=0, end_id=94890010473799) [381]OB_MALLOC COST TOO MUCH TIME, cost_time=963114, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=963106], seq:[0] [381]OB_MALLOC COST TOO MUCH TIME, cost_time=963394, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=56, WASH_BLOCK_END=185, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=963148], seq:[0, 1, 2, 3, 4] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=963156, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=8070], seq:[0, 1] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=963228, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=8040], seq:[0, 1] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=897432, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [729]OB_MALLOC COST TOO MUCH TIME, cost_time=789151, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [2025-08-14 09:02:21.765774] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE3-0-0] [lt=9288][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:21.765788] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE3-0-0] [lt=14][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:21.765803] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE3-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162141746023]}) [2025-08-14 09:02:21.756347] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [272][TimerWK0_TimezoneMgr][T1002][YB42AC4D0007-00063C4960D9F14C-0-0] [lt=11] get number(ret=-4038, service_type_=0, range=1, base_id=1755162141756052000, start_id=0, end_id=94890010473799) [2025-08-14 09:02:21.765830] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [272][TimerWK0_TimezoneMgr][T1002][YB42AC4D0007-00063C4960D9F14C-0-0] [lt=9469][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:21.765841] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [272][TimerWK0_TimezoneMgr][T1002][YB42AC4D0007-00063C4960D9F14C-0-0] [lt=11][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:21.766602] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [272][TimerWK0_TimezoneMgr][T1002][YB42AC4D0007-00063C4960D9F14C-0-0] [lt=10][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:21.766588] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.751, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:21.766609] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [272][TimerWK0_TimezoneMgr][T1002][YB42AC4D0007-00063C4960D9F14C-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:21.766617] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [272][TimerWK0_TimezoneMgr][T1002][YB42AC4D0007-00063C4960D9F14C-0-0] [lt=6][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:21.766699] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE3-0-0] [lt=12][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:21.766704] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE3-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:21.766710] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE3-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:21.767296] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [272][TimerWK0_TimezoneMgr][T1002][YB42AC4D0007-00063C4960D9F14C-0-0] [lt=6][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:21.767308] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [272][TimerWK0_TimezoneMgr][T1002][YB42AC4D0007-00063C4960D9F14C-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:21.767316] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [272][TimerWK0_TimezoneMgr][T1002][YB42AC4D0007-00063C4960D9F14C-0-0] [lt=8][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:21.767337] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE3-0-0] [lt=5][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:21.767343] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE3-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:21.767349] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE3-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:21.768244] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE3-0-0] [lt=5][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:21.768254] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE3-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:21.768261] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE3-0-0] [lt=7][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:21.768284] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [272][TimerWK0_TimezoneMgr][T1002][YB42AC4D0007-00063C4960D9F14C-0-0] [lt=15][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:21.768290] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [272][TimerWK0_TimezoneMgr][T1002][YB42AC4D0007-00063C4960D9F14C-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:21.768295] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [272][TimerWK0_TimezoneMgr][T1002][YB42AC4D0007-00063C4960D9F14C-0-0] [lt=6][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:21.768859] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE3-0-0] [lt=6][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:21.768875] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [272][TimerWK0_TimezoneMgr][T1002][YB42AC4D0007-00063C4960D9F14C-0-0] [lt=6][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:21.768880] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [272][TimerWK0_TimezoneMgr][T1002][YB42AC4D0007-00063C4960D9F14C-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:21.768886] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [272][TimerWK0_TimezoneMgr][T1002][YB42AC4D0007-00063C4960D9F14C-0-0] [lt=6][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:21.770564] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1866] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:21.768866] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE3-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:21.770275] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [272][TimerWK0_TimezoneMgr][T1002][YB42AC4D0007-00063C4960D9F14C-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:21.771082] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=11] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:21.776309] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:02:21.776386] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:02:21.776441] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:02:21.776489] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:02:21.776506] INFO [COMMON] print_io_status (ob_io_manager.cpp:2639) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4063, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:3310, total:3314) [2025-08-14 09:02:21.776529] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, param_config_:{memory_limit:536870912, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=865568, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:02:21.776566] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2826) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=33] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4090, [delay/us]: prepare:0, schedule:0, submit:600000000, device:3310, total:3314) [2025-08-14 09:02:21.776586] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:02:21.776906] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=55][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:21.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:21.777063] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:21.777458] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:21.777512] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:21.787123] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919acf10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:21.787149] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:21.797222] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919abe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:21.797271] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:21.797282] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199fe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919acd90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:21.797300] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e80b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:21.807837] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:21.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:21.815538] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=0] get_fetch_log_dst failed or invalid(ret=0, palf_id=1, self="172.77.0.7:2882", fetch_log_dst="172.77.0.7:2882") [2025-08-14 09:02:21.815567] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=0] get_fetch_log_dst failed or invalid(ret=0, palf_id=1001, self="172.77.0.7:2882", fetch_log_dst="172.77.0.7:2882") [2025-08-14 09:02:21.844488] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=422067862, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:21.844519] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=27] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:21.850254] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=62][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:21.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:21.950317] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:21.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:21.961255] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:21.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:22.027477] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:22.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:22.050385] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:21.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:22.061316] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:21.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:22.150453] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:22.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:22.161384] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:22.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:22.250525] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:22.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:22.261433] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:22.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:22.279028] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:22.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:22.309201] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:22.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:22.350598] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:22.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:22.361503] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:22.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:22.450662] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:22.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:22.461565] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:22.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:22.470403] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=409501][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162141.783258 cost: 0.687142 loop:26486, slowcnt: 6 [2025-08-14 09:02:22.520729] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:22.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:22.550725] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:22.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:22.561634] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:22.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:22.650778] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:22.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:22.661708] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:22.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:22.716058] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:02:22.716086] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:02:21.776596] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:21.797311] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a31b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ede10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:22.728043] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=929321][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:21.797|207=930.73ms|210=0us|222=2us|total=930.76ms) [2025-08-14 09:02:22.728073] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4389] timer task use too much time(end=1755162142728072, start=1755162141797310, delta=930762) [2025-08-14 09:02:21.823023] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:21.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:22.653788] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=125][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571630c0, caller:mds_tenant_service.cpp:mtl_start:117, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_145, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571633a0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:22.720129] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:22.721274] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:21.746037] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=10] get number(ret=-4038, service_type_=0, range=1, base_id=1755162141731585000, start_id=0, end_id=140295968341544) [2025-08-14 09:02:22.730739] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=983209][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:22.730759] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162141746023], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:22.730775] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162141746023], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:22.730790] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162141746023]}) [2025-08-14 09:02:22.730802] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts statistics(tenant_id=1002, gts_rpc_cnt=0, get_gts_cache_cnt=70, get_gts_with_stc_cnt=9, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:02:22.730812] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1002}) [2025-08-14 09:02:22.730820] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1002}) [2025-08-14 09:02:21.786906] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:22.731072] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=942730] temporary file module use no blocks [2025-08-14 09:02:22.731081] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:22.731093] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:22.731107] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623043226305, flush_io_finished_ret_:0, flush_io_finished_round_:26316, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:22.728148] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=74244][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:1, time_interval:3.00s, expected_run_time:09:02:22.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:21.766526] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:21.790208] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=12] gc stale ls task succ [2025-08-14 09:02:22.728118] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=903725][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:21.823|207=905.10ms|210=0us|222=2us|total=905.11ms) [2025-08-14 09:02:22.238084] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:22.732670] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=493836] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3892) [2025-08-14 09:02:22.732684] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:22.732667] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=4540][errcode=-4389] timer task use too much time(end=1755162142732666, start=1755162141823020, delta=909646) [2025-08-14 09:02:22.732704] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=6] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:22.732711] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3893) [2025-08-14 09:02:22.732716] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:22.732902] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:22.732909] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3894) [2025-08-14 09:02:22.732914] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:22.732948] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:22.733461] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=594989498, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:22.733537] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=64] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:22.733594] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=32] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161101770923, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755161572801635) [2025-08-14 09:02:22.734170] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] get wrs ts(ls_id={id:1}, delta=594989498, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:22.734183] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [784]OB_MALLOC COST TOO MUCH TIME, cost_time=967832, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=7855], seq:[0, 1] [2025-08-14 09:02:21.766720] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [784][T1002_TBalance][T1][YB42AC4D0007-00063C4964C9A641-0-0] [lt=19] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:22.736039] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [784][T1002_TBalance][T1][YB42AC4D0007-00063C4964C9A641-0-0] [lt=967852][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:22.736054] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [784][T1002_TBalance][T1][YB42AC4D0007-00063C4964C9A641-0-0] [lt=15][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:22.736069] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [784][T1002_TBalance][T1][YB42AC4D0007-00063C4964C9A641-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162141745738]}) [2025-08-14 09:02:22.728087] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=921885, expected_run_ts_us_=1755162141806200, ObClockGenerator::getRealClock()=1755162142728086, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.806, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:22.736455] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8353][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.806, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:22.736491] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:22.736524] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] task scheduled too delay(scheduled_delay=929044, expected_run_ts_us_=1755162141807444, ObClockGenerator::getRealClock()=1755162142736523, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:22.736554] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:22.736568] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=932187, expected_run_ts_us_=1755162141804380, ObClockGenerator::getRealClock()=1755162142736568, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.804, task_priority:1, with_handle_protected:0}) [663]OB_MALLOC COST TOO MUCH TIME, cost_time=960840, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=73, WASH_BLOCK_END=960767], seq:[0, 1, 2] [832]OB_MALLOC COST TOO MUCH TIME, cost_time=949835, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=39, WASH_BLOCK_END=949796], seq:[0, 1, 2] [857]OB_MALLOC COST TOO MUCH TIME, cost_time=893179, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=43, WASH_BLOCK_END=893135], seq:[0, 1, 2] [694]OB_MALLOC COST TOO MUCH TIME, cost_time=855950, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=44, WASH_BLOCK_END=855906], seq:[0, 1, 2] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=744698, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=46, WASH_BLOCK_END=744652], seq:[0, 1, 2] [2025-08-14 09:02:22.735199] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=30] get wrs ts(ls_id={id:1}, delta=422952835, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:22.736721] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1517] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:21.771417] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [272][TimerWK0_TimezoneMgr][T1002][YB42AC4D0007-00063C4960D9F14C-0-0] [lt=1139][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:21.798247] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=12] [sql nio session](*s={this:0x7f98f72b3030, session_id:3221562808, trace_id:YB42AC4D0007-00063C4965DA1F2E-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:267}, err:0, last_decode_time:1755161748766816, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1890, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:22.741050] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=941377] [sql nio session](*s={this:0x7f98f72ab030, session_id:3221589435, trace_id:YB42AC4D0007-00063C4965DA1F35-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:246}, err:0, last_decode_time:1755161762721798, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:2456, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:22.741066] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=15] [sql nio session](*s={this:0x7f98f72a9030, session_id:3221590528, trace_id:YB42AC4D0007-00063C4965DA1F34-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:250}, err:0, last_decode_time:1755161762715415, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:205, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:22.741077] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=11] [sql nio session](*s={this:0x7f98f72b5030, session_id:3221590961, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:10, sql_initiative_shutdown:false, reader:{fd:278}, err:5, last_decode_time:1755162075745307, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:22.741901] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:21.771365] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE3-0-0] [lt=2495][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [477]OB_MALLOC COST TOO MUCH TIME, cost_time=961679, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=64, WASH_BLOCK_END=961614], seq:[0, 1, 2] [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=982295, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=43, WASH_BLOCK_END=982252], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=930510, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=57, WASH_BLOCK_END=930453], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=623781, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=41, WASH_BLOCK_END=623739], seq:[0, 1, 2] [683]OB_MALLOC COST TOO MUCH TIME, cost_time=309623, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=62, WASH_BLOCK_END=309561], seq:[0, 1, 2] [797]OB_MALLOC COST TOO MUCH TIME, cost_time=187345, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=80, WASH_BLOCK_END=187264], seq:[0, 1, 2] [717]OB_MALLOC COST TOO MUCH TIME, cost_time=990098, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=50, WASH_BLOCK_END=986450, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=3597], seq:[0, 1, 2, 3, 4] [513]OB_MALLOC COST TOO MUCH TIME, cost_time=777225, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=42, WASH_BLOCK_END=775095, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=2087], seq:[0, 1, 2, 3, 4] [857]OB_MALLOC COST TOO MUCH TIME, cost_time=990019, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=990012], seq:[0] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=1952202, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=36, WASH_BLOCK_END=961176, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=35448], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:21.779892] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1952235, owner: ObLog, click_count: 5, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=1952217, ALLOC_END=5, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [399]OB_MALLOC COST TOO MUCH TIME, cost_time=984831, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=29156, WASH_BLOCK_END=955675], seq:[0, 1, 2] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=989462, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=33606, WASH_BLOCK_END=955790, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=66], seq:[0, 1, 2, 3, 4] [383]OB_MALLOC COST TOO MUCH TIME, cost_time=983681, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=27889, WASH_BLOCK_END=955792], seq:[0, 1, 2] [2025-08-14 09:02:22.251833] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:22.742286] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=489705][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:02:22.742324] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:22.742337] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:22.742351] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:02:22.742369] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:22.742400] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:22.742426] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:22.742438] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:01:49.731, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:01:42.751], idx=659, next_allow_print_ts="09:02:19.731") [2025-08-14 09:02:22.742451] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:22.742467] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:22.742481] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:22.742493] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:22.742510] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:21.791364] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=21] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:22.742688] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:22.742728] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=949914] temporary file module use no blocks [2025-08-14 09:02:22.742746] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:22.742749] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:22.742759] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:22.742786] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623043230411, flush_io_finished_ret_:0, flush_io_finished_round_:26283, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:22.744193] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [272][TimerWK0_TimezoneMgr][T1002][YB42AC4D0007-00063C4960D9F14C-0-0] [lt=968131][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:22.744206] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [272][TimerWK0_TimezoneMgr][T1002][YB42AC4D0007-00063C4960D9F14C-0-0] [lt=14][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:22.736601] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:22.744239] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7613][errcode=0] task scheduled too delay(scheduled_delay=943790, expected_run_ts_us_=1755162141800439, ObClockGenerator::getRealClock()=1755162142744230, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.800, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:22.744264] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.800, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:22.744279] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=938934, expected_run_ts_us_=1755162141805344, ObClockGenerator::getRealClock()=1755162142744279, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728e330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7290690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.805, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:22.744812] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=28] get wrs ts(ls_id={id:1}, delta=422952835, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:22.744828] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:22.744863] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [272][TimerWK0_TimezoneMgr][T1002][YB42AC4D0007-00063C4960D9F14C-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:22.747815] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [272][TimerWK0_TimezoneMgr][T1002][YB42AC4D0007-00063C4960D9F14C-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:22.725407] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=947370] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:22.747889] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22441] [IO STATUS SERVER] net_in=0kB/s, net_out=0kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=1280000kB/s [2025-08-14 09:02:22.748396] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [272][TimerWK0_TimezoneMgr][T1002][YB42AC4D0007-00063C4960D9F14C-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:22.748937] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [272][TimerWK0_TimezoneMgr][T1002][YB42AC4D0007-00063C4960D9F14C-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:22.749274] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=28] gc stale ls task succ [2025-08-14 09:02:22.751546] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:22.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:21.885652] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=5][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb3df8, timer:0x7f99333cb180, task:0x7f99333cb1f0, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755161959782853, pushed_time:0, delay:1000000}, running_task_token={this:0x7f9931bb3ed8, timer:0x7f99333cb180, task:0x7f99333cb208, task_type:"N9oceanbase8observer18ObTableLoadService25ObClientTaskAutoAbortTaskE", scheduled_time:1755161959782824, pushed_time:1755162045729074, delay:1000000}, total_delay=182102798, threashold=1000000, tenant_id=1001) [2025-08-14 09:02:22.736658] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1832) [784][T1002_TBalance][T1][YB42AC4D0007-00063C4964C9A641-0-0] [lt=10][errcode=-4012] get gts fail(ret=-4012, timeout_us=43102, request_time=[mts=1755162141756478]) [2025-08-14 09:02:22.755396] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:22.759672] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=872695][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb2d58, timer:0x7f9966a57e50, task:0x7f993335ebf0, task_type:"N9oceanbase7obmysql15ObEliminateTaskE", scheduled_time:1755162141284761, pushed_time:0, delay:500000}, running_task_token={this:0x7f9931bb2e38, timer:0x7f9966a57e50, task:0x7f993335ec40, task_type:"N9oceanbase7obmysql20ObConstructQueueTaskE", scheduled_time:1755162047260468, pushed_time:1755162140784829, delay:500000}, total_delay=1474910, threashold=1000000, tenant_id=1001) [2025-08-14 09:02:22.744292] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728e330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7290690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.805, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:22.751929] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [272][TimerWK0_TimezoneMgr][T1002][YB42AC4D0007-00063C4960D9F14C-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:21.997143] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:21.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:22.759705] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [272][TimerWK0_TimezoneMgr][T1002][YB42AC4D0007-00063C4960D9F14C-0-0] [lt=7763][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:22.759701] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15382][errcode=0] task scheduled too delay(scheduled_delay=955974, expected_run_ts_us_=1755162141803726, ObClockGenerator::getRealClock()=1755162142759701, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.803, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:22.759723] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.803, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:22.673613] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571634a0, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957163780, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:5.00s, expected_run_time:09:02:22.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:22.759773] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] task scheduled too delay(scheduled_delay=935122, expected_run_ts_us_=1755162141824616, ObClockGenerator::getRealClock()=1755162142759773, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.824, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:22.759785] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.824, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:22.759796] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=927462, expected_run_ts_us_=1755162141832333, ObClockGenerator::getRealClock()=1755162142759796, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.832, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:22.759806] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.832, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:22.759816] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=894886, expected_run_ts_us_=1755162141864930, ObClockGenerator::getRealClock()=1755162142759816, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ade10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.864, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:22.759826] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ade10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.864, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:22.759836] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=894746, expected_run_ts_us_=1755162141865089, ObClockGenerator::getRealClock()=1755162142759835, ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:22.760283] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1832) [272][TimerWK0_TimezoneMgr][T1002][YB42AC4D0007-00063C4960D9F14C-0-0] [lt=11][errcode=-4012] get gts fail(ret=-4012, timeout_us=1000000, request_time=[mts=1755162141756278]) [2025-08-14 09:02:22.753389] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:02:22.770461] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:22.759712] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=761413][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:21.997|207=762.57ms|210=0us|222=1us|total=762.60ms) [2025-08-14 09:02:22.772289] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=12555][errcode=-4389] timer task use too much time(end=1755162142772288, start=1755162141997141, delta=775147) [2025-08-14 09:02:22.759845] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:22.772324] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12457][errcode=0] task scheduled too delay(scheduled_delay=906901, expected_run_ts_us_=1755162141865422, ObClockGenerator::getRealClock()=1755162142772324, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919adc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:22.772345] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919adc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:22.772357] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=906815, expected_run_ts_us_=1755162141865541, ObClockGenerator::getRealClock()=1755162142772357, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919adb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:22.772367] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919adb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:22.760479] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=905838] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:02:22.772399] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11901] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:02:22.772412] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:22.772420] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:22.760503] INFO [SERVER] run (obmp_disconnect.cpp:79) [972][T1002_L0_G0][T1002][Y0-0000000000000000-0-0] [lt=4] free session successfully(sessid=3221590956) [2025-08-14 09:02:22.762545] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:22.772636] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:22.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:22.759756] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=86008][errcode=0] task scheduled too delay(scheduled_delay=204479, expected_run_ts_us_=1755162142555275, ObClockGenerator::getRealClock()=1755162142759755, ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:22.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:22.753254] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [726][T1001_ObXAHbWor][T1001][Y0-0000000000000000-0-0] [lt=13743] XA scheduler heartbeat task statistics(avg_time=1) [2025-08-14 09:02:22.774946] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=3533] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:22.772432] INFO [SERVER] run (obmp_disconnect.cpp:86) [972][T1002_L0_G0][T1002][Y0-0000000000000000-0-0] [lt=11909] mark session id unused(sessid=3221590956) [2025-08-14 09:02:22.773699] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=13919][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:22.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:00:56.773758] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [339][EventTimer][T0][Y0-0000000000000000-0-0] [lt=1001353][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:00:55.770|207=1.00s|210=0us|222=2us|total=86.99s) [2025-08-14 09:02:21.783269] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:22.772386] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=894274, expected_run_ts_us_=1755162141878111, ObClockGenerator::getRealClock()=1755162142772386, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72865b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.878, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:22.778112] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=5715][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72865b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.878, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:22.778133] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] task scheduled too delay(scheduled_delay=878293, expected_run_ts_us_=1755162141899839, ObClockGenerator::getRealClock()=1755162142778132, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728de30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7290b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.899, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:22.778144] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728de30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7290b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.899, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:22.722072] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=948938] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:22.778184] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=56025] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:22.778207] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=22] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:22.778215] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:22.778221] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:22.778228] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:22.778246] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=18] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:22.803984] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:22.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:22.833601] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=595095081, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=253080, succ_cluster_heartbeat_count=2195, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161000606550000, v:0}, local_cluster_delta=1142227047, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:02:22.848090] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:02:22.848136] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:02:22.851610] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:22.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:22.951679] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:22.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:23.051737] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:22.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:22.772310] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] task scheduled too delay(scheduled_delay=527675, expected_run_ts_us_=1755162142244633, ObClockGenerator::getRealClock()=1755162142772309, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:22.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:22.778252] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:22.778153] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=875298, expected_run_ts_us_=1755162141902855, ObClockGenerator::getRealClock()=1755162142778153, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.902, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.723798] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=944213][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:21.902, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.723833] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:22.778|207=945.68ms|210=0us|222=1us|total=945.69ms) [2025-08-14 09:02:23.723848] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] timer task use too much time(end=1755162143723847, start=1755162142778152, delta=945695) [2025-08-14 09:02:23.723862] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1612205, expected_run_ts_us_=1755162142111656, ObClockGenerator::getRealClock()=1755162143723861, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.111, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:22.829246] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] [sql nio session](*s={this:0x7f98f72d0030, session_id:3221589415, trace_id:YB42AC4D0007-00063C4965FA43DC-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:245}, err:0, last_decode_time:1755161751714366, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1662, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:23.723947] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=893348] [sql nio session](*s={this:0x7f98f72d5030, session_id:3221590200, trace_id:YB42AC4D0007-00063C4965FA43E0-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:248}, err:0, last_decode_time:1755161776777167, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1593, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:22.832749] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:22.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:22.848173] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:02:22.861840] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:22.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:23.724448] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=861299][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:22.861|207=862.61ms|210=0us|222=1us|total=862.64ms) [2025-08-14 09:02:23.724483] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=-4389] timer task use too much time(end=1755162143724483, start=1755162142861838, delta=862645) [2025-08-14 09:02:22.877827] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c0b90, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571c0e70, time_wheel:0x7f9959fef990, thread_pool:0x7f9959fef170, is_running:1, time_interval:3.00s, expected_run_time:09:02:22.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:23.724712] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=874514] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:02:23.724754] INFO [COMMON] print_io_status (ob_io_manager.cpp:2639) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:2102, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:14739, total:14745) [2025-08-14 09:02:23.732617] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=963297] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:22.874266] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:23.026257] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:23.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:23.472839] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162141.783258 cost: 1.689578 loop:26486, slowcnt: 16 [2025-08-14 09:02:23.151780] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:23.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:23.734322] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=581657][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:23.151|207=582.54ms|210=0us|222=2us|total=582.56ms) [2025-08-14 09:02:23.734344] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4389] timer task use too much time(end=1755162143734343, start=1755162143151777, delta=582566) [2025-08-14 09:02:23.678590] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=4344][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:23.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:23.721806] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=948058][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:22.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:21.779892] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=11] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:23.720698] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8028] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:23.723876] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.111, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.735117] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14396][errcode=0] epoll handler too much time :14439 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:23.731307] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=30] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:23.735144] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=3826] temporary file module use no blocks [2025-08-14 09:02:23.735139] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11231][errcode=0] task scheduled too delay(scheduled_delay=1004415, expected_run_ts_us_=1755162142730722, ObClockGenerator::getRealClock()=1755162143735138, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.735154] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:23.723983] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=889884][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:22.832|207=891.23ms|210=0us|222=2us|total=891.25ms) [2025-08-14 09:02:23.720622] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=7] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:02:23.724519] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=845410][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:22.877|207=846.69ms|210=0us|222=2us|total=846.71ms) [2025-08-14 09:02:23.721380] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6908] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:23.735207] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=10671][errcode=-4389] timer task use too much time(end=1755162143735206, start=1755162142877825, delta=857381) [2025-08-14 09:02:22.759673] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1844) [784][T1002_TBalance][T1][YB42AC4D0007-00063C4964C9A641-0-0] [lt=22979][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1755162141799154, request_time=[mts=1755162141756478]) [2025-08-14 09:02:23.724783] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, param_config_:{memory_limit:536870912, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=865568, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:02:23.735290] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2826) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10484] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:2091, [delay/us]: prepare:0, schedule:0, submit:600000000, device:14739, total:14745) [567]OB_MALLOC COST TOO MUCH TIME, cost_time=902126, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=973920, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=15992, WASH_BLOCK_END=957927], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=959913, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2091, WASH_BLOCK_END=957822], seq:[0, 1, 2] [683]OB_MALLOC COST TOO MUCH TIME, cost_time=960030, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2054, WASH_BLOCK_END=957976], seq:[0, 1, 2] [797]OB_MALLOC COST TOO MUCH TIME, cost_time=960007, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2022, WASH_BLOCK_END=957985], seq:[0, 1, 2] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=946732, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=34, WASH_BLOCK_END=946698], seq:[0, 1, 2] [723]OB_MALLOC COST TOO MUCH TIME, cost_time=928788, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=34, WASH_BLOCK_END=928753], seq:[0, 1, 2] [2025-08-14 09:02:22.151964] EDIAG try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1584240, owner: ObLog, click_count: 6, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=623798, ObMalloc#start=21, ObMalloc#end=959925, FREE_END=485], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x26499505 0x97074a3 0x26498fa9 0x7f9973e081cf 0x7f9973a39dd3 [651]OB_MALLOC COST TOO MUCH TIME, cost_time=928310, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=43, WASH_BLOCK_END=928267], seq:[0, 1, 2] [381]OB_MALLOC COST TOO MUCH TIME, cost_time=506761, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=58, WASH_BLOCK_END=506703], seq:[0, 1, 2] [1031]OB_MALLOC COST TOO MUCH TIME, cost_time=505046, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=50, WASH_BLOCK_END=504996], seq:[0, 1, 2] [2025-08-14 09:02:21.799338] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [477][T1_StandbySchem][T1][YB42AC4D0007-00063C495B6A0318-0-0] [lt=18][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1937254, owner: ObLog, click_count: 6, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=961741, ObMalloc#start=34, ObMalloc#end=973943, FREE_END=1524], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [729]OB_MALLOC COST TOO MUCH TIME, cost_time=901723, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:02:23.735314] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:02:23.735329] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:23.735339] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:23.735346] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=0kB/s, net_out=0kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=1280000kB/s [2025-08-14 09:02:22.760293] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1844) [272][TimerWK0_TimezoneMgr][T1002][YB42AC4D0007-00063C4960D9F14C-0-0] [lt=9][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1755162168698854, request_time=[mts=1755162141756278]) [2025-08-14 09:02:23.736897] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:23.737699] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31] get wrs ts(ls_id={id:1}, delta=595996702, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:23.737723] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:23.737754] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161101770923, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755161572801635) [2025-08-14 09:02:23.740578] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=423960039, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:23.740595] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:23.743518] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=44] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:23.735181] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=11179][errcode=-4389] timer task use too much time(end=1755162143735181, start=1755162142832747, delta=902434) [2025-08-14 09:02:23.733683] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=858115][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:23.734061] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=706730][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:23.262|207=707.80ms|210=0us|222=2us|total=707.83ms) [2025-08-14 09:02:23.744024] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=9946][errcode=-4389] timer task use too much time(end=1755162143744023, start=1755162143026254, delta=717769) [2025-08-14 09:02:23.744314] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:23.744333] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=15] temporary file module use no blocks [2025-08-14 09:02:23.744338] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=5] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:23.744347] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=7] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:23.744357] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623045182584, flush_io_finished_ret_:0, flush_io_finished_round_:26284, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:23.734483] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=12654][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:22.772|207=962.17ms|210=0us|222=2us|total=962.22ms) [2025-08-14 09:02:23.744796] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=10296][errcode=-4389] timer task use too much time(end=1755162143744796, start=1755162142772308, delta=972488) [2025-08-14 09:02:23.744820] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=761288, expected_run_ts_us_=1755162142983531, ObClockGenerator::getRealClock()=1755162143744819, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:22.983, task_priority:1, with_handle_protected:1}) [385]OB_MALLOC COST TOO MUCH TIME, cost_time=999861, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=27881, WASH_BLOCK_END=971980], seq:[0, 1, 2] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=996592, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=24587, WASH_BLOCK_END=972005], seq:[0, 1, 2] [717]OB_MALLOC COST TOO MUCH TIME, cost_time=972060, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=31, WASH_BLOCK_END=972029], seq:[0, 1, 2] [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=975623, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=3590, WASH_BLOCK_END=972033], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=975639, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=3559, WASH_BLOCK_END=972080], seq:[0, 1, 2] [513]OB_MALLOC COST TOO MUCH TIME, cost_time=972054, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=30, WASH_BLOCK_END=972024], seq:[0, 1, 2] [2025-08-14 09:02:21.843193] EDIAG eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1906331, owner: ObLog, click_count: 6, time dist:[FORMAT_END=28, ObMalloc#start=2, ObMalloc#end=930518, ObMalloc#start=18, ObMalloc#end=975650, FREE_END=115], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x27a8a8c8 0x97b7e1a 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:21.787009] EDIAG [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1962676, owner: ObLog, click_count: 6, time dist:[FORMAT_END=14, ObMalloc#start=1, ObMalloc#end=990113, ObMalloc#start=21, ObMalloc#end=972077, FREE_END=450], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1d189030 0x1d18285e 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:21.791272] EDIAG [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1958524, owner: ObLog, click_count: 6, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=982313, ObMalloc#start=31, ObMalloc#end=975680, FREE_END=488], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1f832f91 0x97c36fe 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [857]OB_MALLOC COST TOO MUCH TIME, cost_time=1010641, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=990035], seq:[0, 1] [2025-08-14 09:02:21.848265] EDIAG [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1903987, owner: ObLog, click_count: 7, time dist:[FORMAT_END=117, ObMalloc#start=0, ObMalloc#end=893188, ObMalloc#start=22, ObMalloc#end=1010658, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xff22be4 0x97a9cd7 0x97a9238 0x97a9307 0x97a8fe2 0x97a8cc7 0x97a88d2 0x97a7481 0x97a6a43 0x97a42c5 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [387]OB_MALLOC COST TOO MUCH TIME, cost_time=1004704, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=29159, WASH_BLOCK_END=975544], seq:[0, 1, 2] [2025-08-14 09:02:22.750510] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1004142, owner: ObLog, click_count: 6, time dist:[FORMAT_END=9, ObMalloc#start=0, ObMalloc#end=983691, ObMalloc#start=20, ObMalloc#end=20363, FREE_END=59], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:22.000332] EDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 1754795, owner: ObLog, click_count: 8, time dist:[FORMAT_END=21, ObMalloc#start=1, ObMalloc#end=777233, ObMalloc#start=16, ObMalloc#end=972072, ObMalloc#start=193, ObMalloc#end=5162, FREE_END=97], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x9757ced 0x97569df 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:22.744608] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1011697, owner: ObLog, click_count: 6, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=989479, ObMalloc#start=51, ObMalloc#end=21884, FREE_END=271], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:22.753885] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1002489, owner: ObLog, click_count: 6, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=996600, ObMalloc#start=21, ObMalloc#end=5547, FREE_END=311], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:22.749216] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1007955, owner: ObLog, click_count: 6, time dist:[FORMAT_END=6, ObMalloc#start=0, ObMalloc#end=984852, ObMalloc#start=26, ObMalloc#end=20478, FREE_END=2593], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [401]OB_MALLOC COST TOO MUCH TIME, cost_time=999049, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=19252, WASH_BLOCK_END=979797], seq:[0, 1, 2] [2025-08-14 09:00:57.044409] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [878][T1002_ArcSender][T1002][YB42AC4D0007-00063C495EA99C16-0-0] [lt=782655][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2695850, owner: ObLog, click_count: 5, time dist:[FORMAT_END=24, ObMalloc#start=2, ObMalloc#end=2695819, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3c61 0x113d348b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:22.749177] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1010505, owner: ObLog, click_count: 5, time dist:[FORMAT_END=15, ObMalloc#start=0, ObMalloc#end=1004731, ALLOC_END=5758, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:22.589172] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [797][T1002_FrzTrigge][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1172082, owner: ObLog, click_count: 8, time dist:[FORMAT_END=26, ObMalloc#start=1, ObMalloc#end=187353, ObMalloc#start=17, ObMalloc#end=960019, ObMalloc#start=1122, ObMalloc#end=23455, FREE_END=89], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:22.466640] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1294490, owner: ObLog, click_count: 8, time dist:[FORMAT_END=35, ObMalloc#start=2, ObMalloc#end=309632, ObMalloc#start=20, ObMalloc#end=960048, ObMalloc#start=1264, ObMalloc#end=23349, FREE_END=140], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:23.744837] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:22.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:23.735160] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.744893] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9715][errcode=0] task scheduled too delay(scheduled_delay=1013682, expected_run_ts_us_=1755162142731210, ObClockGenerator::getRealClock()=1755162143744893, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.744908] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.744928] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] task scheduled too delay(scheduled_delay=1012681, expected_run_ts_us_=1755162142732246, ObClockGenerator::getRealClock()=1755162143744927, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891991430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ce10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.735207] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13805][errcode=0] epoll handler too much time :13839 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:23.735167] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:23.745904] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=595996702, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:23.745921] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:23.741224] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=423960039, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:23.745951] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=4718] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:23.749358] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=8] gc stale ls task succ [2025-08-14 09:02:21.848265] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get_min_unreplayed_log_info(lsn={lsn:1604088821}, scn={val:1755100948296325001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873479, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:12492179, ret_code_:0}}, next_to_submit_lsn_:{lsn:1604088821}, next_to_submit_scn_:{val:1755100948296325001, v:0}, base_lsn_:{lsn:1409200128}, base_scn_:{val:1753840802540914000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:1604088821}, end_lsn:{lsn:1604088821}, read_buf:{buf_len_:3747840, buf_:0x7f98f6a05000}, block_size:67104768, log_storage_:0x7f990bfebbf0, log_storage_:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:42, read_io_size:147709952, read_disk_cost_ts:1061450, start_lsn:{lsn:1409200128}}}}}}) [2025-08-14 09:02:23.755162] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=1903987] get wrs ts(ls_id={id:1}, delta=394011767, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:23.755177] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:23.744039] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=451689, expected_run_ts_us_=1755162143292349, ObClockGenerator::getRealClock()=1755162143744039, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:23.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:23.755542] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=11483][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:23.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:23.745422] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10238] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623045170905, flush_io_finished_ret_:0, flush_io_finished_round_:26317, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:23.744940] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891991430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ce10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.755834] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10874][errcode=0] task scheduled too delay(scheduled_delay=1013427, expected_run_ts_us_=1755162142742406, ObClockGenerator::getRealClock()=1755162143755834, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aac90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.755851] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aac90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.755862] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1008156, expected_run_ts_us_=1755162142747705, ObClockGenerator::getRealClock()=1755162143755862, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.755873] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.755884] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1013348, expected_run_ts_us_=1755162142742535, ObClockGenerator::getRealClock()=1755162143755884, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.755894] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.755905] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1008598, expected_run_ts_us_=1755162142747306, ObClockGenerator::getRealClock()=1755162143755905, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e62b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.756022] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:23.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:23.758812] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=942316] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:02:23.759728] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4292] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:23.760618] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=7125] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:02:22.749177] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:23.761246] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1010505][errcode=0] epoll handler too much time :1012060 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:23.761262] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:23.755915] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e62b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.761279] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=5352][errcode=0] task scheduled too delay(scheduled_delay=1015578, expected_run_ts_us_=1755162142745700, ObClockGenerator::getRealClock()=1755162143761279, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73940b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.761309] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73940b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.761321] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1003821, expected_run_ts_us_=1755162142757499, ObClockGenerator::getRealClock()=1755162143761321, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eeb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.761331] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eeb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.762309] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9926] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:23.761938] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:02:23.761342] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1003696, expected_run_ts_us_=1755162142757645, ObClockGenerator::getRealClock()=1755162143761341, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73936b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eec90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.764989] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=3640][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73936b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eec90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.765019] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] task scheduled too delay(scheduled_delay=1005203, expected_run_ts_us_=1755162142759815, ObClockGenerator::getRealClock()=1755162143765018, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.765030] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.765041] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=998491, expected_run_ts_us_=1755162142766550, ObClockGenerator::getRealClock()=1755162143765041, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.770408] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:23.761125] INFO [STORAGE] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:123) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] wead read timestamp is too old(timestamp={val:1755161749741193000, v:0}, gts_scn={val:1755161762721798002, v:0}, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1755161749741193000, v:0}}) [2025-08-14 09:02:23.765052] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.770578] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=5515][errcode=0] task scheduled too delay(scheduled_delay=1010038, expected_run_ts_us_=1755162142760539, ObClockGenerator::getRealClock()=1755162143770578, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72765b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.770599] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72765b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.770611] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1001913, expected_run_ts_us_=1755162142768698, ObClockGenerator::getRealClock()=1755162143770611, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.770629] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.772208] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9409] get wrs ts(ls_id={id:1001}, delta=399042873, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:22.759249] EDIAG [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1013015, owner: ObLog, click_count: 7, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=999065, ObMalloc#start=31, ObMalloc#end=13907, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a1f867 0x27a135c3 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [395]OB_MALLOC COST TOO MUCH TIME, cost_time=1026148, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=29165, WASH_BLOCK_END=975578, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=9462], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:22.749131] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1026180, owner: ObLog, click_count: 5, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=1026168, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [567]OB_MALLOC COST TOO MUCH TIME, cost_time=881805, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=972912, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=972904], seq:[0] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=992812, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=37602], seq:[0, 1] [2025-08-14 09:02:23.757740] WDIAG [DETECT] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [723][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=947909][errcode=0] cost too much time:ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t), (this=|threshold=100.00ms|start at 09:02:23.757|total=949.37ms) [2025-08-14 09:02:23.757858] WDIAG ev_invoke_pending (ev.c:2136) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1011697][errcode=0] epoll handler too much time :1013260 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:23.773694] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=20950] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:23.773705] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15810] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:23.773709] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:23.757907] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1002489][errcode=0] epoll handler too much time :1004031 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:23.773719] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623043226372, flush_io_finished_ret_:0, flush_io_finished_round_:26305, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:23.773739] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15798] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:23.758729] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1007955][errcode=0] epoll handler too much time :1009519 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:23.759649] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [409][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=945248] [sql nio session](*s={this:0x7f98f7334030, session_id:3221557375, trace_id:YB42AC4D0007-00063C4962CA3040-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:262}, err:0, last_decode_time:1755161121788142, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:2389, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:22.759249] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=18] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:23.231181] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=6] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:23.773838] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=541833][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162143231176], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:23.773852] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162143231176], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:23.773865] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162143231176]}) [2025-08-14 09:02:23.773874] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=65, get_gts_with_stc_cnt=4, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:02:23.773883] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:23.773890] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:23.773905] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=5] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:23.773911] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162143773902], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:23.773918] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162143773902]}) [2025-08-14 09:02:23.773924] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5] gts statistics(tenant_id=1001, gts_rpc_cnt=0, get_gts_cache_cnt=79, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:02:23.774085] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:23.774093] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:23.774103] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162143774081], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:23.774111] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162143774081]}) [2025-08-14 09:02:23.774117] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162143774081]}) [2025-08-14 09:02:22.749131] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:23.776891] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1026180][errcode=0] epoll handler too much time :1027770 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:23.776908] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:23.770639] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1004271, expected_run_ts_us_=1755162142766368, ObClockGenerator::getRealClock()=1755162143770639, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.777127] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=6477][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.777151] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=1015458, expected_run_ts_us_=1755162142761692, ObClockGenerator::getRealClock()=1755162143777151, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.777163] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.777174] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1002621, expected_run_ts_us_=1755162142774552, ObClockGenerator::getRealClock()=1755162143777174, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aab10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.774, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.790333] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=966093] gc stale ls task succ [2025-08-14 09:02:24.274334] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=7] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:24.274350] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162144274331], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:24.274365] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162144274331], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:24.274377] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162144274331]}) [2025-08-14 09:02:24.274397] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=19] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162144274331]}) [2025-08-14 09:02:24.274408] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:24.274414] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162144274407], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:24.274422] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162144274407]}) [2025-08-14 09:02:24.274428] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162144274407]}) [2025-08-14 09:02:24.274600] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:24.274607] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:24.274618] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162144274597], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:24.274627] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162144274597]}) [2025-08-14 09:02:23.773767] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15006] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:24.723490] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=948286][errcode=0] epoll handler too much time :949726 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:23.773784] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [409][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14112] [sql nio session](*s={this:0x7f98f7333030, session_id:3221589989, trace_id:YB42AC4D0007-00063C4962CA3068-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:273}, err:0, last_decode_time:1755161766744050, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1903, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:24.723613] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [409][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=948394] [sql nio session](*s={this:0x7f98f7339030, session_id:3221590023, trace_id:YB42AC4D0007-00063C4962CA3069-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:254}, err:0, last_decode_time:1755161784744265, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1683, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:24.723626] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [409][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] [sql nio session](*s={this:0x7f98f733d030, session_id:3221590940, trace_id:YB42AC4D0007-00063C4962CA306E-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:247}, err:0, last_decode_time:1755162096770219, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1286, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:23.773819] WDIAG ev_invoke_pending (ev.c:2136) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=1013015][errcode=0] epoll handler too much time :1014574 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:24.723654] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=948392] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:23.843981] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=8718][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:23.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:24.726140] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=880821][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:23.843|207=882.16ms|210=0us|222=2us|total=882.18ms) [2025-08-14 09:02:24.726553] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9264-0-0] [lt=32][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162143842260) [2025-08-14 09:02:24.726576] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9264-0-0] [lt=19][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162143842260}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162143737764}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162143737764}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:24.720274] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:23.732895] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:23.777184] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aab10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.774, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:24.727269] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=948645][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:23.777|207=950.09ms|210=1us|222=2us|total=950.12ms) [2025-08-14 09:02:24.727293] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4389] timer task use too much time(end=1755162144727292, start=1755162143777173, delta=950119) [2025-08-14 09:02:24.727305] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1954590, expected_run_ts_us_=1755162142772714, ObClockGenerator::getRealClock()=1755162144727304, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739ef90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.772, task_priority:1, with_handle_protected:0}) [856]OB_MALLOC COST TOO MUCH TIME, cost_time=892450, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=39, WASH_BLOCK_END=892406, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=3], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:23.833755] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10272][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:24.727601] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=892492][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:24.726012] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=596984564, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:24.727672] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1652] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:24.728289] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=596988279, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:24.728316] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:24.728871] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=596988279, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:24.728886] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:24.736029] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=1952235] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:24.744003] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:24.744039] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:24.745614] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=71] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:24.747632] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=17] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:24.747749] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:24.748275] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:24.748318] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:24.752779] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=310] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [419]OB_MALLOC COST TOO MUCH TIME, cost_time=915380, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 11, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=915315], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=1003213, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=972931], seq:[0, 1] [2025-08-14 09:02:22.750555] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2003150, owner: ObLog, click_count: 7, time dist:[FORMAT_END=6, ObMalloc#start=0, ObMalloc#end=999877, ObMalloc#start=32, ObMalloc#end=1003229, ALLOC_END=3, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [1031]OB_MALLOC COST TOO MUCH TIME, cost_time=993188, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=11257, WASH_BLOCK_END=981927, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=2], seq:[0, 1, 2, 3, 4] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=981203, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=32, WASH_BLOCK_END=981170], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=921962, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=36, WASH_BLOCK_END=921926], seq:[0, 1, 2] [2025-08-14 09:02:24.753286] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:24.755264] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=12563] PNIO [ratelimit] time: 1755162144755260, bytes: 0, bw: 0.000000 MB/s, add_ts: 1002821, add_bytes: 0 [2025-08-14 09:02:24.755479] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:21.760175] EDIAG [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=18156][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2995750, owner: ObLog, click_count: 6, time dist:[FORMAT_END=2993904, ObMalloc#start=1, ObMalloc#end=64, ObMalloc#start=23, ObMalloc#end=27, FREE_END=1730], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b51df0f 0x986bcde 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [797]OB_MALLOC COST TOO MUCH TIME, cost_time=996020, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=11239, WASH_BLOCK_END=984781], seq:[0, 1, 2] [683]OB_MALLOC COST TOO MUCH TIME, cost_time=996005, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=11219, WASH_BLOCK_END=984786], seq:[0, 1, 2] [729]OB_MALLOC COST TOO MUCH TIME, cost_time=915882, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=34, WASH_BLOCK_END=911683, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=4164], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:22.742705] EDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] LOGGER COST TOO MUCH TIME, cost: 2018416, owner: ObLog, click_count: 10, time dist:[ALLOC_BLOCK#start=2, ALLOC_BLOCK#end=1, FORMAT_END=2014990, ObMalloc#start=0, ObMalloc#end=53, ObMalloc#start=21, ObMalloc#end=27, ObMalloc#start=1811, ObMalloc#end=42, FREE_END=1469], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x13b06ffb 0x958a39e 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:22.742955] EDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE3-0-0] [lt=969382][errcode=-4038] LOGGER COST TOO MUCH TIME, cost: 2022580, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2014869, ALLOC_END=7710, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1b501ccc 0x95dec2b 0x95de38d 0x1b899e14 0x1b83a0a6 0x1b839e80 0x1b863b6d 0x93a2971 0x934006e 0x933e2ee 0x93363f3 0x932e8ab 0x96ac524 0x96a672d 0x96a5033 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x1e6837e2 0x1e683258 0x1056234f 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:24.727325] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739ef90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:24.755572] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28197][errcode=0] task scheduled too delay(scheduled_delay=1975571, expected_run_ts_us_=1755162142779999, ObClockGenerator::getRealClock()=1755162144755571, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a45b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:24.755615] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a45b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:24.755632] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] task scheduled too delay(scheduled_delay=1980102, expected_run_ts_us_=1755162142775530, ObClockGenerator::getRealClock()=1755162144755632, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eee10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:24.755646] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eee10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:24.756605] WDIAG [COORDINATOR] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [471][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=19][errcode=0] cost too much time:ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_(), (this=|threshold=1.00s|start at 09:02:24.756|total=1.99s) [2025-08-14 09:02:22.750555] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:24.756762] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2003150][errcode=0] epoll handler too much time :2006213 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:24.756784] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:24.756800] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:23.763002] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=529225] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3895) [2025-08-14 09:02:24.759025] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=8] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:24.756531] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10377] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:23.770682] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=10] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:02:24.726167] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=-4389] timer task use too much time(end=1755162144726166, start=1755162143843978, delta=882188) [2025-08-14 09:02:24.755660] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=1979951, expected_run_ts_us_=1755162142775708, ObClockGenerator::getRealClock()=1755162144755660, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73931b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:24.760370] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=4701][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73931b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:24.760403] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] task scheduled too delay(scheduled_delay=1973295, expected_run_ts_us_=1755162142787107, ObClockGenerator::getRealClock()=1755162144760402, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:24.760422] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:24.760650] WDIAG [COORDINATOR] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [471][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] cost too much time:ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_failure(), (this=|threshold=1.00s|start at 09:02:24.760|total=2.00s) [2025-08-14 09:02:24.760670] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [471][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:23.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:24.760686] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [471][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:02:24.760|total=2.00s) [2025-08-14 09:02:24.760433] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1974019, expected_run_ts_us_=1755162142786414, ObClockGenerator::getRealClock()=1755162144760433, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:24.764358] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=3918][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:24.764385] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] task scheduled too delay(scheduled_delay=1977636, expected_run_ts_us_=1755162142786748, ObClockGenerator::getRealClock()=1755162144764384, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7273c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:24.764405] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7273c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.779381] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=16] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:02:21.734663] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:244) [407][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] connection close(sessid=3221590956, proxy_sessid=0, tenant_id=1002, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-08-14 09:02:22.742955] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE3-0-0] [lt=969382][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:23.775408] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=15] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:02:24.764416] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1980494, expected_run_ts_us_=1755162142783922, ObClockGenerator::getRealClock()=1755162144764416, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d27b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:24.769218] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=4793][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d27b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:24.769251] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] task scheduled too delay(scheduled_delay=1979865, expected_run_ts_us_=1755162142789385, ObClockGenerator::getRealClock()=1755162144769250, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d94b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:24.769265] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d94b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:24.769279] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=1988959, expected_run_ts_us_=1755162142780319, ObClockGenerator::getRealClock()=1755162144769279, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.780, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:24.768630] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE3-0-0] [lt=2022580][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:24.769292] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.780, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:24.769856] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=561][errcode=0] task scheduled too delay(scheduled_delay=1989367, expected_run_ts_us_=1755162142780489, ObClockGenerator::getRealClock()=1755162144769856, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739bbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.780, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:24.769872] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739bbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.780, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:24.769885] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=1988116, expected_run_ts_us_=1755162142781769, ObClockGenerator::getRealClock()=1755162144769885, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fa10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:22.778109] EDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=993329][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1995583, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=7, ALLOC_BLOCK#end=3, FORMAT_END=1995567, ObMalloc#start=1, ObMalloc#end=1, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:23.735195] EDIAG [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14540][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1040025, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=3, FORMAT_END=1040013, ObMalloc#start=0, ObMalloc#end=2, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0xfb90f6f 0xfb90ac1 0x9941516 0x993eecd 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:23.775036] EDIAG do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1001021, owner: ObLog, click_count: 7, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=981222, ObMalloc#start=29, ObMalloc#end=19753, ALLOC_END=3, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x202c530b 0x202c6888 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:24.726262] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10060] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:24.776525] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=50186][errcode=0] epoll handler too much time :50276 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:24.747724] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=20] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:24.776575] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28803] temporary file module use no blocks [2025-08-14 09:02:24.776585] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:24.776599] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:24.776615] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623046184145, flush_io_finished_ret_:0, flush_io_finished_round_:26285, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:23.735745] WDIAG [DETECT] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [851][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=842933][errcode=0] cost too much time:ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t), (this=|threshold=100.00ms|start at 09:02:24.776|total=1.89s) [2025-08-14 09:02:22.778109] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=993329][errcode=0] epoll handler too much time :994842 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:24.776745] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=1995583] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:24.776760] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:23.735195] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14540] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:02:24.727248] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=992849][errcode=0] epoll handler too much time :994363 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:24.776891] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=49557] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:24.769810] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE3-0-0] [lt=1172] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162144274597]}) [2025-08-14 09:02:23.778107] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [663][T1001_StandbySc][T1001][YB42AC4D0007-00063C495C7A0307-0-0] [lt=996267][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1000124, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1000121, ObMalloc#start=0, ObMalloc#end=3, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:23.743546] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1973988, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1034616, ALLOC_END=939371, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [402]OB_MALLOC COST TOO MUCH TIME, cost_time=1215670, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=48, WASH_BLOCK_END=1215622], seq:[0, 1, 2] [768]OB_MALLOC COST TOO MUCH TIME, cost_time=1040725, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=59, WASH_BLOCK_END=1040666], seq:[0, 1, 2] [316]OB_MALLOC COST TOO MUCH TIME, cost_time=993322, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=47, WASH_BLOCK_END=993275], seq:[0, 1, 2] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=1941719, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=52, WASH_BLOCK_END=1941643, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=23], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:23.776924] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1941771, owner: ObLog, click_count: 5, time dist:[FORMAT_END=17, ObMalloc#start=1, ObMalloc#end=1941733, ALLOC_END=19, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [857]OB_MALLOC COST TOO MUCH TIME, cost_time=1944868, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=43, WASH_BLOCK_END=1944825], seq:[0, 1, 2] [2025-08-14 09:02:23.772227] EDIAG [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1946556, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1660, ObMalloc#start=1, ObMalloc#end=1944875, ALLOC_END=20, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0xfa06846 0xfa05f62 0xfa05c84 0xfa05a50 0x1b51bd85 0x1b51b9c4 0x97a8d82 0x97a88d2 0x97a7481 0x97a6a43 0x97a42c5 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [552]OB_MALLOC COST TOO MUCH TIME, cost_time=960324, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 14, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=960260, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=1894521, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=41, WASH_BLOCK_END=1894448, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=31], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:24.678991] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=55701][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1041434, owner: ObLog, click_count: 8, time dist:[FORMAT_END=48, ObMalloc#start=2, ObMalloc#end=1040748, ObMalloc#start=33, ObMalloc#end=106, ObMalloc#start=248, ObMalloc#end=42, FREE_END=207], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [416]OB_MALLOC COST TOO MUCH TIME, cost_time=1783735, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=51, WASH_BLOCK_END=1783684], seq:[0, 1, 2] [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=941030, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=43, WASH_BLOCK_END=940986], seq:[0, 1, 2] [686]OB_MALLOC COST TOO MUCH TIME, cost_time=920491, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=43, WASH_BLOCK_END=920448], seq:[0, 1, 2] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=893147, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=53, WASH_BLOCK_END=893094], seq:[0, 1, 2] [2025-08-14 09:02:23.935575] EDIAG [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1789688, owner: ObLog, click_count: 5, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=1783744, ALLOC_END=5933, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1f9d7fcf 0x99b8be2 0x1f981833 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:23.824560] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1900747, owner: ObLog, click_count: 5, time dist:[FORMAT_END=30, ObMalloc#start=1, ObMalloc#end=1894531, ALLOC_END=6185, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:24.503751] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [402][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=142089][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1222737, owner: ObLog, click_count: 8, time dist:[FORMAT_END=32, ObMalloc#start=1, ObMalloc#end=1215683, ObMalloc#start=6552, ObMalloc#end=72, ObMalloc#start=279, ObMalloc#end=57, FREE_END=61], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [567]OB_MALLOC COST TOO MUCH TIME, cost_time=906009, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [997]OB_MALLOC COST TOO MUCH TIME, cost_time=1009031, tenant_id=1002, label=CalcConstExpr, ctx_id=0, prio=0, size=2079744, owner: ObMalloc, click_count: 1, time dist:[ALLOC_CHUNK_END=42379], seq:[0] [717]OB_MALLOC COST TOO MUCH TIME, cost_time=964701, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=964694], seq:[0] [2025-08-14 09:02:23.834414] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:23.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:24.777514] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=941669][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:23.834|207=943.10ms|210=2us|222=2us|total=943.12ms) [2025-08-14 09:02:24.777535] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task use too much time(end=1755162144777534, start=1755162143834413, delta=943121) [2025-08-14 09:02:23.775036] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=17] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:24.778231] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=917682] get wrs ts(ls_id={id:1}, delta=425002804, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:24.778265] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=30] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:24.778305] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119797572, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755162138745740) [2025-08-14 09:02:24.769898] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fa10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:24.778922] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9006][errcode=0] task scheduled too delay(scheduled_delay=1995700, expected_run_ts_us_=1755162142783221, ObClockGenerator::getRealClock()=1755162144778921, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f727dc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f729ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:24.778945] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f727dc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f729ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:24.778961] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] task scheduled too delay(scheduled_delay=1995265, expected_run_ts_us_=1755162142783695, ObClockGenerator::getRealClock()=1755162144778961, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f727d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f729af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:24.778976] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f727d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f729af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:24.757728] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=993219][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:24.779799] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=22023] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:24.779810] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3896) [2025-08-14 09:02:24.779817] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:24.779952] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=14] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:24.779967] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162144779947], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:24.779982] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162144779947], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:24.779985] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:24.779993] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3897) [2025-08-14 09:02:24.779993] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162144779947]}) [2025-08-14 09:02:24.779997] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:24.780001] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:24.780010] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:24.780019] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=6] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:24.780024] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162144780017], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:24.780032] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162144780017]}) [2025-08-14 09:02:24.780198] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:24.780208] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:24.780202] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [758][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB909F-0-0] [lt=39][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1001, svr="172.77.0.7:2882", version={val:1755161719775179002, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162144764328) [2025-08-14 09:02:24.780220] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162144780194], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:24.780222] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [758][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB909F-0-0] [lt=18][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1001, req={req_server:"172.77.0.7:2882", version:{val:1755161719775179002, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162144764328}, twrs={inited:true, tenant_id:1001, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162144777635}, server_version_for_stat:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162144777635}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:24.780300] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1040025] LogLoopThread round_cost_time(us)(round_cost_time=1056230) [2025-08-14 09:02:24.780318] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=7] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:02:24.780374] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [317][pnio1][T1001][YB42AC4D0007-00063C4960CB909F-0-0] [lt=28][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1001, delta=3039755, last_succ_cluster_heartbeat_tstamp_=1755162141740618) [2025-08-14 09:02:24.790366] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=13] gc stale ls task succ [2025-08-14 09:02:25.280108] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:25.280141] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=31] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3898) [2025-08-14 09:02:25.280150] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:25.280163] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:25.280170] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3899) [2025-08-14 09:02:25.283261] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=8] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:25.283280] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162145283257], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:25.283296] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162145283257], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:25.283309] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162145283257]}) [2025-08-14 09:02:25.283329] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=16] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:25.283334] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162145283327], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:25.283342] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162145283327]}) [2025-08-14 09:02:25.283519] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:25.283527] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:25.283551] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162145283515], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:25.283561] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162145283515]}) [2025-08-14 09:02:24.778894] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=425002804, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:25.720331] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=940005] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:24.778994] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] task scheduled too delay(scheduled_delay=1993149, expected_run_ts_us_=1755162142785844, ObClockGenerator::getRealClock()=1755162144778994, ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.720373] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=939954][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.743546] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:25.720561] WDIAG ev_invoke_pending (ev.c:2136) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1973988][errcode=0] epoll handler too much time :1977023 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:25.720580] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:24.764540] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=22] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:25.720604] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=954618] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:24.752815] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=24] gc stale ls task succ [2025-08-14 09:02:24.780287] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1832) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE3-0-0] [lt=7672][errcode=-4012] get gts fail(ret=-4012, timeout_us=1000000, request_time=[mts=1755162141756408]) [2025-08-14 09:02:24.861246] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=34035][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:24.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:24.985291] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:24.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:25.011635] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:25.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:25.265059] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=668][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:23.776924] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:23.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:25.721660] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1941771][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:23.776|207=1.94s|210=1us|222=1us|total=1.94s) [2025-08-14 09:02:23.772227] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:25.532913] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=253311][errcode=0] EASY SLOW: thread: 7f995447d300, lastrun: 1755162144.770309 cost: 0.762600 loop:26519, slowcnt: 7 [2025-08-14 09:02:25.280175] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:25.726092] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=445072] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:25.726172] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=78] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3900) [2025-08-14 09:02:25.726180] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:24.761117] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [719][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571f9d70, caller:ob_table_lock_service.cpp:start:276, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE6$_1273, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa090, time_wheel:0x7f99375f7d10, thread_pool:0x7f99375f74f0, is_running:0, time_interval:10.00s, expected_run_time:09:02:32.332, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:25.726268] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [719][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=963690][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:02:25.726|total=3.29s) [2025-08-14 09:02:25.720418] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=40][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:24.778|207=941.42ms|210=0us|222=2us|total=941.45ms) [2025-08-14 09:02:25.726634] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=6206][errcode=-4389] timer task use too much time(end=1755162145726634, start=1755162144778993, delta=947641) [2025-08-14 09:02:25.726646] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=2932903, expected_run_ts_us_=1755162142793742, ObClockGenerator::getRealClock()=1755162145726646, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891814fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.793, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.726663] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891814fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.793, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:24.726682] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [316][pnio1][T1][YB42AC4D0007-00063C49607B9264-0-0] [lt=11][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1, delta=5101083, last_succ_cluster_heartbeat_tstamp_=1755162139625599) [2025-08-14 09:02:25.721682] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4389] timer task use too much time(end=1755162145721681, start=1755162143776923, delta=1944758) [2025-08-14 09:02:25.726738] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=5044][errcode=0] task scheduled too delay(scheduled_delay=1934387, expected_run_ts_us_=1755162143792349, ObClockGenerator::getRealClock()=1755162145726737, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:23.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:25.726774] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:23.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:25.721216] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=734810][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:24.985|207=735.92ms|210=0us|222=2us|total=735.94ms) [2025-08-14 09:02:25.727718] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=6491][errcode=-4389] timer task use too much time(end=1755162145727717, start=1755162144985288, delta=742429) [2025-08-14 09:02:25.720658] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1844) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE3-0-0] [lt=938948][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1755162168697440, request_time=[mts=1755162141756408]) [2025-08-14 09:02:25.721832] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:25.721266] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=455512][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:02:25.727785] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6498][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:25.727799] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:25.721189] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=858639][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:24.861|207=859.94ms|210=0us|222=2us|total=859.96ms) [2025-08-14 09:02:25.721245] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=708525][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:25.116|207=709.61ms|210=0us|222=2us|total=709.62ms) [2025-08-14 09:02:25.720615] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:23.935575] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:02:25.728055] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=1789688] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:02:25.728096] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:02:25.728144] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:02:23.824560] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:23.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:25.722868] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=1946556] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=387501068, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=251670, succ_cluster_heartbeat_count=1488, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161006924036004, v:0}, local_cluster_delta=1138798830, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:02:25.722883] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=28] get wrs ts(ls_id={id:1}, delta=425946937, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:25.732806] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:25.726676] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=2930228, expected_run_ts_us_=1755162142796447, ObClockGenerator::getRealClock()=1755162145726675, ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.735070] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8380][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.735095] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] task scheduled too delay(scheduled_delay=2927821, expected_run_ts_us_=1755162142807273, ObClockGenerator::getRealClock()=1755162145735094, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7facb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.735108] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7facb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.727826] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=6627][errcode=-4389] timer task use too much time(end=1755162145727826, start=1755162144861244, delta=866582) [2025-08-14 09:02:25.727838] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=6581][errcode=-4389] timer task use too much time(end=1755162145727837, start=1755162145011633, delta=716204) [2025-08-14 09:02:25.727957] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7330] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:25.736558] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8586] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:25.736568] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:25.736576] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:25.736584] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:25.736591] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:25.727812] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:02:25.736621] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8787][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:25.736636] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:25.728205] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1900747][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:23.824|207=1.90s|210=0us|222=2us|total=1.90s) [2025-08-14 09:02:25.736670] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=8451][errcode=-4389] timer task use too much time(end=1755162145736669, start=1755162143824558, delta=1912111) [2025-08-14 09:02:25.728162] INFO [COMMON] print_io_status (ob_io_manager.cpp:2639) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:0, norm_iops:0, bw:2044, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:3331, total:3336) [2025-08-14 09:02:25.737791] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9590] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, param_config_:{memory_limit:536870912, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=865568, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:02:25.737839] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2826) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=41] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:2045, [delay/us]: prepare:0, schedule:0, submit:600000000, device:3331, total:3336) [2025-08-14 09:02:25.737858] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:02:25.737870] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:25.737879] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:25.739112] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:25.735123] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=2739466, expected_run_ts_us_=1755162142995656, ObClockGenerator::getRealClock()=1755162145735122, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199fbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919acc10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.995, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.739307] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=4176][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199fbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919acc10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:22.995, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.736527] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8674][errcode=0] task scheduled too delay(scheduled_delay=731038, expected_run_ts_us_=1755162145005487, ObClockGenerator::getRealClock()=1755162145736526, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:25.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:25.739340] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] task scheduled too delay(scheduled_delay=2020404, expected_run_ts_us_=1755162143718935, ObClockGenerator::getRealClock()=1755162145739339, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728bb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.718, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.739338] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=2805][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:25.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:25.736470] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=597991190, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [474]OB_MALLOC COST TOO MUCH TIME, cost_time=1750205, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=50, WASH_BLOCK_END=1723192, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=26957], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:23.994623] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1750260, owner: ObLog, click_count: 5, time dist:[FORMAT_END=37, ObMalloc#start=1, ObMalloc#end=1750221, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:23.838736] EDIAG try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=1584240][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1910406, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=915407, FORMAT_END=994993, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x26499505 0x97074a3 0x26498fa9 0x7f9973e081cf 0x7f9973a39dd3 [717]OB_MALLOC COST TOO MUCH TIME, cost_time=974451, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=45, WASH_BLOCK_END=49, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=964716], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:23.930571] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [741][T1001_MultiVers][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1821163, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1821163, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0xf9fb357 0x97ae92e 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:25.736654] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:25.740471] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=3806][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:01:49.731, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:01:42.751], idx=659, next_allow_print_ts="09:02:19.731") [2025-08-14 09:02:25.740487] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:25.740502] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:25.740514] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:25.737889] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=0kB/s, net_out=0kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=1280000kB/s [2025-08-14 09:02:25.744621] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:25.747258] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:25.747285] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:23.994623] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:23.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:25.747553] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1750260][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:23.994|207=1.75s|210=0us|222=2us|total=1.75s) [2025-08-14 09:02:25.747569] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] timer task use too much time(end=1755162145747568, start=1755162143994621, delta=1752947) [2025-08-14 09:02:25.747581] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] task scheduled too delay(scheduled_delay=1502947, expected_run_ts_us_=1755162144244633, ObClockGenerator::getRealClock()=1755162145747581, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:24.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:25.747595] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:24.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:25.739359] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728bb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.718, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750274] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10895][errcode=0] task scheduled too delay(scheduled_delay=2030467, expected_run_ts_us_=1755162143719806, ObClockGenerator::getRealClock()=1755162145750274, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180fbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.719, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750293] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180fbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.719, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750305] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=2002562, expected_run_ts_us_=1755162143747742, ObClockGenerator::getRealClock()=1755162145750305, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750315] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750327] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=2001892, expected_run_ts_us_=1755162143748434, ObClockGenerator::getRealClock()=1755162145750327, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750337] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750347] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=2001272, expected_run_ts_us_=1755162143749075, ObClockGenerator::getRealClock()=1755162145750347, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750357] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750367] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1994640, expected_run_ts_us_=1755162143755726, ObClockGenerator::getRealClock()=1755162145750367, ret=0, ret="OB_SUCCESS", *this={this:0x7f98918105b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750378] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918105b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750388] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1984992, expected_run_ts_us_=1755162143765395, ObClockGenerator::getRealClock()=1755162145750388, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750405] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750415] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1986782, expected_run_ts_us_=1755162143763633, ObClockGenerator::getRealClock()=1755162145750415, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750424] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750435] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1982162, expected_run_ts_us_=1755162143768273, ObClockGenerator::getRealClock()=1755162145750435, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891815730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750444] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891815730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750454] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1980058, expected_run_ts_us_=1755162143770395, ObClockGenerator::getRealClock()=1755162145750453, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eca90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750463] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eca90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750472] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1979855, expected_run_ts_us_=1755162143770617, ObClockGenerator::getRealClock()=1755162145750472, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e94b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750483] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e94b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750493] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1979706, expected_run_ts_us_=1755162143770786, ObClockGenerator::getRealClock()=1755162145750493, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750502] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750511] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1975389, expected_run_ts_us_=1755162143775121, ObClockGenerator::getRealClock()=1755162145750511, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180ea30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750520] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180ea30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750532] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1967663, expected_run_ts_us_=1755162143782869, ObClockGenerator::getRealClock()=1755162145750532, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750541] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750551] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1966445, expected_run_ts_us_=1755162143784105, ObClockGenerator::getRealClock()=1755162145750551, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750560] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750572] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1961485, expected_run_ts_us_=1755162143789087, ObClockGenerator::getRealClock()=1755162145750572, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196be10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750581] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196be10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750591] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1965253, expected_run_ts_us_=1755162143785337, ObClockGenerator::getRealClock()=1755162145750591, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728d430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7290f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750600] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728d430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7290f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750610] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1954845, expected_run_ts_us_=1755162143795764, ObClockGenerator::getRealClock()=1755162145750610, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750621] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750631] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1954724, expected_run_ts_us_=1755162143795906, ObClockGenerator::getRealClock()=1755162145750631, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73922b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750639] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73922b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750649] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1954612, expected_run_ts_us_=1755162143796036, ObClockGenerator::getRealClock()=1755162145750649, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750658] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750670] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1951533, expected_run_ts_us_=1755162143799136, ObClockGenerator::getRealClock()=1755162145750670, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.799, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750679] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.799, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750690] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1957755, expected_run_ts_us_=1755162143792934, ObClockGenerator::getRealClock()=1755162145750690, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750699] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750709] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1941498, expected_run_ts_us_=1755162143809210, ObClockGenerator::getRealClock()=1755162145750709, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.809, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750725] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.809, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750734] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1945873, expected_run_ts_us_=1755162143804860, ObClockGenerator::getRealClock()=1755162145750734, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eaa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750743] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eaa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750753] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1913214, expected_run_ts_us_=1755162143837539, ObClockGenerator::getRealClock()=1755162145750753, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73945b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.837, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750762] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73945b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.837, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750777] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=1839008, expected_run_ts_us_=1755162143911768, ObClockGenerator::getRealClock()=1755162145750777, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.911, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750787] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:23.911, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750808] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1032732, expected_run_ts_us_=1755162144718075, ObClockGenerator::getRealClock()=1755162145750808, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.718, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750817] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.718, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750827] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1024578, expected_run_ts_us_=1755162144726249, ObClockGenerator::getRealClock()=1755162145750827, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.726, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750838] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.726, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750849] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1022597, expected_run_ts_us_=1755162144728251, ObClockGenerator::getRealClock()=1755162145750848, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a67b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7397290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.728, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750858] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a67b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7397290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.728, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750868] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1019663, expected_run_ts_us_=1755162144731205, ObClockGenerator::getRealClock()=1755162145750868, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919d9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750877] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919d9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750889] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1019006, expected_run_ts_us_=1755162144731883, ObClockGenerator::getRealClock()=1755162145750889, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7277c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eac90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750898] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7277c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eac90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750909] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1002313, expected_run_ts_us_=1755162144748595, ObClockGenerator::getRealClock()=1755162145750909, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750918] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750928] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1002212, expected_run_ts_us_=1755162144748715, ObClockGenerator::getRealClock()=1755162145750928, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750939] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750949] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=994597, expected_run_ts_us_=1755162144756352, ObClockGenerator::getRealClock()=1755162145750949, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7277230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750959] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7277230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750968] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=993237, expected_run_ts_us_=1755162144757731, ObClockGenerator::getRealClock()=1755162145750968, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7272330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a5090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750978] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7272330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a5090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750989] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=995070, expected_run_ts_us_=1755162144755919, ObClockGenerator::getRealClock()=1755162145750989, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919822b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.750999] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919822b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751008] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=994397, expected_run_ts_us_=1755162144756611, ObClockGenerator::getRealClock()=1755162145751008, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751018] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751028] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=994119, expected_run_ts_us_=1755162144756908, ObClockGenerator::getRealClock()=1755162145751027, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d71b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751039] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d71b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751048] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=993325, expected_run_ts_us_=1755162144757723, ObClockGenerator::getRealClock()=1755162145751048, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751058] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751068] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=993718, expected_run_ts_us_=1755162144757349, ObClockGenerator::getRealClock()=1755162145751067, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a27b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751076] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a27b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751088] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=995274, expected_run_ts_us_=1755162144755814, ObClockGenerator::getRealClock()=1755162145751088, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a40b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751098] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a40b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751107] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=999081, expected_run_ts_us_=1755162144752026, ObClockGenerator::getRealClock()=1755162145751107, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7faf30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751116] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7faf30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751126] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=990528, expected_run_ts_us_=1755162144760597, ObClockGenerator::getRealClock()=1755162145751126, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7352330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196fc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751137] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7352330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196fc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751147] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=979388, expected_run_ts_us_=1755162144771758, ObClockGenerator::getRealClock()=1755162145751147, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aca90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751156] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aca90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751166] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=971534, expected_run_ts_us_=1755162144779631, ObClockGenerator::getRealClock()=1755162145751166, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f734fc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891918610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751175] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f734fc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891918610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751187] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=966244, expected_run_ts_us_=1755162144784943, ObClockGenerator::getRealClock()=1755162145751187, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751197] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751206] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=965635, expected_run_ts_us_=1755162144785571, ObClockGenerator::getRealClock()=1755162145751206, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751216] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751227] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=956481, expected_run_ts_us_=1755162144794746, ObClockGenerator::getRealClock()=1755162145751227, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.794, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751245] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.794, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751255] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=946719, expected_run_ts_us_=1755162144804536, ObClockGenerator::getRealClock()=1755162145751255, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751264] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751273] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=944213, expected_run_ts_us_=1755162144807059, ObClockGenerator::getRealClock()=1755162145751273, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751284] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751292] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=944136, expected_run_ts_us_=1755162144807156, ObClockGenerator::getRealClock()=1755162145751292, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751301] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751310] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=950156, expected_run_ts_us_=1755162144801154, ObClockGenerator::getRealClock()=1755162145751310, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7397b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751319] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7397b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751330] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=949610, expected_run_ts_us_=1755162144801719, ObClockGenerator::getRealClock()=1755162145751330, ret=0, ret="OB_SUCCESS", *this={this:0x7f98918299b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891916210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751339] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918299b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891916210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751349] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=925488, expected_run_ts_us_=1755162144825860, ObClockGenerator::getRealClock()=1755162145751349, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.825, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751357] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.825, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751366] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=925294, expected_run_ts_us_=1755162144826071, ObClockGenerator::getRealClock()=1755162145751365, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.826, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751376] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.826, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751385] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=930611, expected_run_ts_us_=1755162144820773, ObClockGenerator::getRealClock()=1755162145751385, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.820, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751394] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.820, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751413] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] task scheduled too delay(scheduled_delay=923074, expected_run_ts_us_=1755162144828338, ObClockGenerator::getRealClock()=1755162145751412, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72879b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.828, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751422] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72879b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.828, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751433] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=905023, expected_run_ts_us_=1755162144846410, ObClockGenerator::getRealClock()=1755162145751433, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.846, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751443] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:24.846, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751466] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72760b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:25.719, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751475] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:25.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751485] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:25.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751498] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739be30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:25.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751508] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:25.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751517] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7edc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:25.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751527] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ebe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:25.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:25.751536] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:25.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:23.838736] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=1584240] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:02:25.752879] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:25.752910] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=966350] gc stale ls task succ [2025-08-14 09:02:24.776990] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=20] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:25.752952] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=974482] temporary file module use no blocks [2025-08-14 09:02:25.752959] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=5] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:25.752969] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=9] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:25.752980] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623046213508, flush_io_finished_ret_:0, flush_io_finished_round_:26306, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:25.739369] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] task scheduled too delay(scheduled_delay=650052, expected_run_ts_us_=1755162145089316, ObClockGenerator::getRealClock()=1755162145739369, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:25.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:25.753023] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13619][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:02:25.720600] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6350] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:25.753103] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32453][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:02:25.736601] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:02:25.740534] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:25.753686] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13125][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:25.753873] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:25.753886] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:25.753896] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:02:25.661313] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=2993][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:1, time_interval:3.00s, expected_run_time:09:02:25.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:24.760085] EDIAG [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=3540][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1000096, owner: ObLog, click_count: 9, time dist:[ALLOC_BLOCK#start=6, ALLOC_BLOCK#end=960341, ALLOC_BLOCK#start=10085, ALLOC_BLOCK#end=91, ALLOC_BLOCK#start=10084, ALLOC_BLOCK#end=100, FORMAT_END=19389, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x1d0f5b0a 0x1d0f57c6 0x1d17ad7c 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:24.764489] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:02:24.770319] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:25.754444] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=982638][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:02:23.930571] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [741][T1001_MultiVers][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] task scheduled too delay(scheduled_delay=1592261, expected_run_ts_us_=1755162142338308, ObClockGenerator::getRealClock()=1755162143930570, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fa6e0, caller:ob_multi_version_garbage_collector.cpp:start:115, function_type:ZN9oceanbase19concurrency_control30ObMultiVersionGarbageCollector5startEvE6$_1159, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa9c0, time_wheel:0x7f99535a3990, thread_pool:0x7f99535a3170, is_running:1, time_interval:60.00s, expected_run_time:09:02:22.338, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:25.754511] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [741][T1001_MultiVers][T1001][Y0-0000000000000000-0-0] [lt=1821163][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:02:25.754524] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [741][T1001_MultiVers][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162145754524, start=1755162143930568, delta=1823956) [2025-08-14 09:02:25.755466] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:25.756390] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=266] update_check_range(min_check_version=2540, max_check_version=2540, global_version=2640) [2025-08-14 09:02:25.760371] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:02:21.775189] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:801) [368][VTblLocAsyncUp0][T1][YB42AC4D0007-00063C4968299C62-0-0] [lt=11][errcode=-4012] result set open failed(ret=-4012, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}) [2025-08-14 09:02:25.760473] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:21.759843] EDIAG [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12049][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2995981, owner: ObLog, click_count: 10, time dist:[FORMAT_END=2994361, ObMalloc#start=0, ObMalloc#end=28, ObMalloc#start=19, ObMalloc#end=35, FREE_END=1538, ALLOC_BLOCK#start=960089, ALLOC_BLOCK#end=65, ALLOC_BLOCK#start=17337, ALLOC_BLOCK#end=81], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b51df0f 0x986bcde 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [500]OB_MALLOC COST TOO MUCH TIME, cost_time=980141, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=980135], seq:[0] [500]OB_MALLOC COST TOO MUCH TIME, cost_time=980270, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=50, WASH_BLOCK_END=48, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=980168], seq:[0, 1, 2, 3, 4] [729]OB_MALLOC COST TOO MUCH TIME, cost_time=868449, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1], seq:[0, 1] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=783259, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:02:25.747229] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:25.761626] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14374][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:02:24.760085] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=3540] temporary file module use no blocks [2025-08-14 09:02:25.761722] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=1000096] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:25.761733] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:25.761743] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623046195584, flush_io_finished_ret_:0, flush_io_finished_round_:26318, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:25.761910] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=39395] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:02:25.761928] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:25.751805] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8640] get wrs ts(ls_id={id:1}, delta=396006164, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:25.763318] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11492] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:25.739396] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2917] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:25.765357] WDIAG ev_invoke_pending (ev.c:2136) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23661][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:02:25.765931] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24024] get wrs ts(ls_id={id:1}, delta=598008841, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:25.765945] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:25.770710] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=28] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:25.770760] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=30] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:25.770777] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=16] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:25.770796] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=535] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:25.772676] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=22] PNIO [ratelimit] time: 1755162145772674, bytes: 0, bw: 0.000000 MB/s, add_ts: 1017414, add_bytes: 0 [2025-08-14 09:02:25.732215] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9314] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:25.776733] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:25.778335] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=1001021] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:25.778366] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:25.778990] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=44435] get wrs ts(ls_id={id:1}, delta=425972178, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:25.779004] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:25.827777] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:02:25.836577] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=8669][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:02:25.865664] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=598008841, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:25.865685] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:25.871994] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:02:25.999553] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:02:26.233648] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] timer task use too much time(end=1755162146233647, start=1755162145999551, delta=234096) [2025-08-14 09:02:26.254074] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:26.254099] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:02:26.254117] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:26.254127] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:26.254137] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:02:26.254152] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:26.254162] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:26.254179] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:26.254188] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:01:49.731, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:01:42.751], idx=659, next_allow_print_ts="09:02:19.731") [2025-08-14 09:02:26.254206] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:26.254219] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:26.254230] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:26.254249] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:26.254261] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:26.254470] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:26.254482] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:26.254492] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:02:26.623626] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=542][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:02:26.626509] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=192714][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:02:26.714415] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:26.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:26.719999] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:26.721509] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5922] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:26.728390] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:26.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:26.731115] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919abd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:26.721, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:26.731148] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919abb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:26.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:26.736903] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=337] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:26.739196] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:26.742214] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=171] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:26.742589] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=267] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:25.753493] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16865] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:02:26.715245] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get_max_applied_scn(scn={val:1755161744727824001, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:49, palf_committed_end_lsn_:{lsn:7323686009}, palf_committed_end_scn_:{val:1755161747742105001, v:0}, last_check_scn_:{val:1755161749741193000, v:0}, max_applied_cb_scn_:{val:1755161744727824001, v:0}}) [2025-08-14 09:02:26.746273] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=30971] get_min_unreplayed_log_info(lsn={lsn:7323685240}, scn={val:1755100946796590002, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873485, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:8946764, ret_code_:0}}, next_to_submit_lsn_:{lsn:7323685240}, next_to_submit_scn_:{val:1755100946796590002, v:0}, base_lsn_:{lsn:7180210176}, base_scn_:{val:1753840803166611000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:7323685240}, end_lsn:{lsn:7323685240}, read_buf:{buf_len_:3747840, buf_:0x7f98f7a05000}, block_size:67104768, log_storage_:0x7f9901131bf0, log_storage_:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:28, read_io_size:98508800, read_disk_cost_ts:645339, start_lsn:{lsn:7180210176}}}}}}) [2025-08-14 09:02:25.822515] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:02:26.746360] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=922446][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:26.746375] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:26.746904] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=81] get wrs ts(ls_id={id:1001}, delta=401019533, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:26.746921] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:26.748181] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=2995750] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:26.748208] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=14] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:26.748872] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=425972178, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:26.748885] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:26.749508] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=425972178, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:26.749519] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:24.793553] INFO [ARCHIVE] do_thread_task_ (ob_archive_timer.cpp:125) [862][T1002_ArcTimer][T1002][YB42AC4D0007-00063C495E599C16-0-0] [lt=970128] archive round not in doing status, just skip(key={incarnation:-1, dest_id:-1, round:-1}, state={status:"INVALID"}) [2025-08-14 09:02:26.731161] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:26.726, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:26.749911] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18716][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fdc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:26.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:26.749934] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:26.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:26.749945] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728be10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:26.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:26.750132] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=598008841, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:26.750143] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:26.750755] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=598008841, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:26.149492] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [571][T1_ArcSender][T1][YB42AC4D0007-00063C495BA99C16-0-0] [lt=21][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:02:26.750802] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:26.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:26.737204] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:26.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:26.750834] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:26.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:26.751062] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:26.751347] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:25.975526] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=19][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:02:26.745678] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:26.141099] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=14][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:02:25.720638] EDIAG [ELECT] advance_ballot_number_and_reset_related_states_ (election_proposer.cpp:210) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=241][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1029276, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1024373, ObMalloc#start=1, ObMalloc#end=72, ObMalloc#start=25, ObMalloc#end=55, FREE_END=4750], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9d4e62 0xf9cf0e7 0xf9d3ec1 0xfa09ca2 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:26.746083] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4015] clock out of order(cur_ts=1755162145772739, cur_ts_=1755162145747357, delta=25382) [2025-08-14 09:02:26.754505] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=988427] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:02:26.756267] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:24.793553] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_timer.cpp:125) [862][T1002_ArcTimer][T1002][YB42AC4D0007-00063C495E599C16-0-0] [lt=970128][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1953160, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1953159, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x11406c8f 0x114063aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:26.754593] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:25.788029] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=50006] blacklist_loop exec finished(cost_time=15, is_enabled=true, send_cnt=0) [2025-08-14 09:02:26.759670] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:26.753600] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=11] gc stale ls task succ [2025-08-14 09:02:26.754641] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:26.759961] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=5301][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:02:26.759991] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:26.760005] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:26.760023] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:02:26.760040] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:26.751783] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=2995981] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:26.760093] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72847b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:26.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:26.760120] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:26.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:26.760150] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891806f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:26.751, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:26.760163] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:26.750, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:26.762622] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=22] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:26.762657] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=31] temporary file module use no blocks [2025-08-14 09:02:26.762667] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:26.762683] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:26.762698] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623048201534, flush_io_finished_ret_:0, flush_io_finished_round_:26319, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:26.764876] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:26.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:26.764914] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:26.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:26.750765] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:26.770249] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73539b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7396b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:26.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:26.770434] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=1310] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 444993536 used= 389699232 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 70 washed_blocks= 287 washed_size= 70221824 [MEMORY] request_cached_chunk_cnt= 4 [MEMORY] hold= 389699232 used= 379452675 count= 32873 avg_used= 11542 mod=SUMMARY [2025-08-14 09:02:26.770461] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=206] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:26.770473] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=19] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 230686720 used= 228884480 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 block_cnt= 110 chunk_cnt= 110 mod=Memstore [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:02:26.770486] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= EXECUTE_CTX_ID hold= 1163264 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 15 washed_size= 13516800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:26.770504] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:02:26.770520] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 53714944 used= 47487424 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 84 washed_blocks= 334 washed_size= 126640128 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 47487424 used= 43098388 count= 3994 avg_used= 10790 mod=SUMMARY [2025-08-14 09:02:26.770536] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1 ctx_id= WORK_AREA hold= 1458176 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 13 washed_blocks= 66 washed_size= 25804800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:26.770552] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 14204928 used= 10970224 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 277 washed_size= 31932416 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10970224 used= 6078924 count= 18573 avg_used= 327 mod=SUMMARY [2025-08-14 09:02:26.770571] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 111149056 used= 108896256 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 block_cnt= 211 chunk_cnt= 53 mod=CoStack [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 mod=SUMMARY [2025-08-14 09:02:26.770591] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 56623104 used= 56623104 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 56623104 used= 0 count= 27 avg_used= 0 mod=SUMMARY [2025-08-14 09:02:26.770608] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 34365376 limit= 429496720 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 34365376 used= 31086608 count= 475 avg_used= 65445 mod=SUMMARY [2025-08-14 09:02:26.770624] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 8003584 used= 4947392 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 10 washed_blocks= 23 washed_size= 12967936 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4947392 used= 4793344 count= 604 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:26.770639] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:26.770664] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 65536 used= 31024 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 2031616 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 31024 used= 27058 count= 20 avg_used= 1352 mod=SUMMARY [2025-08-14 09:02:26.770681] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1 ctx_id= RPC_CTX_ID hold= 8536064 used= 2727936 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2727936 used= 1795712 count= 115 avg_used= 15614 mod=SUMMARY [2025-08-14 09:02:26.770698] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 3481600 used= 2391232 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 10 washed_size= 4907008 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2391232 used= 2367040 count= 36 avg_used= 65751 mod=SUMMARY [2025-08-14 09:02:26.770717] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 3964928 used= 3080192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 3 washed_size= 229376 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3080192 used= 2983936 count= 376 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:26.770735] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= SCHEMA_SERVICE hold= 212992 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1884160 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:26.770747] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:26.770763] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1 ctx_id= MERGE_NORMAL_CTX_ID hold= 884736 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 6 washed_blocks= 9 washed_size= 11698176 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:26.770805] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=12] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:25.765637] EDIAG [RPC] handle_resp (ob_poc_rpc_proxy.cpp:192) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=998529][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1004467, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1004459, ALLOC_END=4, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x27a861af 0x97bcba5 0x97bc4ca 0x97b9834 0x97b893b 0x27a8a6d6 0x97b7ba0 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:26.758440] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:26.770896] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12436][errcode=0] epoll handler too much time :12467 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:26.750806] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=972732] finish to resize log external storage handler(current_ts=1755162145743987, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-08-14 09:02:25.765637] WDIAG [RPC] handle_resp (ob_poc_rpc_proxy.cpp:192) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=998529][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:02:26.771655] WDIAG stat_cleanup (debug.c:35) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=1004467][errcode=0] PNIO [delay_warn] cost too much time: 2045014us, procedure: pn_pktc_resp_cb [2025-08-14 09:02:26.771670] WDIAG eloop_delay_warn (debug.h:93) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] PNIO [delay_warn] eloop handle events delay high: 2045014, malloc=0/0 write=0/0 read=0/1 server_process=0/0 client_cb=2045014/1 [2025-08-14 09:02:26.768132] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:26.771891] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [998][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F58-0-0] [lt=24][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1002, svr="172.77.0.7:2882", version={val:1755161758221798000, v:0}, valid_part_count=0, total_part_count=2, generate_timestamp=1755162146756255) [2025-08-14 09:02:26.771913] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [998][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F58-0-0] [lt=20][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1002, req={req_server:"172.77.0.7:2882", version:{val:1755161758221798000, v:0}, valid_part_count:0, total_part_count:2, generate_timestamp:1755162146756255}, twrs={inited:true, tenant_id:1002, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162145751788}, server_version_for_stat:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162145751788}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:26.760053] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:26.771984] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11902][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:26.772001] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:01:49.731, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:01:42.751], idx=659, next_allow_print_ts="09:02:19.731") [2025-08-14 09:02:26.772001] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [316][pnio1][T1002][YB42AC4D0007-00063C49612B8F58-0-0] [lt=10][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1002, delta=5050409, last_succ_cluster_heartbeat_tstamp_=1755162141721592) [2025-08-14 09:02:26.772015] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:26.772773] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:26.760, task_priority:1, with_handle_protected:1}) [2025-08-14 08:59:25.740019] EDIAG LIBalloc (ob_malloc_allocator.cpp:924) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=950746][errcode=-4013] LOGGER COST TOO MUCH TIME, cost: 1033619, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1024894, ALLOC_END=8721, APPEND_END=3], seq:[0, 1, 2], BACKTRACE: 0xf9b6d19 0xf9b6367 0xf9b6058 0xf989dfb 0x973c6f6 0x973c18d 0x7f9973af5dfa 0x7f9973af5efd 0x260b8434 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:26.772950] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant: 500, limit: 9223372036854775807 hold: 695713792 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 340643840 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 37437440 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 85983232 limit= 9223372036854775807 [MEMORY] ctx_id= LIBEASY hold_bytes= 3145728 limit= 9223372036854775807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 10485760 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 2834432 limit= 9223372036854775807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 37273600 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 78925824 limit= 9223372036854775807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 98983936 limit= 9223372036854775807 [2025-08-14 09:02:26.772985] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 340643840 used= 301122992 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 490 washed_blocks= 2214 washed_size= 904953856 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 301122992 used= 293732241 count= 12660 avg_used= 23201 mod=SUMMARY [2025-08-14 09:02:26.773010] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 500 ctx_id= GLIBC hold= 37437440 used= 21975920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 188 washed_blocks= 823 washed_size= 371507200 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 21975920 used= 10371189 count= 131703 avg_used= 78 mod=SUMMARY [857]OB_MALLOC COST TOO MUCH TIME, cost_time=946007, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=945973], seq:[0, 1, 2, 3] [2025-08-14 09:02:26.773024] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 500 ctx_id= CO_STACK hold= 85983232 used= 83091456 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 83091456 used= 82936896 count= 161 avg_used= 515136 block_cnt= 161 chunk_cnt= 41 mod=CoStack [MEMORY] hold= 83091456 used= 82936896 count= 161 avg_used= 515136 mod=SUMMARY [2025-08-14 09:02:26.773036] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 500 ctx_id= LIBEASY hold= 3145728 used= 3070816 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1048576 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3070816 used= 2859784 count= 223 avg_used= 12824 mod=SUMMARY [2025-08-14 09:02:26.773047] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 500 ctx_id= LOGGER_CTX_ID hold= 12582912 used= 16646144 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 16646144 used= 16637952 count= 8 avg_used= 2079744 mod=SUMMARY [2025-08-14 09:02:26.773067] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 500 ctx_id= RPC_CTX_ID hold= 2834432 used= 1335296 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 52 washed_blocks= 65 washed_size= 106217472 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1335296 used= 881792 count= 55 avg_used= 16032 mod=SUMMARY [2025-08-14 09:02:26.773077] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 500 ctx_id= PKT_NIO hold= 37273600 used= 35816960 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 43 washed_blocks= 64 washed_size= 84647936 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 35816960 used= 34887680 count= 117 avg_used= 298185 mod=SUMMARY [2025-08-14 09:02:26.773097] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 500 ctx_id= SCHEMA_SERVICE hold= 78925824 used= 68912768 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 22 washed_size= 5767168 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 68912768 used= 67650911 count= 4857 avg_used= 13928 mod=SUMMARY [2025-08-14 09:02:26.773108] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 500 ctx_id= UNEXPECTED_IN_500 hold= 98983936 used= 97133904 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 97133904 used= 86750592 count= 3013 avg_used= 28792 mod=SUMMARY [2025-08-14 09:02:26.773120] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant: 508, limit: 1073741824 hold: 15613952 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 5128192 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10485760 limit= 9223372036854775807 [2025-08-14 09:02:26.773128] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 508 ctx_id= DEFAULT_CTX_ID hold= 5128192 used= 3765328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3765328 used= 3723912 count= 84 avg_used= 44332 mod=SUMMARY [2025-08-14 09:02:26.773149] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 508 ctx_id= CO_STACK hold= 10485760 used= 10321920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10321920 used= 10302720 count= 20 avg_used= 515136 mod=SUMMARY [2025-08-14 09:02:26.773187] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant: 509, limit: 1073741824 hold: 15613952 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 5128192 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10485760 limit= 9223372036854775807 [2025-08-14 09:02:26.773195] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 509 ctx_id= DEFAULT_CTX_ID hold= 5128192 used= 3765328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3765328 used= 3723912 count= 84 avg_used= 44332 mod=SUMMARY [2025-08-14 09:02:26.773214] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 509 ctx_id= CO_STACK hold= 10485760 used= 10321920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10321920 used= 10302720 count= 20 avg_used= 515136 mod=SUMMARY [2025-08-14 09:02:26.773270] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant: 1001, limit: 1073741824 hold: 606650368 cache_hold: 44040192 cache_used: 44040192 cache_item_count: 21 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 302309376 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 54525952 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 3080192 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 483328 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 45514752 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 5521408 limit= 157286400 [MEMORY] ctx_id= GLIBC hold_bytes= 15499264 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 73424896 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35651584 limit= 214748360 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 3252224 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 12582912 limit= 1073741800 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 5570560 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3866624 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 1179648 limit= 9223372036854775807 [2025-08-14 09:02:26.773292] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= DEFAULT_CTX_ID hold= 302309376 used= 268479952 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 81 washed_blocks= 406 washed_size= 113410048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 268479952 used= 262374598 count= 3766 avg_used= 69669 mod=SUMMARY [2025-08-14 09:02:26.773301] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= MEMSTORE_CTX_ID hold= 54525952 used= 54099968 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 54099968 used= 54075216 count= 26 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:02:26.773308] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1001 ctx_id= EXECUTE_CTX_ID hold= 3080192 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 8 washed_size= 7405568 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:26.773316] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:02:26.773324] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= PLAN_CACHE_CTX_ID hold= 45514752 used= 36449984 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 77 washed_blocks= 201 washed_size= 132743168 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 36449984 used= 32916258 count= 3312 avg_used= 9938 mod=SUMMARY [2025-08-14 09:02:26.773331] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= WORK_AREA hold= 5521408 used= 0 limit= 157286400 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 9 washed_blocks= 40 washed_size= 17547264 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:26.773346] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1001 ctx_id= GLIBC hold= 15499264 used= 10889728 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 25 washed_blocks= 164 washed_size= 47415296 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10889728 used= 6020141 count= 18313 avg_used= 328 mod=SUMMARY [2025-08-14 09:02:26.773355] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= CO_STACK hold= 73424896 used= 71221248 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 32 washed_blocks= 34 washed_size= 2072576 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 71221248 used= 71088768 count= 138 avg_used= 515136 block_cnt= 138 chunk_cnt= 36 mod=CoStack [MEMORY] hold= 71221248 used= 71088768 count= 138 avg_used= 515136 mod=SUMMARY [2025-08-14 09:02:26.773367] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= KVSTORE_CACHE_ID hold= 44040192 used= 44040192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 44040192 used= 0 count= 21 avg_used= 0 mod=SUMMARY [2025-08-14 09:02:26.773374] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 33906624 limit= 214748360 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33906624 used= 30629648 count= 468 avg_used= 65447 mod=SUMMARY [2025-08-14 09:02:26.773382] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= TX_CALLBACK_CTX_ID hold= 3252224 used= 2342592 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 3 washed_blocks= 8 washed_size= 3039232 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2342592 used= 2269696 count= 286 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:26.773390] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:26.773399] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= RPC_CTX_ID hold= 12582912 used= 786432 limit= 1073741800 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 786432 used= 520192 count= 32 avg_used= 16256 mod=SUMMARY [2025-08-14 09:02:26.773419] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=15] [MEMORY] tenant_id= 1001 ctx_id= TX_DATA_TABLE hold= 5570560 used= 2342784 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 8 washed_size= 2818048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2342784 used= 2319424 count= 30 avg_used= 77314 mod=SUMMARY [2025-08-14 09:02:26.773429] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= MDS_DATA_ID hold= 3866624 used= 2785280 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 5 washed_size= 4521984 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2785280 used= 2698240 count= 340 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:26.773443] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1001 ctx_id= MERGE_NORMAL_CTX_ID hold= 1179648 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 8 washed_blocks= 24 washed_size= 15597568 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:26.773460] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant: 1002, limit: 8589934592 hold: 1946038272 cache_hold: 48234496 cache_used: 48234496 cache_item_count: 23 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 1009610752 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 415236096 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 1114112 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 950272 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 211812352 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 1441792 limit= 429496725 [MEMORY] ctx_id= GLIBC hold_bytes= 12582912 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 149913600 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 37748736 limit= 1717986900 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 7766016 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 5980160 limit= 8589934500 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 5718016 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3047424 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 1032192 limit= 9223372036854775807 [2025-08-14 09:02:26.773488] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1002 ctx_id= DEFAULT_CTX_ID hold= 1009610752 used= 918703056 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 14 [MEMORY] hold= 185860096 used= 185534336 count= 114 avg_used= 1627494 block_cnt= 114 chunk_cnt= 104 mod=MysqlRequesReco [MEMORY] hold= 167792640 used= 167772192 count= 1 avg_used= 167772192 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 83927040 used= 83886080 count= 2 avg_used= 41943040 block_cnt= 2 chunk_cnt= 2 mod=LogGroupBuffer [MEMORY] hold= 67190784 used= 67109344 count= 4 avg_used= 16777336 block_cnt= 4 chunk_cnt= 4 mod=MrgMemPoolChk [MEMORY] hold= 918703056 used= 901499425 count= 9459 avg_used= 95305 mod=SUMMARY [2025-08-14 09:02:26.773499] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= MEMSTORE_CTX_ID hold= 415236096 used= 411992064 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 411992064 used= 411803568 count= 198 avg_used= 2079816 block_cnt= 198 chunk_cnt= 198 mod=Memstore [MEMORY] hold= 411992064 used= 411803568 count= 198 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:02:26.773510] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= EXECUTE_CTX_ID hold= 1114112 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 7 washed_size= 7274496 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:26.773517] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= TRANS_CTX_MGR_ID hold= 950272 used= 933888 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1146880 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 933888 used= 921984 count= 2 avg_used= 460992 mod=SUMMARY [2025-08-14 09:02:26.773526] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= PLAN_CACHE_CTX_ID hold= 211812352 used= 201995008 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 176930880 used= 160228962 count= 13450 avg_used= 11912 block_cnt= 13450 chunk_cnt= 101 mod=SqlPhyPlan [MEMORY] hold= 201995008 used= 183048143 count= 17906 avg_used= 10222 mod=SUMMARY [2025-08-14 09:02:26.773533] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= WORK_AREA hold= 1441792 used= 0 limit= 429496725 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 14 washed_blocks= 54 washed_size= 27918336 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:26.773541] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= GLIBC hold= 12582912 used= 11425280 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 11425280 used= 6322556 count= 19216 avg_used= 329 mod=SUMMARY [2025-08-14 09:02:26.773550] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= CO_STACK hold= 149913600 used= 147087360 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 68 washed_blocks= 70 washed_size= 3178496 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 147087360 used= 146813760 count= 285 avg_used= 515136 block_cnt= 285 chunk_cnt= 73 mod=CoStack [MEMORY] hold= 147087360 used= 146813760 count= 285 avg_used= 515136 mod=SUMMARY [2025-08-14 09:02:26.773561] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1002 ctx_id= KVSTORE_CACHE_ID hold= 48234496 used= 48234496 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 48234496 used= 0 count= 23 avg_used= 0 mod=SUMMARY [2025-08-14 09:02:26.773569] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= META_OBJ_CTX_ID hold= 37748736 used= 35741632 limit= 1717986900 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 35741632 used= 32457488 count= 496 avg_used= 65438 mod=SUMMARY [2025-08-14 09:02:26.773576] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= TX_CALLBACK_CTX_ID hold= 7766016 used= 6934400 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 17 washed_size= 6914048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 6934400 used= 6729728 count= 848 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:26.773584] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= LOB_CTX_ID hold= 147456 used= 16384 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 16384 used= 15872 count= 2 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:26.773594] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= RPC_CTX_ID hold= 5980160 used= 2580480 limit= 8589934500 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 6 washed_blocks= 6 washed_size= 10797056 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2580480 used= 1693824 count= 111 avg_used= 15259 mod=SUMMARY [2025-08-14 09:02:26.773603] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= TX_DATA_TABLE hold= 5718016 used= 4521856 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 9 washed_blocks= 16 washed_size= 13156352 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4521856 used= 4480128 count= 40 avg_used= 112003 mod=SUMMARY [2025-08-14 09:02:26.773612] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= MDS_DATA_ID hold= 3047424 used= 1998848 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 4 washed_size= 1146880 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1998848 used= 1936384 count= 244 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:26.773619] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1002 ctx_id= MDS_CTX_ID hold= 147456 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:26.773630] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:26.773636] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= MERGE_NORMAL_CTX_ID hold= 1032192 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 12 washed_size= 13647872 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:26.776809] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:26.776825] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] temporary file module use no blocks [2025-08-14 09:02:26.776831] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=5] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:26.776840] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=8] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:26.772031] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:26.782254] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10200][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:26.782269] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:26.782279] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:26.782446] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:26.782458] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:26.782468] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:02:26.776850] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623048216514, flush_io_finished_ret_:0, flush_io_finished_round_:26287, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:26.778502] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=14] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:26.846422] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:26.853717] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=226] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:02:26.946491] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:26.997918] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:26.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:27.046540] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:27.146598] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:27.246657] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:27.249422] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:27.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:27.259458] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:27.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:27.346717] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:27.446738] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:27.493605] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:27.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:27.546782] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:27.624161] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571634a0, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957163780, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:5.00s, expected_run_time:09:02:27.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:27.624198] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:27.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:27.646857] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:26.828452] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:26.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:27.717194] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=887394][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:26.828|207=888.74ms|210=0us|222=2us|total=888.77ms) [321]OB_MALLOC COST TOO MUCH TIME, cost_time=944207, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=944205], seq:[0] [822]OB_MALLOC COST TOO MUCH TIME, cost_time=939117, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=939111], seq:[0] [822]OB_MALLOC COST TOO MUCH TIME, cost_time=939150, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=939138], seq:[0, 1] [2025-08-14 09:02:26.717259] EDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=490025][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1005885, , BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x18257a6e 0x1823b8ca 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:00.723053] EDIAG [RPC] handle_resp (ob_poc_rpc_proxy.cpp:192) [315][pnio1][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1012871, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1012864, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x27a861af 0x97bcba5 0x97bc4ca 0x27ad7f11 0x97bc298 0x97b86e3 0x27a8a6d6 0x97b7ba0 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:27.717227] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4389] timer task use too much time(end=1755162147717227, start=1755162146828450, delta=888777) [2025-08-14 09:02:27.717585] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EEB1-0-0] [lt=947295][errcode=0] cost too much time:mds_tenant_service.cpp:int oceanbase::storage::mds::ObTenantMdsTimer::get_tablet_oldest_scn_(oceanbase::storage::ObTablet &, share::SCN &), (this=|threshold=5.00ms|start at 09:02:26.781|344=0us|total=936.32ms) [2025-08-14 09:02:27.717337] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:288) [570][T1_ArcFetcher][T1][YB42AC4D0007-00063C495B999C16-0-0] [lt=12] ObArchiveFetcher is running(thread_index=0) [2025-08-14 09:02:26.752230] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [280][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=23821][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:86, tid:764}]) [2025-08-14 09:02:26.785284] WARN [LIB] alloc (ob_malloc_allocator.cpp:924) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=1033619][errcode=-4013] [OOPS]: alloc failed reason is that server memory has reached the upper limit(server_hold: 4294418432, server_limit: 4294967296, alloc_size: 2097152) [2025-08-14 09:02:25.737961] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:8607757, max_lsn:1604090331, max_scn:{val:1755161756730847001, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:1556737504}, reuse_lsn:{lsn:1604090331}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:1562147291}}, last_submit_lsn:{lsn:1604090203}, last_submit_end_lsn:{lsn:1604090331}, last_submit_log_id:8607757, last_submit_log_pid:50, max_flushed_lsn:{lsn:1604090203}, max_flushed_end_lsn:{lsn:1604090331}, max_flushed_log_pid:50, committed_end_lsn:{lsn:1604090090}, last_slide_log_id:8607755, last_slide_scn:{val:1755161749741193000, v:0}, last_slide_lsn:{lsn:1604089968}, last_slide_end_lsn:{lsn:1604090090}, last_slide_log_pid:50, last_slide_log_accum_checksum:1446576759, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f990bfe4530}, [StateMgr]={this:0x7f990bfe89b0, self:"172.77.0.7:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"PENDING", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:51}, leader:"0.0.0.0:0", leader_epoch:-1, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:1604090331}, scan_disk_log_finished:true, last_check_start_id:8607756, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161901828039, palf_role_change_cb:0x7f990bfef470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:50, prev:{version:1, config:{config_version:{proposal_id:49, config_seq:45}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:50, config_seq:46}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_mode_pid:50}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_end_lsn:{lsn:1604088821}, prev_mode_pid:50}, reconfig_barrier:{prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_end_lsn:{lsn:1604088821}, prev_mode_pid:50}, persistent_config_version:{proposal_id:50, config_seq:46}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755161738854289, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755161738854289, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f990bfe5c70}, [ModeMgr]={palf_id:1, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:50, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:1543409664}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:51}, log_config_meta:{version:3, proposal_id:50, prev:{version:1, config:{config_version:{proposal_id:49, config_seq:45}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:50, config_seq:46}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_mode_pid:50}, log_snapshot_meta:{version:2, base_lsn:{lsn:1543409664}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:712704}, readable_log_tail:{lsn:712704}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:1610514432}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/meta", dir_fd:225, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66392064, block_header_serialize_buf_:0x7f990bfeabb8, flashback_version:0}, log_storage:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1755161901828095, this:0x7f990bfe90f0}, [Reconfirm]={palf_id:1, self:"172.77.0.7:2882", state:"RECONFIRM_MODE_META", new_proposal_id:51, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"172.77.0.7:2882", majority_max_accept_pid:50, majority_max_lsn:{lsn:1604090331}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:1755161901828373, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f990bfe8b30}) [857]OB_MALLOC COST TOO MUCH TIME, cost_time=952372, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=946055], seq:[0, 1] [2025-08-14 09:02:25.737961] EDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1983701, owner: ObLog, click_count: 9, time dist:[FORMAT_END=1043111, ObMalloc#start=1, ObMalloc#end=69, ObMalloc#start=22, ObMalloc#end=55, ObMalloc#start=1279, ObMalloc#end=939158, ALLOC_END=5, APPEND_END=1], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0xfc0e73d 0x993f560 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [321]OB_MALLOC COST TOO MUCH TIME, cost_time=948730, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=944223], seq:[0, 1] [2025-08-14 09:02:26.717259] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=490025][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:27.724694] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8302] tx gc loop thread is running(MTL_ID()=1) [2025-08-14 09:02:27.724714] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=1005885] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:27.724724] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3902) [2025-08-14 09:02:27.724730] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:27.724705] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] try gc retain ctx [2025-08-14 09:02:27.724781] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=76] try to retry start_working [2025-08-14 09:02:27.724915] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:27.724923] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3903) [2025-08-14 09:02:27.724927] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:26.843979] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=175] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:02:27.724994] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=879643] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:02:27.725034] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:02:27.725070] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:02:27.725086] INFO [COMMON] print_io_status (ob_io_manager.cpp:2639) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:2175, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:883348, total:883354) [2025-08-14 09:02:27.725106] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, param_config_:{memory_limit:536870912, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=865568, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:02:27.725342] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18933] get_max_applied_scn(scn={val:1755161547738516001, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:50, palf_committed_end_lsn_:{lsn:5282737655}, palf_committed_end_scn_:{val:1755161547738516001, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161547738516001, v:0}}) [2025-08-14 09:02:27.724912] WDIAG stat_cleanup (debug.c:35) [315][pnio1][T0][Y0-0000000000000000-0-0] [lt=1012871][errcode=0] PNIO [delay_warn] cost too much time: 88958624us, procedure: pn_pktc_resp_cb [2025-08-14 09:02:26.771770] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119812272, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-08-14 09:02:27.725628] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=952409] get_max_applied_scn(scn={val:1755161749741193000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:50, palf_committed_end_lsn_:{lsn:1604090090}, palf_committed_end_scn_:{val:1755161749741193000, v:0}, last_check_scn_:{val:1755161749741193001, v:0}, max_applied_cb_scn_:{val:1755161749741193000, v:0}}) [2025-08-14 09:02:27.725644] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get_min_unreplayed_log_info(lsn={lsn:1604088821}, scn={val:1755100948296325001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873479, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:12492179, ret_code_:0}}, next_to_submit_lsn_:{lsn:1604088821}, next_to_submit_scn_:{val:1755100948296325001, v:0}, base_lsn_:{lsn:1409200128}, base_scn_:{val:1753840802540914000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:1604088821}, end_lsn:{lsn:1604088821}, read_buf:{buf_len_:3747840, buf_:0x7f98f6a05000}, block_size:67104768, log_storage_:0x7f990bfebbf0, log_storage_:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:42, read_io_size:147709952, read_disk_cost_ts:1061450, start_lsn:{lsn:1409200128}}}}}}) [2025-08-14 09:02:27.725702] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=45] get wrs ts(ls_id={id:1}, delta=397983366, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:27.725714] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:27.726322] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1001}, delta=402997949, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:27.726336] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:27.726385] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=397984580, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:27.726394] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:27.725141] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2826) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=31] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:2175, [delay/us]: prepare:0, schedule:0, submit:600000000, device:883348, total:883354) [2025-08-14 09:02:27.726721] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=1547] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:02:27.726739] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:27.726754] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:27.726771] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=0kB/s, net_out=0kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=1280000kB/s [2025-08-14 09:02:27.726937] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1001}, delta=402997949, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:27.726949] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:27.726972] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=397984580, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:27.726980] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:27.725465] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] get_max_applied_scn(scn={val:1755161719775179002, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:45, palf_committed_end_lsn_:{lsn:8461098908}, palf_committed_end_scn_:{val:1755161719775179002, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161719775179002, v:0}}) [2025-08-14 09:02:26.777829] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=10] PNIO [ratelimit] time: 1755162146777828, bytes: 0, bw: 0.000000 MB/s, add_ts: 1005154, add_bytes: 0 [2025-08-14 09:02:27.727930] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9268-0-0] [lt=26][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162147717251) [2025-08-14 09:02:27.725367] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] get_min_unreplayed_log_info(lsn={lsn:5282737655}, scn={val:1755100947825033001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161624848121, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:13063726, ret_code_:0}}, next_to_submit_lsn_:{lsn:5282737655}, next_to_submit_scn_:{val:1755100947825033001, v:0}, base_lsn_:{lsn:4831543296}, base_scn_:{val:1755063445733337001, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7605000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:5282737655}, end_lsn:{lsn:5282737655}, read_buf:{buf_len_:3747840, buf_:0x7f98f7605000}, block_size:67104768, log_storage_:0x7f993e529bf0, log_storage_:{log_tail:{lsn:5282737655}, readable_log_tail:{lsn:5282737655}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:5234171904}, min_scn:{val:1755152244802535004, v:0}, curr_block_id:78, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:206, min_block_id:42, max_block_id:79, curr_writable_block_id:78}, logical_block_size_:67104768, curr_block_writable_size_:18539017, block_header_serialize_buf_:0x7f993e52a8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:85, read_io_size:301367296, read_disk_cost_ts:2364604, start_lsn:{lsn:4831543296}}}}}}) [2025-08-14 09:02:27.728035] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9268-0-0] [lt=104][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162147717251}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162146750162}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162146750162}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:27.728374] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1001}, delta=402997949, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:27.728390] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:27.727896] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=2421] get_min_unreplayed_log_info(lsn={lsn:8461098908}, scn={val:1755100947600130001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161920778323, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:23084275, ret_code_:0}}, next_to_submit_lsn_:{lsn:8461098908}, next_to_submit_scn_:{val:1755100947600130001, v:0}, base_lsn_:{lsn:7515734016}, base_scn_:{val:1753840867171901045, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6e05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:8461098908}, end_lsn:{lsn:8461098908}, read_buf:{buf_len_:3747840, buf_:0x7f98f6e05000}, block_size:67104768, log_storage_:0x7f99297a3bf0, log_storage_:{log_tail:{lsn:8461098908}, readable_log_tail:{lsn:8461098908}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8455200768}, min_scn:{val:1755160384851871002, v:0}, curr_block_id:126, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:215, min_block_id:93, max_block_id:127, curr_writable_block_id:126}, logical_block_size_:67104768, curr_block_writable_size_:61206628, block_header_serialize_buf_:0x7f99297a48b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:198, read_io_size:699822080, read_disk_cost_ts:4178035, start_lsn:{lsn:7515734016}}}}}}) [2025-08-14 09:02:27.729877] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1502] get wrs ts(ls_id={id:1}, delta=427953706, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:27.729891] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:23.761158] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:662) [272][TimerWK0_TimezoneMgr][T1002][YB42AC4D0007-00063C4960D9F14C-0-0] [lt=973720][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f98b4ba1620, tx_id:{txid:0}, state:1, addr:"172.77.0.7:2882", tenant_id:1002, session_id:1, assoc_session_id:1, client_sid:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1755162141731585, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180067074, seq_base:1755162141721205, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}) [2025-08-14 09:02:27.730381] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=427954740, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:27.730388] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:27.731307] WDIAG pktc_handle_req_queue (pktc_post.h:138) [315][pnio1][T0][Y0-0000000000000000-0-0] [lt=681][errcode=0] PNIO [delay_warn] delay high: 74005516 [2025-08-14 09:02:27.728605] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2674] get wrs ts(ls_id={id:1}, delta=599987257, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:27.731348] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2736] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:27.728340] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=6] tx gc loop thread is running(MTL_ID()=1001) [2025-08-14 09:02:27.735155] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=6804] try gc retain ctx [2025-08-14 09:02:27.735164] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=9] try to retry start_working [2025-08-14 09:02:26.962874] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb4798, timer:0x7f9966ee3f40, task:0x7f99333e53a0, task_type:"N9oceanbase5table20ObTableClientInfoMgr27ObTableClientInfoRetireTaskE", scheduled_time:1755161724792698, pushed_time:0, delay:60000000}, running_task_token={this:0x7f9931bb4178, timer:0x7f9966ee3f40, task:0x7f99333d2fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161723481487, pushed_time:1755161737766048, delay:900000000}, total_delay=422170175, threashold=1000000, tenant_id=1001) [2025-08-14 09:02:26.866437] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a5d18, timer:0x7f99200ff180, task:0x7f99200ff1a8, task_type:"N9oceanbase8observer18ObTableLoadService17ObCheckTenantTaskE", scheduled_time:1755162085759970, pushed_time:0, delay:1000000}, running_task_token={this:0x7f991b7a5fb8, timer:0x7f99200ff180, task:0x7f99200ff1f0, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162050755138, pushed_time:1755162084760013, delay:1000000}, total_delay=61106465, threashold=1000000, tenant_id=1002) [2025-08-14 09:02:27.731339] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [315][pnio1][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559640, sock=0x7f995f4cc048, code=-47 [2025-08-14 09:02:27.736693] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:27.738478] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=455][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:27.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:27.738531] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=263][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:27.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:26.790687] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=220] gc stale ls task succ [2025-08-14 09:02:27.735461] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162147725328}, version_delta=599996945) [2025-08-14 09:02:27.739360] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3405] get wrs ts(ls_id={id:1}, delta=600000082, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:27.739376] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:27.739854] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=600000082, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:27.739865] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:27.740668] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [315][pnio1][T0][Y0-0000000000000000-0-0] [lt=4098][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559649, sock=0x7f995f4cc048, code=-47 [2025-08-14 09:02:27.740697] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [315][pnio1][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559657, sock=0x7f995f4cc048, code=-47 [2025-08-14 09:02:27.741319] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=17] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:27.742226] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:27.742497] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:27.744668] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=217] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:27.740597] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1983701] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161974728707, last_flushed_end_lsn:{lsn:1604090331}}]) [2025-08-14 09:02:27.746438] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5716] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:27.746995] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:27.747428] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=269] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:27.747462] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:27.732256] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=241][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.724, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:27.750378] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:27.754330] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2956] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:26.732705] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1022109, owner: ObLog, click_count: 11, time dist:[ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=92, ALLOC_BLOCK#start=10496, ALLOC_BLOCK#end=107, ALLOC_BLOCK#start=980262, ALLOC_BLOCK#end=86, FORMAT_END=10190, ObMalloc#start=1, ObMalloc#end=20865, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:27.756054] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=6322] gc stale ls task succ [2025-08-14 09:02:26.732705] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:27.756392] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1022109][errcode=0] epoll handler too much time :1023697 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:27.756423] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:25.727744] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:662) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE3-0-0] [lt=7073][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f98b5c049a0, tx_id:{txid:0}, state:1, addr:"172.77.0.7:2882", tenant_id:1002, session_id:3221590966, assoc_session_id:3221590966, client_sid:3221590966, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1755162141731585, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180067074, seq_base:1755162141721205, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}) [2025-08-14 09:02:27.739938] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=6] finish update min start scn(max_decided_scn={val:1755161719775179002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1755161287778691002, v:0}, keep_alive_scn_:{val:1755161287778691002, v:0}, update_ts_:1755162147735290}) [2025-08-14 09:02:27.757652] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=21] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:27.757681] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=24] temporary file module use no blocks [2025-08-14 09:02:27.757691] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:27.758142] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=35] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:27.739917] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=6] finish update min start scn(max_decided_scn={val:1755161547738516001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1755161493768447002, v:0}, keep_alive_scn_:{val:1755161493768447002, v:0}, update_ts_:1755162147735290}) [2025-08-14 09:02:27.759566] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=19] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:27.759599] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=3431][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:27.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:27.760844] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17519][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:27.760878] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:27.760893] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:27.762958] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=30] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:27.762975] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] temporary file module use no blocks [2025-08-14 09:02:27.762984] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:27.757705] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:27.763496] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=5782] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623049193395, flush_io_finished_ret_:0, flush_io_finished_round_:26308, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:27.734253] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EEB1-0-0] [lt=18][errcode=0] cost too much time:mds_table_handler.cpp:int oceanbase::storage::mds::ObMdsTableHandler::try_release_nodes_below(const share::SCN &), (this=|threshold=5.00ms|start at 09:02:27.717|155=1us|159=0us|total=16.67ms) [2025-08-14 09:02:27.762996] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:27.764493] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=1494] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623049202497, flush_io_finished_ret_:0, flush_io_finished_round_:26320, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:27.725481] INFO [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:171) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=258] handle gts request statistics(total_rt=1923932, total_cnt=15, avg_rt=1.202457500000000000e+05, avg_cnt=1.500000000000000000e+00) [2025-08-14 09:02:25.727744] EDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:662) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE3-0-0] [lt=7073][errcode=-4283] LOGGER COST TOO MUCH TIME, cost: 2025647, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1996896, ObMalloc#start=1, ObMalloc#end=28746, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b864fb7 0x1b864817 0x93a2971 0x934006e 0x933e2ee 0x93363f3 0x932e8ab 0x96ac524 0x96a672d 0x96a5033 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x1e6837e2 0x1e683258 0x1056234f 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:27.764643] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=39101][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162147220801], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:27.764664] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162147220801], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:27.764683] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162147220801]}) [2025-08-14 09:02:27.767438] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EEB1-0-0] [lt=30160][errcode=0] cost too much time:mds_tenant_service.cpp:int oceanbase::storage::mds::ObTenantMdsTimer::try_recycle_mds_table_(oceanbase::storage::ObTablet &, const share::SCN &), (this=|threshold=5.00ms|start at 09:02:27.717|375=0us|total=49.85ms) [2025-08-14 09:02:27.717803] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=699282][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:27.174|207=700.34ms|210=0us|222=1us|total=746.70ms) [2025-08-14 09:02:27.767474] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=49594][errcode=-4389] timer task use too much time(end=1755162147767474, start=1755162147017457, delta=750017) [694]OB_MALLOC COST TOO MUCH TIME, cost_time=955150, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=955143], seq:[0] [2025-08-14 09:02:27.767488] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=761999, expected_run_ts_us_=1755162147005487, ObClockGenerator::getRealClock()=1755162147767487, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:27.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:27.767504] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:27.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:27.767516] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=678200, expected_run_ts_us_=1755162147089316, ObClockGenerator::getRealClock()=1755162147767516, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:27.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:27.767525] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:27.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:27.757738] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:02:27.728129] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [317][pnio1][T1][YB42AC4D0007-00063C49607B9268-0-0] [lt=8][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1, delta=3001447, last_succ_cluster_heartbeat_tstamp_=1755162144726682) [2025-08-14 09:02:27.767810] WDIAG [RPC] handle_resp (ob_poc_rpc_proxy.cpp:192) [317][pnio1][T0][Y0-0000000000000000-0-0] [lt=39615][errcode=0] async_cb handler cost too much time(total_time=39686, decode_time=5, process_time=39678, session_destroy_time=3, ret=0, pcode=1815) [2025-08-14 09:02:27.770907] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:27.777313] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=6392][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:27.777334] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d36b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ebb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.767, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:27.777344] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:27.767535] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=563237, expected_run_ts_us_=1755162147204297, ObClockGenerator::getRealClock()=1755162147767535, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:27.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:27.783205] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=6209] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:27.783222] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] temporary file module use no blocks [2025-08-14 09:02:27.783229] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:27.783253] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=24] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:27.783265] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623049222838, flush_io_finished_ret_:0, flush_io_finished_round_:26288, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:27.735515] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:355) [835][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=14] finish check and clear obj lock(ls->get_ls_id()={id:1001}) [2025-08-14 09:02:27.773770] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [726][T1001_ObXAHbWor][T1001][Y0-0000000000000000-0-0] [lt=20445] XA scheduler heartbeat task statistics(avg_time=1) [2025-08-14 09:02:27.781340] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13769][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:27.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:27.786851] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=5483][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:27.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:27.785769] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:27.786886] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1112][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72739b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a5e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:27.786899] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72779b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eab10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.777, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:27.790753] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=946661] gc stale ls task succ [2025-08-14 09:02:28.225011] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:28.225037] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=25] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3904) [2025-08-14 09:02:28.225045] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:27.786910] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728bc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.720125] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=931798][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:27.786|207=933.21ms|210=0us|222=2us|total=933.25ms) [2025-08-14 09:02:28.720138] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=494323] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:28.720158] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4389] timer task use too much time(end=1755162148720157, start=1755162147786909, delta=933248) [2025-08-14 09:02:28.720172] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=933451, expected_run_ts_us_=1755162147786720, ObClockGenerator::getRealClock()=1755162148720172, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919902b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:27.838530] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:27.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:27.851254] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:28.720372] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=867801][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:27.997054] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:27.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:28.018313] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:28.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:28.626681] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:28.555, task_priority:1, with_handle_protected:1}) [857]OB_MALLOC COST TOO MUCH TIME, cost_time=895954, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [729]OB_MALLOC COST TOO MUCH TIME, cost_time=893146, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [694]OB_MALLOC COST TOO MUCH TIME, cost_time=959538, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=955169], seq:[0, 1] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=945509, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=945504], seq:[0] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=945458, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=945419, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=956365, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=53, WASH_BLOCK_END=10844, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=945467], seq:[0, 1, 2, 3, 4] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=945520, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=945493, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [315]OB_MALLOC COST TOO MUCH TIME, cost_time=945532, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=945503, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=945540, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=945505, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=953061, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=47, WASH_BLOCK_END=7465, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=945549], seq:[0, 1, 2, 3, 4] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=943744, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=17, ALLOC_CHUNK_END=943696, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3, 4, 5] [651]OB_MALLOC COST TOO MUCH TIME, cost_time=943185, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=943152, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=941393, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=941345, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=939109, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=939072, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=938971, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=938942, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3, 4, 5] [822]OB_MALLOC COST TOO MUCH TIME, cost_time=938366, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=938329, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=907389, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=907345, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=885307, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=885278, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=797858, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=797824, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [869]OB_MALLOC COST TOO MUCH TIME, cost_time=767329, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=767301, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [2025-08-14 09:02:28.720495] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=701118][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:28.183|207=702.18ms|210=1us|222=1us|total=702.20ms) [2025-08-14 09:02:28.720160] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=21] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3905) [2025-08-14 09:02:28.722351] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1851][errcode=-4389] timer task use too much time(end=1755162148722350, start=1755162148018311, delta=704039) [2025-08-14 09:02:28.720526] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=195] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:28.721978] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=694] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:28.720201] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919902b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.722753] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=2544][errcode=0] task scheduled too delay(scheduled_delay=937286, expected_run_ts_us_=1755162147785466, ObClockGenerator::getRealClock()=1755162148722753, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.722776] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.631947] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=275][errcode=0] EASY SLOW: thread: 7f995447d300, lastrun: 1755162147.770483 cost: 0.861462 loop:26522, slowcnt: 8 [2025-08-14 09:02:27.735344] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=771301][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb2b98, timer:0x7f9966ee3f40, task:0x7f99322ce030, task_type:"N9oceanbase8observer20ObTableQueryASyncMgrE", scheduled_time:1755161725751508, pushed_time:0, delay:0}, running_task_token={this:0x7f9931bb4178, timer:0x7f9966ee3f40, task:0x7f99333d2fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161723481487, pushed_time:1755161737766048, delay:900000000}, total_delay=421983836, threashold=1000000, tenant_id=1001) [2025-08-14 09:02:28.722787] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=937774, expected_run_ts_us_=1755162147785013, ObClockGenerator::getRealClock()=1755162148722787, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.725190] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=2397][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.725219] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] task scheduled too delay(scheduled_delay=930770, expected_run_ts_us_=1755162147794448, ObClockGenerator::getRealClock()=1755162148725218, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7277730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.794, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.725229] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7277730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.794, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.725104] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8] get wrs ts(ls_id={id:1}, delta=428948062, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:28.726252] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1141] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:28.723580] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=422] current monitor number(seq_=-1) [403]OB_MALLOC COST TOO MUCH TIME, cost_time=949488, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=945529], seq:[0, 1] [2025-08-14 09:02:27.282634] EDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] LOGGER COST TOO MUCH TIME, cost: 1444778, , BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x13b06ffb 0x958a39e 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [396]OB_MALLOC COST TOO MUCH TIME, cost_time=949645, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=945530, WASH_BLOCK_END=4115], seq:[0, 1, 2] [315]OB_MALLOC COST TOO MUCH TIME, cost_time=949632, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=945541, WASH_BLOCK_END=4091], seq:[0, 1, 2] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=947741, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=943753, WASH_BLOCK_END=3988], seq:[0, 1, 2] [2025-08-14 09:02:28.724462] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=398982048, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:28.729312] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=4828] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:27.778575] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=236] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:28.729531] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=949518][errcode=0] epoll handler too much time :950964 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:27.282634] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:28.729639] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=1444778][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:02:28.729662] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:28.729674] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:28.729686] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:02:28.729703] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:28.729716] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:28.729745] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:28.729757] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:01:49.731, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:01:42.751], idx=659, next_allow_print_ts="09:02:19.731") [2025-08-14 09:02:28.729769] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:28.729785] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:28.729839] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] get wrs ts(ls_id={id:1001}, delta=404000744, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:28.729862] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:28.729896] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=398987375, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:28.729906] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:28.722362] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=713226, expected_run_ts_us_=1755162148009134, ObClockGenerator::getRealClock()=1755162148722361, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:28.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:28.725253] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] task scheduled too delay(scheduled_delay=930238, expected_run_ts_us_=1755162147795014, ObClockGenerator::getRealClock()=1755162148725253, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.730827] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=5563][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.730866] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] task scheduled too delay(scheduled_delay=912858, expected_run_ts_us_=1755162147818007, ObClockGenerator::getRealClock()=1755162148730866, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.730885] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.730902] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] task scheduled too delay(scheduled_delay=910810, expected_run_ts_us_=1755162147820091, ObClockGenerator::getRealClock()=1755162148730902, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.820, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.730918] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.820, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.730927] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=900868, expected_run_ts_us_=1755162147830058, ObClockGenerator::getRealClock()=1755162148730927, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.730943] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.730954] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=778310, expected_run_ts_us_=1755162147952643, ObClockGenerator::getRealClock()=1755162148730953, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196aa90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.952, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.730974] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196aa90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.952, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.730986] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=777946, expected_run_ts_us_=1755162147953039, ObClockGenerator::getRealClock()=1755162148730986, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.953, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.730997] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.953, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.731009] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=760494, expected_run_ts_us_=1755162147970515, ObClockGenerator::getRealClock()=1755162148731009, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891992fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196de90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.970, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.731021] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891992fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196de90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:27.970, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.729799] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:28.725926] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=136] [COMPACTION DAG STATUS] (start_time=1755162137789269, end_time=1755162148725925, dag_status={}) [2025-08-14 09:02:27.764701] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=17150] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136829134139001, v:0}, clog_base_lsn:{lsn:8186781696}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1753840867372367034, v:0}, all_id_meta:{id_meta:[{limited_id:1755161729775179000, latest_log_ts:{val:1755161719775179001, v:0}}, {limited_id:18000001, latest_log_ts:{val:1755157440846556001, v:0}}, {limited_id:60000001, latest_log_ts:{val:1755135657266496001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:46, palf_env_:0x7f99321fa030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-08-14 09:02:28.735522] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=969344] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:28.735552] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=21] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:28.736675] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:27.764709] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18561] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136826952869001, v:0}, clog_base_lsn:{lsn:5099962368}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1755071646732636000, v:0}, all_id_meta:{id_meta:[{limited_id:1755161555795565000, latest_log_ts:{val:1755161545796642001, v:0}}, {limited_id:22000001, latest_log_ts:{val:1755135645787806003, v:0}}, {limited_id:60000001, latest_log_ts:{val:1755135646400653001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:50, palf_env_:0x7f9950bf4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-08-14 09:02:28.737218] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=971034] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:28.737248] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:28.730386] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1001}, delta=404002193, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:28.737311] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6912] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [321]OB_MALLOC COST TOO MUCH TIME, cost_time=952654, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=939120, WASH_BLOCK_END=13534], seq:[0, 1, 2] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=952540, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=938980, WASH_BLOCK_END=13559], seq:[0, 1, 2] [822]OB_MALLOC COST TOO MUCH TIME, cost_time=951958, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=938375, WASH_BLOCK_END=13582], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=920968, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=907398, WASH_BLOCK_END=13570], seq:[0, 1, 2] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=898938, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=885316, WASH_BLOCK_END=13622], seq:[0, 1, 2] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=811524, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=797868, WASH_BLOCK_END=13656], seq:[0, 1, 2] [651]OB_MALLOC COST TOO MUCH TIME, cost_time=958211, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=943204, WASH_BLOCK_END=11615, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=2431], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:24.779010] EDIAG [MDS] flush (mds_table_mgr.cpp:184) [14905][T1001_TimerWK8_Flush][T1001][YB42AC4D0007-00063C4977399C31-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3961146, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=2, FORMAT_END=3950652, ObMalloc#start=1, ObMalloc#end=10480, ALLOC_END=6, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0x1dc06744 0x1dc0374a 0x994534c 0x1c3fd293 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [869]OB_MALLOC COST TOO MUCH TIME, cost_time=788373, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=767338, WASH_BLOCK_END=21035], seq:[0, 1, 2] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=962767, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=941403, WASH_BLOCK_END=21364], seq:[0, 1, 2] [2025-08-14 09:02:28.725111] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=1023] [COMPACTION DAG STATUS] (start_time=1755162137763545, end_time=1755162148722981, dag_status={}) [2025-08-14 09:02:27.770495] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:28.740312] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=968351][errcode=0] epoll handler too much time :969827 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:27.781262] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EEB1-0-0] [lt=26][errcode=0] cost too much time:mds_tenant_service.cpp:int oceanbase::storage::mds::ObTenantMdsTimer::process_with_tablet_(oceanbase::storage::ObTablet &), (this=|threshold=10.00ms|start at 09:02:26.781|315=0us|317=936.33ms|319=49.86ms|total=1.00s) [2025-08-14 09:02:27.857832] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=10] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:02:27.785617] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=257][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:27.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:28.743228] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=956162][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:27.785|207=957.61ms|210=0us|222=2us|total=957.64ms) [2025-08-14 09:02:28.743259] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=-4389] timer task use too much time(end=1755162148743258, start=1755162147785614, delta=957644) [2025-08-14 09:02:28.743269] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=950919, expected_run_ts_us_=1755162147792349, ObClockGenerator::getRealClock()=1755162148743269, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:27.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:28.744327] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:28.739177] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:28.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:28.729929] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=965618][errcode=0] epoll handler too much time :967091 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:28.720393] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=722242][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:27.997|207=723.34ms|210=0us|222=1us|total=748.53ms) [2025-08-14 09:02:28.745586] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=25153][errcode=-4389] timer task use too much time(end=1755162148745585, start=1755162147997051, delta=748534) [2025-08-14 09:02:28.745601] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=500967, expected_run_ts_us_=1755162148244633, ObClockGenerator::getRealClock()=1755162148745601, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:28.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:28.745618] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:28.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:28.745639] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8030, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8310, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:10.00s, expected_run_time:09:02:28.730, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:28.745651] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:28.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:28.725145] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=988301][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb2578, timer:0x7f9934119ee0, task:0x7f9934405028, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr17RefreshConfigTaskE", scheduled_time:1755162146762409, pushed_time:0, delay:10000000}, running_task_token={this:0x7f9931bb2658, timer:0x7f9934119ee0, task:0x7f9934405038, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr12TabletGCTaskE", scheduled_time:1755161981770099, pushed_time:1755162136772529, delay:20000}, total_delay=1962736, threashold=1000000, tenant_id=1001) [2025-08-14 09:02:28.743284] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:27.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:24.779010] INFO [MDS] flush (mds_table_mgr.cpp:184) [14905][T1001_TimerWK8_Flush][T1001][YB42AC4D0007-00063C4977399C31-0-0] [lt=9] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1755136829134139001, v:0}}, {tablet_id:{id:60379}, rec_scn:{val:1755136829134139002, v:0}}, {tablet_id:{id:50380}, rec_scn:{val:1755136829134139003, v:0}}, {tablet_id:{id:545}, rec_scn:{val:1755136829134139004, v:0}}, {tablet_id:{id:233}, rec_scn:{val:1755136829134139005, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1755136829134139006, v:0}}, {tablet_id:{id:50192}, rec_scn:{val:1755136829134139007, v:0}}, {tablet_id:{id:60350}, rec_scn:{val:1755136829134139008, v:0}}, {tablet_id:{id:60507}, rec_scn:{val:1755136829134139009, v:0}}, {tablet_id:{id:60259}, rec_scn:{val:1755136829134139010, v:0}}, {tablet_id:{id:188}, rec_scn:{val:1755136829134139011, v:0}}, {tablet_id:{id:224}, rec_scn:{val:1755136829134139012, v:0}}, {tablet_id:{id:308}, rec_scn:{val:1755136829134139013, v:0}}, {tablet_id:{id:369}, rec_scn:{val:1755136829134139014, v:0}}, {tablet_id:{id:493}, rec_scn:{val:1755136829134139015, v:0}}, {tablet_id:{id:101011}, rec_scn:{val:1755136829134139016, v:0}}, {tablet_id:{id:521}, rec_scn:{val:1755136829134139017, v:0}}, {tablet_id:{id:101109}, rec_scn:{val:1755136829134139018, v:0}}, {tablet_id:{id:284}, rec_scn:{val:1755136829134139019, v:0}}, {tablet_id:{id:50149}, rec_scn:{val:1755136829134139020, v:0}}, {tablet_id:{id:307}, rec_scn:{val:1755136829134139021, v:0}}, {tablet_id:{id:50240}, rec_scn:{val:1755136829134139022, v:0}}, {tablet_id:{id:50355}, rec_scn:{val:1755136829134139023, v:0}}, {tablet_id:{id:60457}, rec_scn:{val:1755136829134139024, v:0}}, {tablet_id:{id:182}, rec_scn:{val:1755136829134139025, v:0}}, {tablet_id:{id:101098}, rec_scn:{val:1755136829134139026, v:0}}, {tablet_id:{id:60387}, rec_scn:{val:1755136829134139027, v:0}}, {tablet_id:{id:419}, rec_scn:{val:1755136829134139028, v:0}}, {tablet_id:{id:60305}, rec_scn:{val:1755136829134139029, v:0}}, {tablet_id:{id:60297}, rec_scn:{val:1755136829134139030, v:0}}, {tablet_id:{id:60342}, rec_scn:{val:1755136829244588001, v:0}}, {tablet_id:{id:378}, rec_scn:{val:1755136829244588002, v:0}}, }, max_consequent_callbacked_scn={val:1755161719775179002, v:0}, *this={this:0x7f99297d7150, is_inited:true, freezing_scn:{val:1755136801284520000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136829134139001, v:0}, clog_base_lsn:{lsn:8186781696}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1753840867372367034, v:0}, all_id_meta:{id_meta:[{limited_id:1755161729775179000, latest_log_ts:{val:1755161719775179001, v:0}}, {limited_id:18000001, latest_log_ts:{val:1755157440846556001, v:0}}, {limited_id:60000001, latest_log_ts:{val:1755135657266496001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:46, palf_env_:0x7f99321fa030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-08-14 09:02:28.747016] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:28.747041] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:28.740000] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=937774] log compressor cycles once. (ret=0, cost_time=183970403, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25622441984, fast_delete_log_mode=false) [2025-08-14 09:02:28.747067] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:28.722353] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=2189][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:27.957644] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [869][T1002_MultiVers][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1522333, expected_run_ts_us_=1755162146435309, ObClockGenerator::getRealClock()=1755162147957643, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fe960, caller:ob_multi_version_garbage_collector.cpp:start:115, function_type:ZN9oceanbase19concurrency_control30ObMultiVersionGarbageCollector5startEvE6$_1159, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fec40, time_wheel:0x7f99535f7990, thread_pool:0x7f99535f7170, is_running:1, time_interval:60.00s, expected_run_time:09:02:26.435, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:28.747309] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [869][T1002_MultiVers][T1002][Y0-0000000000000000-0-0] [lt=788451][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:27.957|187=789.65ms|203=0us|207=18us|210=0us|222=2us|total=789.68ms) [2025-08-14 09:02:28.747328] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [869][T1002_MultiVers][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4389] timer task use too much time(end=1755162148747327, start=1755162147957641, delta=789686) [2025-08-14 09:02:27.783138] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=4590] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:28.747469] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=24872] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:28.747487] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=17] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3906) [2025-08-14 09:02:28.747494] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:28.747511] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=3324][errcode=-4389] timer task use too much time(end=1755162148747510, start=1755162147817313, delta=930197) [2025-08-14 09:02:28.726304] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=36] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119797572, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755162138745740) [2025-08-14 09:02:28.748881] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21989] get wrs ts(ls_id={id:1}, delta=428954838, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:28.732656] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=2851][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:28.748912] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=29] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=2, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:28.748934] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16248][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:28.749025] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:28.749073] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:28.749096] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:28.749112] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:28.749122] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:02:28.750410] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:28.751124] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7074] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=2, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:28.751717] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=600991501, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:28.751733] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:28.722256] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] gts statistics(tenant_id=1002, gts_rpc_cnt=0, get_gts_cache_cnt=15, get_gts_with_stc_cnt=9, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:02:28.753155] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=30850] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162147220801]}) [2025-08-14 09:02:28.753172] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1002}) [2025-08-14 09:02:28.753182] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1002}) [2025-08-14 09:02:28.755077] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=961050][errcode=0] get_fetch_log_dst failed or invalid(ret=0, palf_id=1001, self="172.77.0.7:2882", fetch_log_dst="172.77.0.7:2882") [2025-08-14 09:02:28.755098] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:02:28.756490] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=632] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:28.757757] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=7656][errcode=-4015] clock out of order(cur_ts=1755162148757756, cur_ts_=1755162148730017, delta=27739) [2025-08-14 09:02:28.743211] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=814863] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:02:28.757870] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14598] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:02:28.757914] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:02:28.757934] INFO [COMMON] print_io_status (ob_io_manager.cpp:2639) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:3965, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:7420, total:7425) [2025-08-14 09:02:28.757959] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, param_config_:{memory_limit:536870912, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=865568, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:02:28.758007] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2826) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=43] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:3965, [delay/us]: prepare:0, schedule:0, submit:600000000, device:7420, total:7425) [2025-08-14 09:02:28.758027] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:02:28.758040] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:28.758053] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:28.758064] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SERVER] net_in=0kB/s, net_out=0kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=1280000kB/s [2025-08-14 09:02:28.758404] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] get_fetch_log_dst failed or invalid(ret=0, palf_id=1, self="172.77.0.7:2882", fetch_log_dst="172.77.0.7:2882") [2025-08-14 09:02:28.758433] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=26] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:02:28.753464] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1421) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1001, is_allowed_broadcast=true) [2025-08-14 09:02:28.756336] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=313][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:28.759115] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=2773][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:28.759138] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162147220801]}) [2025-08-14 09:02:28.759690] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:28.759923] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=7059] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25622491136, fast_delete_log_mode=false) [2025-08-14 09:01:01.759511] EDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:76) [829][T1002_LogRessvr][T1002][YB42AC4D0007-00063C495D199C16-0-0] [lt=987189][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1009249, owner: ObLog, click_count: 7, time dist:[FORMAT_END=960857, ObMalloc#start=1, ObMalloc#end=48390, ALLOC_END=1, APPEND_END=0, ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=15], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x1135bce6 0x9a0eb0a 0x1134e1b4 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:28.757721] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EEB1-0-0] [lt=958256] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-08-14 09:02:28.762399] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15602] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:28.742479] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=31] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:28.763147] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20635][errcode=0] epoll handler too much time :20677 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:28.763259] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=18] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:28.763703] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=18] gc stale ls task succ [2025-08-14 09:02:28.768814] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=9856] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:02:28.769672] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:28.769812] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=305][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:28.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:28.769841] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:28.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:28.769866] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=535][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72774b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:28.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.769892] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fbe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:28.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.769906] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:28.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.769923] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:28.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.769937] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:28.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.777623] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:28.779256] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:28.779277] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:28.779292] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:28.779836] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=2816][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:28.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:28.779953] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919aef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:28.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.779975] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7270a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a4190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:28.762, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.779998] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:28.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.780012] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:28.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.780025] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:28.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.780165] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=984216] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:02:28.780246] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=992289] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:02:28.783330] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=21] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:28.783359] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=23] temporary file module use no blocks [2025-08-14 09:02:28.783370] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:28.783395] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=22] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:28.783411] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623050223055, flush_io_finished_ret_:0, flush_io_finished_round_:26289, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:28.789371] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:28.789393] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:28.789404] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:28.790008] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f734de30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891919810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:28.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.790042] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:28.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.790063] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:28.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.790088] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d59b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:28.773, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.790104] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efa10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:28.777, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.790121] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eba10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:28.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.790134] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:28.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.790220] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=982071] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:02:28.790780] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=11] gc stale ls task succ [2025-08-14 09:02:28.793076] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=956474] PNIO [ratelimit] time: 1755162148793073, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007637, add_bytes: 0 [2025-08-14 09:02:28.799487] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:28.799511] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:28.799519] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:28.800076] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:28.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.800106] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:28.790, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.809252] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fc8b0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcb90, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:1, time_interval:3.00s, expected_run_time:09:02:28.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:28.809595] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:28.809612] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:28.809618] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:28.810062] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:28.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:28.810103] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40b7f0, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40bad0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:10.00s, expected_run_time:09:02:28.802, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:28.819691] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:28.819714] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:28.819722] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:28.823645] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [876][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E999C16-0-0] [lt=23] ObArchiveSender is running(thread_index=3) [2025-08-14 09:02:28.826989] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=429050454, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:28.827027] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=35] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:28.827526] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [758][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB90A4-0-0] [lt=37][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1001, svr="172.77.0.7:2882", version={val:1755161719775179002, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162148826363) [2025-08-14 09:02:28.827566] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [758][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB90A4-0-0] [lt=26][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1001, req={req_server:"172.77.0.7:2882", version:{val:1755161719775179002, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162148826363}, twrs={inited:true, tenant_id:1001, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162148826414}, server_version_for_stat:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162148826414}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:28.827666] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [316][pnio1][T1001][YB42AC4D0007-00063C4960CB90A4-0-0] [lt=13][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1001, delta=4047291, last_succ_cluster_heartbeat_tstamp_=1755162144780373) [2025-08-14 09:02:28.828404] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=429052341, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:28.828438] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=31] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:28.829799] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:28.829814] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:28.829825] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:28.829989] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=399088467, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:28.830009] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:28.830330] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:28.827, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.830370] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [979][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F5B-0-0] [lt=25][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1002, svr="172.77.0.7:2882", version={val:1755161758221798000, v:0}, valid_part_count=0, total_part_count=2, generate_timestamp=1755162148829944) [2025-08-14 09:02:28.830393] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [979][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F5B-0-0] [lt=21][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1002, req={req_server:"172.77.0.7:2882", version:{val:1755161758221798000, v:0}, valid_part_count:0, total_part_count:2, generate_timestamp:1755162148829944}, twrs={inited:true, tenant_id:1002, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162148729885}, server_version_for_stat:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162148729885}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:28.830488] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [317][pnio1][T1002][YB42AC4D0007-00063C49612B8F5B-0-0] [lt=9][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1002, delta=2058487, last_succ_cluster_heartbeat_tstamp_=1755162146772001) [2025-08-14 09:02:28.835617] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=13] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:28.837320] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:28.839894] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:28.839916] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:28.839927] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:28.840389] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739af30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196bc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:28.837, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.847690] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:28.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:28.849988] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:28.850013] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:28.850021] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:28.850447] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:28.841, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.850473] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:28.849, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.850487] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:28.849, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.851893] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=601112734, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:28.851921] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:28.852183] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=4230][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:28.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:28.860094] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:28.860115] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:28.860122] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:28.860131] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162147220801]}) [2025-08-14 09:02:28.870218] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:28.870248] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:28.873991] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=1754795][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a5fb8, timer:0x7f994b3feee0, task:0x7f994b3fef80, task_type:"N9oceanbase8observer18ObTableLoadService21ObClientTaskPurgeTaskE", scheduled_time:1755162027715356, pushed_time:0, delay:1000000}, running_task_token={this:0x7f994e1a5df8, timer:0x7f994b3feee0, task:0x7f994b3fef50, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162023758648, pushed_time:1755162122757495, delay:1000000}, total_delay=121158633, threashold=1000000, tenant_id=1) [2025-08-14 09:02:28.876330] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c0b90, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571c0e70, time_wheel:0x7f9959fef990, thread_pool:0x7f9959fef170, is_running:1, time_interval:3.00s, expected_run_time:09:02:28.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:28.880328] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:28.880347] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:28.890392] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:28.890414] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:28.900501] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:28.900523] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:28.900752] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:28.892, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.910548] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:28.910570] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:28.920654] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:28.920677] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:28.920880] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:28.917, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.927779] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=429152596, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=252240, succ_cluster_heartbeat_count=1892, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161003593738002, v:0}, local_cluster_delta=1145334037, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:02:28.928528] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=39] get wrs ts(ls_id={id:1}, delta=429152339, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:28.928545] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:28.930697] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:28.931029] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:28.922, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:28.935673] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=23] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:28.937401] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:28.940803] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:28.943081] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=883750] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:02:28.946016] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=20739][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb3c38, timer:0x7f99333cb180, task:0x7f99333cb1c0, task_type:"N9oceanbase8observer18ObTableLoadService15ObHeartBeatTaskE", scheduled_time:1755161967770316, pushed_time:0, delay:10000000}, running_task_token={this:0x7f9931bb3ed8, timer:0x7f99333cb180, task:0x7f99333cb208, task_type:"N9oceanbase8observer18ObTableLoadService25ObClientTaskAutoAbortTaskE", scheduled_time:1755161959782824, pushed_time:1755162045729074, delay:1000000}, total_delay=181175699, threashold=1000000, tenant_id=1001) [2025-08-14 09:02:28.947724] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:28.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:28.950883] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:28.951852] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=601212456, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:28.951872] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:28.952230] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:28.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:28.960147] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:02:28.960213] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:02:28.960286] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:02:28.960334] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:02:28.960386] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:02:28.960404] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=1, sync_total=0, async_cnt= 1 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:28.960429] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:28.960439] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SERVER] net_in=0kB/s, net_out=0kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=1280000kB/s [2025-08-14 09:02:28.960980] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:28.961000] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=17] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162147220801]}) [2025-08-14 09:02:28.971063] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:28.981155] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:28.991240] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:28.992136] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:28.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.001291] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.011358] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.011443] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:29.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.011467] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:29.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.021415] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.023593] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:29.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.028468] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] get wrs ts(ls_id={id:1}, delta=429252374, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:29.028495] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:29.031525] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.035741] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=21] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:29.037469] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:29.041586] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.048256] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:28.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.051647] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.052255] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:28.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.052790] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=601313508, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:29.052808] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:29.061709] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.061730] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162147220801]}) [2025-08-14 09:02:29.071806] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.081881] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.091974] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.091999] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:29.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.102056] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.112105] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.122201] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.128535] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=429353113, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:29.128561] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:29.132285] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.136255] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=27] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:29.137531] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:29.142369] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.148275] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:29.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.152839] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=601412963, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:29.152865] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:29.152893] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.153264] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:29.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.162985] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.163007] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=20] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162147220801]}) [2025-08-14 09:02:29.173096] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.183140] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.193225] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.203299] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.213380] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.213411] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:29.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.223462] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.228603] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=429452028, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:29.228628] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:29.228680] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [758][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB90A8-0-0] [lt=40][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1001, svr="172.77.0.7:2882", version={val:1755161719775179002, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162149227961) [2025-08-14 09:02:29.228710] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [758][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB90A8-0-0] [lt=24][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1001, req={req_server:"172.77.0.7:2882", version:{val:1755161719775179002, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162149227961}, twrs={inited:true, tenant_id:1001, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162149227988}, server_version_for_stat:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162149227988}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:29.229337] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=30] get wrs ts(ls_id={id:1}, delta=429452028, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:29.229358] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:29.233536] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.236328] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=17] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:29.237590] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=56] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:29.243622] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.246144] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:29.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.247557] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:29.247582] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=24] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3907) [2025-08-14 09:02:29.247591] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:29.247602] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:29.247617] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=15] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3908) [2025-08-14 09:02:29.247622] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:29.247802] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:29.247820] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=17] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3909) [2025-08-14 09:02:29.247830] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:29.248327] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:29.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.249316] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:29.249343] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:02:29.249362] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:29.249375] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:29.249395] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:02:29.249413] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:29.249434] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:29.249453] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:29.249465] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:01:49.731, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:01:42.751], idx=659, next_allow_print_ts="09:02:19.731") [2025-08-14 09:02:29.249479] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:29.249528] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:29.249542] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:29.249553] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:29.249567] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:29.249750] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:29.249771] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:29.249785] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:02:29.252879] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=601513548, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:29.252902] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:29.253028] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B926D-0-0] [lt=27][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162149252345) [2025-08-14 09:02:29.253054] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B926D-0-0] [lt=21][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162149252345}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162149252370}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162149252370}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:29.253129] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [317][pnio1][T1][YB42AC4D0007-00063C49607B926D-0-0] [lt=13][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1, delta=1524999, last_succ_cluster_heartbeat_tstamp_=1755162147728129) [2025-08-14 09:02:29.253316] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:29.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.253497] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=6] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:29.253513] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162149253491], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:29.253536] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162149253491], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:29.253549] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162149253491]}) [2025-08-14 09:02:29.253561] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=44, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:02:29.253574] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162149253491]}) [2025-08-14 09:02:29.253587] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=11] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:29.253595] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162149253586], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:29.253607] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162149253586]}) [2025-08-14 09:02:29.253616] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts statistics(tenant_id=1001, gts_rpc_cnt=0, get_gts_cache_cnt=43, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:02:29.253625] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162149253586]}) [2025-08-14 09:02:29.253660] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=601514614, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:29.253674] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:29.253706] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.253825] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.253841] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162149253821], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:29.256218] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:29.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.263781] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.263803] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=20] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162149253821]}) [2025-08-14 09:02:29.263817] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:29.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.273891] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.275120] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:29.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.283957] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.294022] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.304107] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.305310] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:29.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.314212] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.324298] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.329420] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=56] get wrs ts(ls_id={id:1}, delta=429553271, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:29.329453] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=30] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:29.334382] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.336365] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=16] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:29.337644] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=28] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:29.344472] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.348379] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:29.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.353399] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:29.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.353774] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=601614359, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:29.353798] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:29.354549] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.364630] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.364654] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=22] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162149253821]}) [2025-08-14 09:02:29.374710] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.384795] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.394883] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.404951] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.415053] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.425152] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.435217] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.436433] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=17] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:29.437672] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=19] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:29.445316] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.448454] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:29.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.453462] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:29.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.453739] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=601714364, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:29.453765] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:29.455400] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.465492] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.465519] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=23] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162149253821]}) [2025-08-14 09:02:29.475618] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.485679] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.495741] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.497809] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:29.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.505824] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.515884] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.525956] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.526444] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:29.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.536037] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.536459] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=20] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:29.537718] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=19] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:29.546132] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.548513] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:29.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.553523] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:29.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.556202] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.566282] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.566305] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=20] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162149253821]}) [2025-08-14 09:02:29.576371] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.586451] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.596531] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.606611] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.616691] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.626761] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.627307] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=93727][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:29.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.636523] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=14] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:29.636847] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.637777] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=22] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:29.646923] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.648562] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:29.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.657004] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.667084] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.667106] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162149253821]}) [2025-08-14 09:02:29.677190] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.687257] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.697342] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.707406] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.717526] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.720339] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1844] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:29.721700] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=410] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:29.723508] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=90962][errcode=0] EASY SLOW: thread: 7f99542fb300, lastrun: 1755162115.752503 cost: 33.971003 loop:26416, slowcnt: 282 [2025-08-14 09:02:29.263842] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:29.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:00.748580] EDIAG [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:487) [354][EvtHisUpdTask][T1][YB42AC4D0007-00063C495FE9A745-0-0] [lt=22342][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1010313, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1010312, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x9c5fb23 0xfad7c8f 0xfad7980 0xfad7757 0x201f5064 0x201f0d83 0x15ee1248 0x93c5e8b 0x96ac66e 0x96a672d 0x96a5033 0x97aa16f 0x124eac91 0x97a9944 0x261395f9 0x1f900517 0x1f8fe2bf 0x9a5f486 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:00.748880] EDIAG [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:487) [373][Occam][T1][YB42AC4D0007-00063C4960599C9C-0-0] [lt=971162][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2029136, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1010002, ObMalloc#start=1, ObMalloc#end=1019130, ALLOC_END=2, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x9c5fb23 0xfad7c8f 0xfad7980 0xfad7757 0x201f5064 0x201f0d83 0x15ee1248 0x93c5e8b 0x96ac66e 0x96a672d 0x96a5033 0x97aa16f 0x124eac91 0x97a9944 0x261395f9 0xfa0e430 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:29.653583] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:29.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.725276] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99319fd110, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd3f0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:0, time_interval:10.00s, expected_run_time:09:02:34.726, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.725292] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:02:29.725|total=4.97s) [2025-08-14 09:02:29.725314] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] task executed too delay(delay_time=4957884, *p_task_={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:0, time_interval:250.00ms, expected_run_time:09:02:29.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.725367] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4016] task executed too delay(delay_time=4957929, *p_task_={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:0, time_interval:250.00ms, expected_run_time:09:02:29.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.725413] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4016] task executed too delay(delay_time=3972402, *p_task_={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:0, time_interval:500.00ms, expected_run_time:09:02:29.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.725486] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] task executed too delay(delay_time=3972446, *p_task_={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:0, time_interval:500.00ms, expected_run_time:09:02:29.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.725539] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] task executed too delay(delay_time=974136, *p_task_={this:0x7f99319fc190, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc470, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:0, time_interval:10.00s, expected_run_time:09:02:38.720, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:28.759492] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:278) [14905][T1001_TimerWK8_Flush][T1001][YB42AC4D0007-00063C4977399C31-0-0] [lt=3961146] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-08-14 09:02:29.725932] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [14905][T1001_TimerWK8][T1001][Y0-0000000000000000-0-0] [lt=964973][errcode=-4389] timer task too much delay in priority queue(token={this:0x7f9931bb30d8, timer:0x7f99333a15a8, task:0x7f99333a1608, task_type:"N9oceanbase7storage10checkpoint19ObCheckPointService20ObTraversalFlushTaskE", scheduled_time:1755162051762421, pushed_time:1755162052836253, delay:5000000}, delay_in_priority_queue=1073832, threashold=1000000) [567]OB_MALLOC COST TOO MUCH TIME, cost_time=186124, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [729]OB_MALLOC COST TOO MUCH TIME, cost_time=310598, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:01:02.757136] EDIAG eloop_delay_warn (debug.h:93) [319][pnio1][T0][Y0-0000000000000000-0-0] [lt=15821][errcode=0] LOGGER COST TOO MUCH TIME, cost: 86856198, owner: ObLog, click_count: 5, time dist:[FORMAT_END=8, ObMalloc#start=1, ObMalloc#end=971080, ALLOC_END=85885108, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x27ad6769 0x27a8a7e8 0x97b7bb0 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:29.725951] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [14905][T1001_TimerWK8][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f9931bb30d8, timer:0x7f99333a15a8, task:0x7f99333a1608, task_type:"N9oceanbase7storage10checkpoint19ObCheckPointService20ObTraversalFlushTaskE", scheduled_time:1755162051762421, pushed_time:1755162052836253, delay:5000000}, delay_in_thread_pool=88901485, threashold=500000) [2025-08-14 09:02:29.725963] WDIAG alarm_if_necessary (ob_timer_service.cpp:143) [14905][T1001_TimerWK8][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task's elapsed_time is greater than period(token={this:0x7f9931bb30d8, timer:0x7f99333a15a8, task:0x7f99333a1608, task_type:"N9oceanbase7storage10checkpoint19ObCheckPointService20ObTraversalFlushTaskE", scheduled_time:1755162051762421, pushed_time:1755162052836253, delay:5000000}, elapsed_time=7988193, period=5000000) [2025-08-14 09:02:29.727761] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.737334] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:29.728, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:29.737359] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f734c030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891807390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:29.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:29.737840] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:29.737849] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.737856] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:29.740655] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EEB1-0-0] [lt=3833][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571fc8b0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcb90, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:0, time_interval:3.00s, expected_run_time:09:02:31.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.740680] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EEB1-0-0] [lt=23][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:02:29.740|total=3.93s) [2025-08-14 09:02:28.722934] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:943) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE3-0-0] [lt=2025647][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f98dd75fa98, id:3221590966, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1002, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f98b5c049a0}) [2025-08-14 09:02:29.742180] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:29.727110] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=2739] current monitor number(seq_=-1) [2025-08-14 09:01:02.757136] WDIAG eloop_delay_warn (debug.h:93) [319][pnio1][T0][Y0-0000000000000000-0-0] [lt=15821][errcode=0] PNIO [delay_warn] eloop handle events delay high: 827006, malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:29.732985] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:29.744733] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11729][errcode=0] epoll handler too much time :11759 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:29.744748] WDIAG pktc_handle_req_queue (pktc_post.h:138) [319][pnio1][T0][Y0-0000000000000000-0-0] [lt=86856198][errcode=0] PNIO [delay_warn] delay high: 85975602 [2025-08-14 09:02:29.730029] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=957225] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:29.744771] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [319][pnio1][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559637, sock=0x7f995f5e54a8, code=-47 [2025-08-14 09:02:29.744845] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1433] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:29.745326] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:29.745397] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [319][pnio1][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559645, sock=0x7f995f5e54a8, code=-47 [2025-08-14 09:02:29.746322] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:29.746384] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [319][pnio1][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559653, sock=0x7f995f5e54a8, code=-47 [2025-08-14 09:02:29.746406] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [319][pnio1][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559661, sock=0x7f995f5e54a8, code=-47 [2025-08-14 09:02:29.746423] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [319][pnio1][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1002, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162100037036", warnings:[]}, tenant_id_=1002, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:02:29.746461] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [319][pnio1][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559669, sock=0x7f995f5e54a8, code=-47 [2025-08-14 09:02:29.746479] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [319][pnio1][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162100821439", warnings:[]}, tenant_id_=1, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:02:29.746495] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [319][pnio1][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559677, sock=0x7f995f5e54a8, code=-47 [2025-08-14 09:02:29.746487] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=3820] get wrs ts(ls_id={id:1001}, delta=405016761, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:29.746507] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [319][pnio1][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1002, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162101241558", warnings:[]}, tenant_id_=1002, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:02:29.746508] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:29.746521] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [319][pnio1][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559685, sock=0x7f995f5e54a8, code=-47 [2025-08-14 09:02:29.746532] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [319][pnio1][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162103734987", warnings:[]}, tenant_id_=1, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:02:29.746560] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=400003392, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:29.746570] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [319][pnio1][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559692, sock=0x7f995f5e54a8, code=-47 [2025-08-14 09:02:29.746571] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:29.746581] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [319][pnio1][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1002, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162107877754", warnings:[]}, tenant_id_=1002, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:02:29.746594] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [319][pnio1][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559700, sock=0x7f995f5e54a8, code=-47 [2025-08-14 09:02:29.746604] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [319][pnio1][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162109048461", warnings:[]}, tenant_id_=1, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:02:29.746617] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [319][pnio1][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559708, sock=0x7f995f5e54a8, code=-47 [2025-08-14 09:02:29.746627] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [319][pnio1][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162109548721", warnings:[]}, tenant_id_=1, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:02:29.746638] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [319][pnio1][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559717, sock=0x7f995f5e54a8, code=-47 [2025-08-14 09:02:29.746700] INFO [CLOG] update_upstream (ob_remote_location_adaptor.cpp:76) [829][T1002_LogRessvr][T1002][YB42AC4D0007-00063C495D199C16-0-0] [lt=1009249] primary tenant, just skip [2025-08-14 09:02:29.725034] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=460491][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:29.263|207=461.19ms|210=0us|222=1us|total=482.93ms) [2025-08-14 09:02:29.746778] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=21711][errcode=-4389] timer task use too much time(end=1755162149746778, start=1755162149263842, delta=482936) [2025-08-14 09:02:29.746795] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=241308, expected_run_ts_us_=1755162149505487, ObClockGenerator::getRealClock()=1755162149746795, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:29.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.746835] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] task scheduled too delay(scheduled_delay=157518, expected_run_ts_us_=1755162149589316, ObClockGenerator::getRealClock()=1755162149746835, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:29.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.746882] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:29.746901] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [373][Occam][T1][YB42AC4D0007-00063C4960599C9C-0-0] [lt=2029136][errcode=-4038] open result set failed(ret=-4038) [2025-08-14 09:02:29.747078] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1001}, delta=405016761, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:29.747095] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:29.747114] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=400003392, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:29.747124] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:29.747515] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=991224] [COMPACTION DAG STATUS] (start_time=1755162138752161, end_time=1755162149747514, dag_status={}) [2025-08-14 09:02:29.747870] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=962818] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:29.747968] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aae10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:29.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:29.747993] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:29.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:29.748010] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e67b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:29.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:29.748022] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:29.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:29.748061] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:29.748071] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.748080] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:29.748529] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=26] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:29.748550] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=20] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3910) [2025-08-14 09:02:29.748561] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:29.748586] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=20] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:29.748600] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:227) [1031][PxTargetMgr0][T1001][Y0-0000000000000000-0-0] [lt=11] refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3911) [2025-08-14 09:02:29.748610] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:29.748786] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:29.748798] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=12] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3912) [2025-08-14 09:02:29.748797] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1001}, delta=405016761, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:29.748807] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:29.748811] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:29.748856] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:29.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.750171] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:29.750196] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:02:29.750217] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:29.750229] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:29.750273] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:29.750351] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:29.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.750375] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:29.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.750415] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [354][EvtHisUpdTask][T1][YB42AC4D0007-00063C495FE9A745-0-0] [lt=1010313][errcode=-4038] open result set failed(ret=-4038) [2025-08-14 09:02:29.751412] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:29.751466] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=14] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:29.754181] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=11] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:29.754206] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162149754176], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:29.754224] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162149754176], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:29.754248] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162149754176]}) [2025-08-14 09:02:29.754260] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:29.754271] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:29.754284] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:29.754292] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162149754282], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:29.754311] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162149754282]}) [2025-08-14 09:02:29.754505] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:29.749104] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:02:29.750252] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:02:29.755196] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=4924][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:29.755214] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:29.755250] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:29.755263] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:01:49.731, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:01:42.751], idx=659, next_allow_print_ts="09:02:19.731") [2025-08-14 09:02:29.755276] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:29.755293] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:29.755305] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:29.755309] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=289] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:29.755319] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:29.755333] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:29.755506] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:29.755524] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:29.755536] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:02:29.755622] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:29.754528] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=22] get number(ret=-4038, service_type_=0, range=1, base_id=1755162149750541000, start_id=0, end_id=140295968341544) [2025-08-14 09:02:29.755780] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=1248][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.755795] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162149754500], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:29.756000] INFO [CLOG] operator() (ob_log_replay_service.cpp:1678) [14905][T1001_TimerWK8_ReplayProcessSt][T1001][Y0-0000000000000000-0-0] [lt=15] get_replay_process success(id={id:1}, submitted_log_size=945364892, unsubmitted_log_size=945364892, replayed_log_size=945364892, unreplayed_log_size=0) [2025-08-14 09:02:29.756043] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:174) [14905][T1001_TimerWK8_ReplayProcessSt][T1001][Y0-0000000000000000-0-0] [lt=20] dump tenant replay process(tenant_id=1001, unsubmitted_log_size(MB)=901, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=901, replayed_log_size(MB)=901, last_submitted_log_size(MB)=901, last_replayed_log_size(MB)=901, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-08-14 09:02:29.756059] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [14905][T1001_TimerWK8][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f9931bb29d8, timer:0x7f99375fb110, task:0x7f99332044d0, task_type:"N9oceanbase10logservice17ReplayProcessStatE", scheduled_time:1755162057738451, pushed_time:1755162057739916, delay:10000000}, delay_in_thread_pool=91986071, threashold=500000) [2025-08-14 09:02:29.757157] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=935290] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:02:29.757180] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:02:29.757194] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:29.757206] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:29.758482] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:801) [354][EvtHisUpdTask][T1][YB42AC4D0007-00063C495FE9A745-0-0] [lt=19][errcode=-4038] result set open failed(ret=-4038, executor={ObIExecutor:, sql:"INSERT INTO __all_rootservice_event_history (gmt_create, module, event, name1, value1, name2, value2, value3, value4, value5, value6, rs_svr_ip, rs_svr_port) VALUES (usec_to_time(1755161441756996), 'server', 'load_servers', 'ret', 0, 'has_build', 1, '', '', '', '', '172.77.0.7', 2882)"}) [2025-08-14 09:02:29.759267] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:29.759348] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=6032] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:02:29.759787] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=19] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:29.764676] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=44] gc stale ls task succ [2025-08-14 09:02:29.765279] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919bfe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aa810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:29.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:29.765322] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:29.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:29.765336] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:29.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:29.765348] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:29.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:29.765360] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:29.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:29.765372] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:29.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:29.765420] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:29.765440] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:29.765450] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:29.762630] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:29.770271] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:02:29.770452] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:801) [373][Occam][T1][YB42AC4D0007-00063C4960599C9C-0-0] [lt=16][errcode=-4038] result set open failed(ret=-4038, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1755161991751967), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.77.0.7:2882", reason:the only request', 'HAPPENED_TIME', '2025-08-14 08:44:54.846391', '', '', '172.77.0.7', 2882)"}) [2025-08-14 09:02:29.771284] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:29.772489] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:29.772520] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26] temporary file module use no blocks [2025-08-14 09:02:29.772531] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:29.772550] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=18] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:29.772563] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623051209292, flush_io_finished_ret_:0, flush_io_finished_round_:26322, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:29.772613] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=55] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:29.772630] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=15] temporary file module use no blocks [2025-08-14 09:02:29.772639] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:29.772702] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [556][T1_OBJLockGC][T1][Y0-0000000000000000-0-0] [lt=53][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fafd0, caller:ob_table_lock_service.cpp:start:276, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE6$_1273, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fb2b0, time_wheel:0x7f994b19db50, thread_pool:0x7f994b19d330, is_running:1, time_interval:10.00s, expected_run_time:09:02:29.696, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.775363] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:29.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.785508] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72707b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a4010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:29.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:29.790853] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:02:29.795587] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:29.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:29.800673] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=20] PNIO [ratelimit] time: 1755162149800670, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007597, add_bytes: 0 [2025-08-14 09:02:29.805638] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:29.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.805696] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739acb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196bb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:29.798, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:29.815734] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:29.806, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:29.815765] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:29.809, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:29.825792] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:29.815, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:29.836612] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=4622] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:29.837883] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=3410] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:29.842720] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2104] get wrs ts(ls_id={id:1}, delta=602103527, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:29.842754] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:29.842895] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9270-0-0] [lt=30][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162149842176) [2025-08-14 09:02:29.842919] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9270-0-0] [lt=19][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162149842176}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162149842203}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162149842203}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:29.843012] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [316][pnio1][T1][YB42AC4D0007-00063C49607B9270-0-0] [lt=20][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1, delta=589883, last_succ_cluster_heartbeat_tstamp_=1755162149253128) [2025-08-14 09:02:29.843563] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=602104596, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:29.843577] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:29.845944] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:29.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:29.845972] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:29.832, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:29.847158] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=400105220, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:29.847184] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:29.847688] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1001}, delta=405119651, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:29.847698] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:29.848924] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:29.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.856023] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:29.841, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:29.857278] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:29.859407] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=54] LogLoopThread round_cost_time(us)(round_cost_time=75) [2025-08-14 09:02:29.866085] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:29.859, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:29.872743] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc4578, timer:0x7f9970f40f80, task:0x564d6d07a498, task_type:"N9oceanbase8observer26ObRemoteMasterRsUpdateTaskE", scheduled_time:1755162028742204, pushed_time:0, delay:600000000}, running_task_token={this:0x7f9970fc5fb8, timer:0x7f9970f40f80, task:0x564d727182f0, task_type:"N9oceanbase5share16ObAshRefreshTaskE", scheduled_time:1755161611717613, pushed_time:1755161614757742, delay:120000000}, total_delay=121130538, threashold=1000000, tenant_id=500) [2025-08-14 09:02:29.936670] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=19] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:29.937939] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:29.943654] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=602203555, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:29.943680] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:29.943781] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=30] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:02:29.947255] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=400205741, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:29.947274] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:29.947787] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=405219110, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:29.947804] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:29.948991] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:29.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:29.957321] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:29.961229] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:02:30.027151] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:30.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:30.036739] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=18] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:30.038020] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=23] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:30.043753] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=602304001, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:30.043774] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:30.047321] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=400305796, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:30.047339] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:30.047844] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=405319165, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:30.047855] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:30.049051] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:29.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:30.057378] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:30.136768] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=20] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:30.138075] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=17] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:30.143782] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=602403863, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:30.143802] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:30.149122] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:30.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:30.157541] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:30.236834] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=20] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:30.238133] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=17] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:30.243841] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=602503944, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:30.243866] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:30.244013] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9274-0-0] [lt=26][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162150243319) [2025-08-14 09:02:30.244050] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9274-0-0] [lt=30][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162150243319}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162150243336}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162150243336}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:30.244816] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=602505457, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:30.244839] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:30.249197] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:30.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:30.255704] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:30.255734] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:02:30.255756] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:30.255768] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:30.255789] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:02:30.255808] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:30.255822] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:30.255841] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:30.255861] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:01:49.731, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:01:42.751], idx=659, next_allow_print_ts="09:02:19.731") [2025-08-14 09:02:30.255874] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:30.255890] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:30.255903] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:30.255915] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:30.255938] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:30.256126] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:30.256141] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:30.256154] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:02:30.256270] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=13] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:30.256283] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162150256259], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:30.256298] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162150256259], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:30.256311] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162150256259]}) [2025-08-14 09:02:30.256325] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=10] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:30.256333] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162150256323], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:30.256344] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162150256323]}) [2025-08-14 09:02:30.256520] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:30.256532] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:30.256544] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162150256518], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:30.256556] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162150256518]}) [2025-08-14 09:02:30.257595] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:30.278727] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:30.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:30.308926] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:30.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:30.336895] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=19] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:30.338204] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=14] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:30.344839] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=602605470, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:30.344861] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:30.349258] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:30.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:30.357665] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:30.436944] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=21] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:30.438264] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=17] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:30.449333] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:30.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:30.457729] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:30.520390] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:30.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:30.537002] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=23] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:30.538312] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:30.549395] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:30.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:30.557787] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:30.627905] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:30.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:30.637059] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=19] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:30.638380] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:30.649455] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:30.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:30.657846] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [729]OB_MALLOC COST TOO MUCH TIME, cost_time=873865, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [2025-08-14 09:02:29.772654] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:29.778629] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:30.718194] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=938144][errcode=0] epoll handler too much time :939575 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:30.718883] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=430942970, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:30.718906] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=9, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [567]OB_MALLOC COST TOO MUCH TIME, cost_time=278024, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16448, [2025-08-14 09:02:30.721704] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73516b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196f510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:30.716, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:30.721997] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:30.722620] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=430947287, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:30.722640] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [857]OB_MALLOC COST TOO MUCH TIME, cost_time=578081, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:02:30.726370] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=400984459, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:30.726394] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:30.726995] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1001}, delta=405999149, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:30.727014] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:30.727036] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=392505237, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=251680, succ_cluster_heartbeat_count=1490, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161006924036004, v:0}, local_cluster_delta=1143802999, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:02:30.727077] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=31] get wrs ts(ls_id={id:1}, delta=400985780, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:30.727089] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:30.727691] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1001}, delta=405999149, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:30.727703] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:30.727906] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [952][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F60-0-0] [lt=27][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1002, svr="172.77.0.7:2882", version={val:1755161758221798000, v:0}, valid_part_count=0, total_part_count=2, generate_timestamp=1755162150147352) [2025-08-14 09:02:30.727931] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [952][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F60-0-0] [lt=19][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1002, req={req_server:"172.77.0.7:2882", version:{val:1755161758221798000, v:0}, valid_part_count:0, total_part_count:2, generate_timestamp:1755162150147352}, twrs={inited:true, tenant_id:1002, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162150727070}, server_version_for_stat:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162150727070}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:30.728063] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [316][pnio1][T1002][YB42AC4D0007-00063C49612B8F60-0-0] [lt=12][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1002, delta=1897574, last_succ_cluster_heartbeat_tstamp_=1755162148830488) [2025-08-14 09:02:30.728140] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=400985780, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:30.728156] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:30.728736] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=405999149, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:30.728756] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:29.854860] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=71418][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:29.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:30.728868] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=872684][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:29.854|207=874.00ms|210=0us|222=2us|total=874.03ms) [2025-08-14 09:02:30.728893] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-4389] timer task use too much time(end=1755162150728892, start=1755162149854858, delta=874034) [2025-08-14 09:02:30.728919] WDIAG [MVCC] study (ob_multi_version_garbage_collector.cpp:319) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4012] study min unallocated GTS failed(ret=-4012) [2025-08-14 09:02:30.728937] INFO [MVCC] study (ob_multi_version_garbage_collector.cpp:392) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13] study multi version garabage collector end(ret=-4012, this={this:0x7f99200ff540, last_study_timestamp:1755161007611441, last_refresh_timestamp:1755161188828450, last_reclaim_timestamp:1755161664766510, last_sstable_overflow_timestamp:0, has_error_when_study:true, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1755161006924036004, v:0}, is_inited:true}, min_unallocated_GTS={val:4611686018427387903, v:0}, min_unallocated_GTS={val:4611686018427387903, v:0}, max_committed_txn_version={val:4611686018427387903, v:0}, min_active_txn_version={val:4611686018427387903, v:0}) [2025-08-14 09:02:30.728965] WDIAG [MVCC] repeat_study (ob_multi_version_garbage_collector.cpp:186) [868][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4012] repeat study failed, we will retry immediately(ret=-4012, this={this:0x7f99200ff540, last_study_timestamp:1755161007611441, last_refresh_timestamp:1755161188828450, last_reclaim_timestamp:1755161664766510, last_sstable_overflow_timestamp:0, has_error_when_study:true, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1755161006924036004, v:0}, is_inited:true}, current_timestamp=1755162148749103) [2025-08-14 09:02:29.961307] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=32] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:02:30.730130] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=767617] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:02:30.730176] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:02:30.730195] INFO [COMMON] print_io_status (ob_io_manager.cpp:2639) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4628, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:486303, total:486308) [2025-08-14 09:02:30.730219] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, param_config_:{memory_limit:536870912, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=865568, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:02:30.730268] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2826) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=45] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:2, avg_bw:4628, [delay/us]: prepare:0, schedule:0, submit:600000000, device:486303, total:486308) [2025-08-14 09:02:30.730288] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:02:30.730302] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:30.730315] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:30.730325] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SERVER] net_in=1kB/s, net_out=1kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=1280000kB/s [2025-08-14 09:02:30.248891] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:30.730480] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=480859] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3913) [2025-08-14 09:02:30.730489] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:30.730510] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:30.730520] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3914) [2025-08-14 09:02:30.730528] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:30.730720] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=30] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:30.730731] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3915) [2025-08-14 09:02:30.730739] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:29.768937] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=988066] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:02:30.719406] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [758][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB90AB-0-0] [lt=31][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1001, svr="172.77.0.7:2882", version={val:1755161719775179002, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162149843051) [2025-08-14 09:02:30.730899] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [758][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB90AB-0-0] [lt=11469][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1001, req={req_server:"172.77.0.7:2882", version:{val:1755161719775179002, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162149843051}, twrs={inited:true, tenant_id:1001, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162150718937}, server_version_for_stat:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162150718937}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:30.732479] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=430951794, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:30.732508] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=27] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [567]OB_MALLOC COST TOO MUCH TIME, cost_time=288380, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=278037], seq:[0, 1] [2025-08-14 09:02:30.733632] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=602988457, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:30.733657] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=3, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:29.770488] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:946) [373][Occam][T1][YB42AC4D0007-00063C4960599C9C-0-0] [lt=30][errcode=-4038] execute failed(ret=-4038, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1755161991751967), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.77.0.7:2882", reason:the only request', 'HAPPENED_TIME', '2025-08-14 08:44:54.846391', '', '', '172.77.0.7', 2882)"}, retry_cnt=16, local_sys_schema_version=1753073979411448, local_tenant_schema_version=1753073979411448) [2025-08-14 09:02:29.770831] INFO [SHARE.LOCATION] vtable_get (ob_vtable_location_service.cpp:134) [1003][BlackListServic][T1][YB42AC4D0007-00063C495F09BD69-0-0] [lt=17475] vtable nonblock renew success(tenant_id=1, table_id=12287, current time=1755162149745451, last renew time=1755161306768636, cache expire time="8s") [2025-08-14 09:02:29.770912] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:946) [354][EvtHisUpdTask][T1][YB42AC4D0007-00063C495FE9A745-0-0] [lt=30][errcode=-4038] execute failed(ret=-4038, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_rootservice_event_history (gmt_create, module, event, name1, value1, name2, value2, value3, value4, value5, value6, rs_svr_ip, rs_svr_port) VALUES (usec_to_time(1755161441756996), 'server', 'load_servers', 'ret', 0, 'has_build', 1, '', '', '', '', '172.77.0.7', 2882)"}, retry_cnt=22, local_sys_schema_version=1753073979411448, local_tenant_schema_version=1753073979411448) [2025-08-14 09:02:29.744266] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [723][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=15914] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=16, total_destructed_detector=16, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f99333562b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-08-14 09:02:30.735178] WDIAG [DETECT] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [723][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=989411][errcode=0] cost too much time:ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t), (this=|threshold=100.00ms|start at 09:02:30.735|total=1.89s) [2025-08-14 09:02:30.735940] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:30.735998] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:30.720, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:30.736026] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:30.727, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:30.736039] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:30.727, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:30.736051] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72851b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:30.721, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:30.736204] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [317][pnio1][T1001][YB42AC4D0007-00063C4960CB90AB-0-0] [lt=16][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1001, delta=1507410, last_succ_cluster_heartbeat_tstamp_=1755162149228793) [2025-08-14 09:02:30.737119] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=19] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:30.737227] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4746] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:29.994523] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:29.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:30.723984] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] EASY SLOW: thread: 7f99542fb300, lastrun: 1755162115.752503 cost: 34.971479 loop:26416, slowcnt: 292 [2025-08-14 09:02:30.737500] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=741851][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:29.994|207=742.98ms|210=0us|222=2us|total=742.99ms) [2025-08-14 09:02:30.737521] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task use too much time(end=1755162150737520, start=1755162149994521, delta=742999) [2025-08-14 09:02:30.737537] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=492903, expected_run_ts_us_=1755162150244633, ObClockGenerator::getRealClock()=1755162150737537, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:30.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:30.737554] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:30.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:30.718124] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=944041] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623051209353, flush_io_finished_ret_:0, flush_io_finished_round_:26310, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:30.720653] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:30.738965] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18284][errcode=0] epoll handler too much time :18331 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:30.722385] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:455) [14905][T1001_TimerWK8_SSTableDefragme][T1001][Y0-0000000000000000-0-0] [lt=15] skip defragment task(ret=0, need_defragment=false) [2025-08-14 09:02:30.739323] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:30.740636] INFO [RPC.OBRPC] do_client_loop (ob_net_keepalive.cpp:679) [406][KeepAliveClient][T0][Y0-0000000000000000-0-0] [lt=13754] dump dest keepalive data states, addr: "172.77.0.7:2882", last_write_ts_=1755162149611695, last_read_ts_=1755162149811061, in_black_=0 [2025-08-14 09:02:30.734226] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=602988457, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:30.740714] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6476] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:30.742260] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=151] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:30.742296] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=304] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:30.739004] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:497) [14905][T1001_TimerWK8_SSTableDefragme][T1001][Y0-0000000000000000-0-0] [lt=16589] successfully defragment data blocks(ret=0, rewrite_cnt=0, block_used_size_.count()=16) [2025-08-14 09:02:30.742482] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:515) [14905][T1001_TimerWK8_SSTableDefragme][T1001][Y0-0000000000000000-0-0] [lt=3470] successfully defragment data blocks(rewrite_cnt=0) [2025-08-14 09:02:30.739472] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1076) [373][Occam][T1][YB42AC4D0007-00063C4960599C9C-0-0] [lt=962438][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:16, local_retry_times:16, err_:-4038, err_:"OB_NOT_MASTER", retry_type:0, client_ret:-4012}, need_retry=false) [2025-08-14 09:02:30.740302] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1076) [354][EvtHisUpdTask][T1][YB42AC4D0007-00063C495FE9A745-0-0] [lt=962052][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:22, local_retry_times:22, err_:-4038, err_:"OB_NOT_MASTER", retry_type:0, client_ret:-4012}, need_retry=false) [2025-08-14 09:02:30.745529] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:30.746345] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:30.746388] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=17481] current monitor number(seq_=-1) [2025-08-14 09:02:30.746494] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=14718] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:30.746889] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:30.746956] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:30.737, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:30.746992] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:30.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:29.859444] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1001, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:8571658, max_lsn:7323687369, max_scn:{val:1755161749741193002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:7278610047}, reuse_lsn:{lsn:7323686131}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:7281744329}}, last_submit_lsn:{lsn:7323686131}, last_submit_end_lsn:{lsn:7323687369}, last_submit_log_id:8571658, last_submit_log_pid:49, max_flushed_lsn:{lsn:7323686009}, max_flushed_end_lsn:{lsn:7323686131}, max_flushed_log_pid:49, committed_end_lsn:{lsn:7323686009}, last_slide_log_id:8571656, last_slide_scn:{val:1755161747742105001, v:0}, last_slide_lsn:{lsn:7323685887}, last_slide_end_lsn:{lsn:7323686009}, last_slide_log_pid:49, last_slide_log_accum_checksum:2505458373, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f990112a530}, [StateMgr]={this:0x7f990112e9b0, self:"172.77.0.7:2882", palf_id:1001, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"PENDING", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:49}, leader:"0.0.0.0:0", leader_epoch:-1, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:7323687369}, scan_disk_log_finished:true, last_check_start_id:8571657, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161767744317, palf_role_change_cb:0x7f9901135470, allow_vote:true}, [ConfigMgr]={palf_id:1001, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:49, prev:{version:1, config:{config_version:{proposal_id:48, config_seq:47}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:49, config_seq:48}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_mode_pid:49}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_end_lsn:{lsn:7323685240}, prev_mode_pid:49}, reconfig_barrier:{prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_end_lsn:{lsn:7323685240}, prev_mode_pid:49}, persistent_config_version:{proposal_id:49, config_seq:48}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755161739755312, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755161739755312, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f990112bc70}, [ModeMgr]={palf_id:1001, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, accepted_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1001, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1753069671378640001, v:0}, base_lsn_for_block_gc:{lsn:7247314944}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:49}, log_config_meta:{version:3, proposal_id:49, prev:{version:1, config:{config_version:{proposal_id:48, config_seq:47}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:49, config_seq:48}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_mode_pid:49}, log_snapshot_meta:{version:2, base_lsn:{lsn:7247314944}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1753069671378640000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}}, log_meta_storage:{log_tail:{lsn:765952}, readable_log_tail:{lsn:765952}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7381524480}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/meta", dir_fd:232, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66338816, block_header_serialize_buf_:0x7f9901130bb8, flashback_version:0}, log_storage:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, palf_epoch:2, last_purge_throttling_ts:1755161769777679, this:0x7f990112f0f0}, [Reconfirm]={palf_id:1001, self:"172.77.0.7:2882", state:"WAITING_LOG_FLUSHED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:1755161769777679, this:0x7f990112eb30}) [2025-08-14 09:02:30.747312] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=886519] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [2025-08-14 09:02:30.747329] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15] current log_task status(palf_id=1001, self="172.77.0.7:2882", log_id=8571657, log_task={header:{begin_lsn:{lsn:7323686009}, end_lsn:{lsn:7323686131}, log_id:8571657, min_scn:{val:1755161749741193000, v:0}, max_scn:{val:1755161749741193000, v:0}, data_len:66, proposal_id:49, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:7323686009}, data_checksum:2222604892, accum_checksum:1531236791, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1755161749745909, freeze_ts:1755161749745914, submit_ts:1755161749745940, flushed_ts:1755161785758706, gen_to_freeze cost time:5, gen_to_submit cost time:31, submit_to_flush cost time:36012766}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [2025-08-14 09:02:30.747804] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [14780][T1001_TimerWK4_SSTableGC][T1001][YB42AC4D0007-00063C4974099C4E-0-0] [lt=6545][errcode=0] cost too much time:mds_dump_node.h:int oceanbase::storage::mds::MdsDumpNode::init(const uint8_t, const uint8_t, const UserMdsNode &, oceanbase::common::ObIAllocator &) [K = oceanbase::compaction::ObMediumCompactionInfoKey, V = oceanbase::compac..., (this=|threshold=1.00ms|start at 09:02:29.757|219=970.75ms|238=19.68ms|total=990.46ms) [2025-08-14 09:02:30.748451] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=37] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:02:30.748488] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=15] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:30.749942] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:30.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:30.750294] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:30.742496] INFO [STORAGE] ~ObITabletFilterOp (ob_tablet_pointer.cpp:751) [14905][T1001_TimerWK8_SSTableDefragme][T1001][Y0-0000000000000000-0-0] [lt=9] tablet skip filter destructed(total_cnt=0, skip_cnt=0, not_in_mem_cnt=0, invalid_attr_cnt=0, total_tablet_cnt=136172, total_skip_cnt=367, not_in_mem_tablet_cnt=0, invalid_attr_tablet_cnt=0) [2025-08-14 09:02:30.751481] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [14905][T1001_TimerWK8][T1001][Y0-0000000000000000-0-0] [lt=8970][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f9931bb2738, timer:0x7f99375dd110, task:0x7f99357f5310, task_type:"N9oceanbase12blocksstable21ObSharedMacroBlockMgr26ObBlockDefragmentationTaskE", scheduled_time:1755162074767333, pushed_time:1755162074775560, delay:30000000}, delay_in_thread_pool=74980522, threashold=500000) [2025-08-14 09:02:30.755534] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=29] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:30.755590] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=13] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:30.755634] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:30.755703] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:30.757368] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:30.757408] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:02:30.757428] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:30.757451] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:30.757464] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:02:30.757481] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:30.757492] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:30.757511] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:30.757522] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:01:49.731, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:01:42.751], idx=659, next_allow_print_ts="09:02:19.731") [2025-08-14 09:02:30.757538] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:30.757552] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:30.757564] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:30.757574] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:30.757587] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:30.757748] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:30.757765] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:30.757776] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:02:30.757875] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:30.757899] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162150757870], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:30.757914] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162150757870], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:30.757928] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162150757870]}) [2025-08-14 09:02:30.757939] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:30.757949] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:30.757960] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=8] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:30.757968] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162150757959], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:30.757978] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162150757959]}) [2025-08-14 09:02:30.758149] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:30.758163] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:30.758150] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:30.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:30.758178] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162150758143], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:30.758181] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891807210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:30.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:30.758200] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:30.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:30.758193] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162150758143]}) [2025-08-14 09:02:30.758212] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:30.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:30.758379] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:30.758454] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:30.758481] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:30.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:30.760731] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=12] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:30.762266] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6285] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:30.765731] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=19] gc stale ls task succ [2025-08-14 09:02:30.769322] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d67b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:30.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:30.770515] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:30.771317] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:02:30.774719] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=30] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:30.774752] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=25] temporary file module use no blocks [2025-08-14 09:02:30.774763] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:30.774778] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:30.774794] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=16] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623052212362, flush_io_finished_ret_:0, flush_io_finished_round_:26323, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:30.777305] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:30.780689] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:30.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:30.783566] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=39] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:30.783593] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=23] temporary file module use no blocks [2025-08-14 09:02:30.783604] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:30.783617] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:30.783632] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623052223270, flush_io_finished_ret_:0, flush_io_finished_round_:26291, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:30.790917] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=19] gc stale ls task succ [2025-08-14 09:02:30.793466] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891807090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:30.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:30.793509] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739aa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:30.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:30.803528] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:30.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:30.803579] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:30.791, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:30.803597] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ecf10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:30.791, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:30.803610] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:30.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:30.803623] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:30.799, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:30.808285] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=20] PNIO [ratelimit] time: 1755162150808282, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007612, add_bytes: 0 [2025-08-14 09:02:30.813636] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:30.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:30.813668] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7351e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196f990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:30.803, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:30.813682] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a22b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:30.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:30.813695] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:30.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:30.828268] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=401087050, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:30.828295] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:30.828333] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=31] get_max_applied_scn(scn={val:1755161744727824001, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:49, palf_committed_end_lsn_:{lsn:7323686009}, palf_committed_end_scn_:{val:1755161747742105001, v:0}, last_check_scn_:{val:1755161749741193000, v:0}, max_applied_cb_scn_:{val:1755161744727824001, v:0}}) [2025-08-14 09:02:30.828358] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] get_min_unreplayed_log_info(lsn={lsn:7323685240}, scn={val:1755100946796590002, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873485, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:8946764, ret_code_:0}}, next_to_submit_lsn_:{lsn:7323685240}, next_to_submit_scn_:{val:1755100946796590002, v:0}, base_lsn_:{lsn:7180210176}, base_scn_:{val:1753840803166611000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:7323685240}, end_lsn:{lsn:7323685240}, read_buf:{buf_len_:3747840, buf_:0x7f98f7a05000}, block_size:67104768, log_storage_:0x7f9901131bf0, log_storage_:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:28, read_io_size:98508800, read_disk_cost_ts:645339, start_lsn:{lsn:7180210176}}}}}}) [2025-08-14 09:02:30.828972] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:30.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:30.829020] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=90] get wrs ts(ls_id={id:1001}, delta=406100419, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:30.829036] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:30.829064] INFO [STORAGE.TRANS] amend (ob_tenant_weak_read_server_version_mgr.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [WRS] tenant has no valid partition on current server, generate weak read snapshot version based on current timestamp(tenant_id=1002, epoch_tstamp_=1755162150828243, total_part_count_=2, valid_inner_part_count_=0, valid_user_part_count_=0, version_={val:1755161758221798000, v:0}, base_version_when_no_valid_partition={val:1755161758221798000, v:0}) [2025-08-14 09:02:30.832441] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=431056221, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:30.832475] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=31] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:30.833768] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ebe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:30.826, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:30.833786] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161101770923, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755161572801635) [2025-08-14 09:02:30.833796] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7350530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ea90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:30.824, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:30.834382] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=603095057, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:30.834403] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:30.837182] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=17] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:30.839385] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=20] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:30.848509] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:02:30.852292] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:30.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:30.858509] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:30.874028] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fbbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:30.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:30.884085] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72860b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:30.873, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:30.928239] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26] get wrs ts(ls_id={id:1}, delta=401186329, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:30.928265] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:30.928893] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1001}, delta=406200764, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:30.928911] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:30.929024] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:30.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:30.932507] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=431156563, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:30.932532] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:30.934374] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=603195325, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:30.934404] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:30.937211] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=28] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:30.939397] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:30.944572] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=28] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:02:30.952342] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:30.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:30.958540] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:30.990034] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:30.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:31.025134] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:31.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:31.028316] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=401286464, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:31.028337] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.028968] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1001}, delta=406300893, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:31.028983] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.029043] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [952][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F62-0-0] [lt=33][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1002, svr="172.77.0.7:2882", version={val:1755161758221798000, v:0}, valid_part_count=0, total_part_count=2, generate_timestamp=1755162151028270) [2025-08-14 09:02:31.029071] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [952][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F62-0-0] [lt=21][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1002, req={req_server:"172.77.0.7:2882", version:{val:1755161758221798000, v:0}, valid_part_count:0, total_part_count:2, generate_timestamp:1755162151028270}, twrs={inited:true, tenant_id:1002, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162151028304}, server_version_for_stat:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162151028304}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:31.029092] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=401287524, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:31.029109] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.029176] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:30.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:31.029748] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1001}, delta=406300893, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:31.029774] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.030675] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:02:31.030733] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:02:31.030779] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:02:31.030834] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:02:31.030857] INFO [COMMON] print_io_status (ob_io_manager.cpp:2639) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:3, norm_iops:0, bw:13623, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:1585, total:1590) [2025-08-14 09:02:31.030889] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, param_config_:{memory_limit:536870912, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=865568, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:02:31.030950] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2826) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=56] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:4, avg_bw:13622, [delay/us]: prepare:0, schedule:0, submit:600000000, device:1585, total:1590) [2025-08-14 09:02:31.030972] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:02:31.030985] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:31.031009] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:31.031020] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SERVER] net_in=1kB/s, net_out=1kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=1280000kB/s [2025-08-14 09:02:31.032098] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [758][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB90AE-0-0] [lt=44][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1001, svr="172.77.0.7:2882", version={val:1755161719775179002, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162151031968) [2025-08-14 09:02:31.032121] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [758][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB90AE-0-0] [lt=22][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1001, req={req_server:"172.77.0.7:2882", version:{val:1755161719775179002, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162151031968}, twrs={inited:true, tenant_id:1001, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162150931907}, server_version_for_stat:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162150931907}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:31.032583] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=431256013, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:31.032603] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.033198] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=431256013, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:31.033212] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.034483] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=603294789, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:31.034506] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.037248] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=19] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:31.039453] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=19] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:31.052395] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:30.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:31.058601] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:31.129169] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=401387383, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:31.129194] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.129250] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:31.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:31.129822] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26] get wrs ts(ls_id={id:1001}, delta=406400752, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:31.129843] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.133287] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=431356874, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:31.133309] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.134407] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=603394819, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:31.134430] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.137327] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=19] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:31.139514] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=21] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:31.152456] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:31.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:31.158660] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:31.229274] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] get wrs ts(ls_id={id:1}, delta=401487511, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:31.229306] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.229325] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:31.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:31.229923] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1001}, delta=406501939, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:31.229944] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.230821] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:31.230846] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=24] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3916) [2025-08-14 09:02:31.230858] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:31.230868] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=6] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:31.230880] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=11] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3917) [2025-08-14 09:02:31.230887] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:31.231072] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:31.231083] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=11] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3918) [2025-08-14 09:02:31.231091] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:31.233356] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=431457738, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:31.233379] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.234466] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] get wrs ts(ls_id={id:1}, delta=603495469, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:31.234494] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.237386] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=19] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:31.239577] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=22] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:31.241395] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:31.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:31.251478] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:31.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:31.252503] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:31.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:31.257927] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:31.257957] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:02:31.257974] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:31.257990] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:31.258012] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:02:31.258031] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:31.258045] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:31.258085] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:31.258097] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:01:49.731, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:01:42.751], idx=659, next_allow_print_ts="09:02:19.731") [2025-08-14 09:02:31.258110] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:31.258126] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:31.258150] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:31.258162] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:31.258175] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:31.258383] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:31.258400] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:31.258414] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:02:31.258665] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=19] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:31.258681] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162151258661], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:31.258708] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162151258661], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:31.258726] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162151258661]}) [2025-08-14 09:02:31.258727] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:31.258740] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=11] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:31.258748] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162151258739], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:31.258768] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162151258739]}) [2025-08-14 09:02:31.258961] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:31.258983] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:31.258997] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162151258957], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:31.259009] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162151258957]}) [2025-08-14 09:02:31.276659] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:31.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:31.306843] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:31.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:31.329267] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=401587164, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:31.329294] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.329396] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:31.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:31.329887] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1001}, delta=406601631, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:31.329906] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.333428] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=27] get wrs ts(ls_id={id:1}, delta=431557391, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:31.333460] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=29] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.333526] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [758][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB90B1-0-0] [lt=24][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1001, svr="172.77.0.7:2882", version={val:1755161719775179002, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162151332771) [2025-08-14 09:02:31.333555] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [758][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB90B1-0-0] [lt=24][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1001, req={req_server:"172.77.0.7:2882", version:{val:1755161719775179002, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162151332771}, twrs={inited:true, tenant_id:1001, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162151332814}, server_version_for_stat:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162151332814}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:31.334186] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=431558487, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:31.334205] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.334393] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B927A-0-0] [lt=31][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162151334281) [2025-08-14 09:02:31.334427] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B927A-0-0] [lt=33][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162151334281}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162151233896}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162151233896}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:31.334507] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [317][pnio1][T1][YB42AC4D0007-00063C49607B927A-0-0] [lt=12][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1, delta=1090340, last_succ_cluster_heartbeat_tstamp_=1755162150244166) [2025-08-14 09:02:31.334843] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=603596209, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:31.334864] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.335428] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=603596209, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:31.335460] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.336906] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7faa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:31.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:31.336944] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72842b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:31.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:31.337454] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=21] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:31.339636] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=19] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:31.352579] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:31.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:31.358774] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:31.429366] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=401687291, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:31.429398] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=29] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.429474] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:31.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:31.430021] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] get wrs ts(ls_id={id:1001}, delta=406700660, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:31.430053] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=29] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.434308] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=431658279, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:31.434335] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.435539] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=603696000, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:31.435562] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.437518] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=17] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:31.439698] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:31.452625] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:31.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:31.458837] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:31.492811] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:31.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:31.528125] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:31.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:31.529415] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=401787923, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:31.529436] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.530093] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=31] get wrs ts(ls_id={id:1001}, delta=406801292, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:31.530116] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.537577] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=21] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:31.539762] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=31] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:31.552687] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:31.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:31.558896] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:31.628483] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:31.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:31.629328] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eef90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:31.614, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:31.629357] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eaf30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:31.614, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:31.639310] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=29] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:31.642309] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=30] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:31.653282] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:31.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:31.661259] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:31.677252] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=84175][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:1, time_interval:3.00s, expected_run_time:09:02:31.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:31.720587] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:31.530316] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:31.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:31.720708] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=190100][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:31.530|207=190.39ms|210=0us|222=1us|total=190.43ms) [2025-08-14 09:02:31.720750] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=-4389] timer task use too much time(end=1755162151720749, start=1755162151530314, delta=190435) [729]OB_MALLOC COST TOO MUCH TIME, cost_time=186758, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:02:31.721421] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=431945884, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:31.721464] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=39] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=3, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.722036] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] get wrs ts(ls_id={id:1}, delta=431945884, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:31.722060] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [567]OB_MALLOC COST TOO MUCH TIME, cost_time=186799, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:02:31.722682] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=603983608, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:31.722708] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=3, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.723294] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=603984767, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:31.723315] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.723378] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=401982090, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:31.723396] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.723957] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1001}, delta=406995459, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:31.723973] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.724060] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:6175) [14780][T1001_TimerWK4_SSTableGC][T1001][YB42AC4D0007-00063C4974099C4E-0-0] [lt=33] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:60113}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1755136957320163000}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1755159231184436000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1755159231184436000) [2025-08-14 09:02:31.724205] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:31.724882] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [952][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F67-0-0] [lt=33][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1002, svr="172.77.0.7:2882", version={val:1755161758221798000, v:0}, valid_part_count=0, total_part_count=2, generate_timestamp=1755162151629404) [2025-08-14 09:02:31.724920] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [952][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F67-0-0] [lt=32][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1002, req={req_server:"172.77.0.7:2882", version:{val:1755161758221798000, v:0}, valid_part_count:0, total_part_count:2, generate_timestamp:1755162151629404}, twrs={inited:true, tenant_id:1002, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162151723366}, server_version_for_stat:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162151723366}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:31.725031] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [316][pnio1][T1002][YB42AC4D0007-00063C49612B8F67-0-0] [lt=15][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1002, delta=695173, last_succ_cluster_heartbeat_tstamp_=1755162151029858) [2025-08-14 09:02:31.725573] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=401984261, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:31.725602] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.726163] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] get wrs ts(ls_id={id:1001}, delta=406997630, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:31.726181] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:01:04.759869] WDIAG [SQL] do_close_plan (ob_result_set.cpp:844) [337][EvtHisUpdTask][T1][YB42AC4D0007-00063C4960299C78-0-0] [lt=28][errcode=-4006] exec result is null(ret=-4006) [2025-08-14 09:02:31.732045] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:31.732070] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=24] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3919) [2025-08-14 09:02:31.732078] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:31.732095] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:31.732109] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=14] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3920) [2025-08-14 09:02:31.732114] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:01:06.734682] EDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=997084][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1006190, owner: ObLog, click_count: 5, time dist:[FORMAT_END=39818, ObMalloc#start=1, ObMalloc#end=966367, ALLOC_END=4, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ed35c 0x97aead0 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:31.732329] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=44] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:31.732350] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:227) [1031][PxTargetMgr0][T1002][Y0-0000000000000000-0-0] [lt=17] refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3921) [2025-08-14 09:02:31.732359] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 08:59:57.736483] EDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1808) [14910][TimerWK6_AutoLSLoc][T0][YB42AC4D0007-00063C4977899C57-0-0] [lt=5][errcode=-4012] LOGGER COST TOO MUCH TIME, cost: 153763524, owner: ObLog, click_count: 9, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=13379, ObMalloc#start=153750129, ObMalloc#end=2, ALLOC_END=1, APPEND_END=3, ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=1], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x124ef141 0x96a4025 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x1eeb57d8 0x1eeb4f81 0x201f4a4f 0x202038b9 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:31.732428] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=1006190][errcode=-4389] timer task use too much time(end=1755162151732427, start=1755162065736084, delta=85996343) [2025-08-14 08:59:57.736483] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1808) [14910][TimerWK6_AutoLSLoc][T0][YB42AC4D0007-00063C4977899C57-0-0] [lt=5][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-08-14 09:02:31.733098] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:31.738148] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:04.759869] EDIAG [SQL] do_close_plan (ob_result_set.cpp:844) [337][EvtHisUpdTask][T1][YB42AC4D0007-00063C4960299C78-0-0] [lt=28][errcode=-4006] LOGGER COST TOO MUCH TIME, cost: 86840112, owner: ObLog, click_count: 7, time dist:[FORMAT_END=1008265, ObMalloc#start=1, ObMalloc#end=963249, ObMalloc#start=84868588, ObMalloc#end=4, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x14944de2 0x9395c9f 0x9725f91 0x124dfd77 0x96a5a4d 0x97aa16f 0x124eac91 0x97a9944 0x261395f9 0x1f900517 0x1f8fe2bf 0x9a5f486 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:31.738384] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=13491][errcode=0] EASY SLOW: thread: 7f99542fb300, lastrun: 1755162115.752503 cost: 35.985878 loop:26416, slowcnt: 302 [2025-08-14 09:02:31.738456] WDIAG [SQL] do_close (ob_result_set.cpp:935) [337][EvtHisUpdTask][T1][YB42AC4D0007-00063C4960299C78-0-0] [lt=86840112][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-08-14 09:02:31.738492] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [337][EvtHisUpdTask][T1][YB42AC4D0007-00063C4960299C78-0-0] [lt=14][errcode=-4012] result set close failed(ret=-4012) [2025-08-14 09:02:31.738504] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:199) [337][EvtHisUpdTask][T1][YB42AC4D0007-00063C4960299C78-0-0] [lt=9][errcode=-4012] result set close failed(ret=-4012) [2025-08-14 09:02:31.741265] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=20773] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:31.741298] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=28] temporary file module use no blocks [2025-08-14 09:02:31.741310] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:31.741328] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=16] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:31.741343] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623053178698, flush_io_finished_ret_:0, flush_io_finished_round_:26311, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:31.732460] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] task scheduled too delay(scheduled_delay=84033483, expected_run_ts_us_=1755162067698974, ObClockGenerator::getRealClock()=1755162151732458, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fb3b0, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fb690, time_wheel:0x7f99589fd990, thread_pool:0x7f9959fff170, is_running:1, time_interval:2.00s, expected_run_time:09:01:07.698, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:30.774614] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=969313] blacklist_loop exec finished(cost_time=16, is_enabled=true, send_cnt=0) [2025-08-14 09:02:31.745633] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:31.745671] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:31.745701] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=19] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:31.746285] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:31.746830] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=23] current monitor number(seq_=-1) [2025-08-14 09:02:31.746913] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=12] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:31.747762] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:31.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:31.748085] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=20] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:31.748511] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:31.748548] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:31.748566] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=13] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:31.748720] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1079) [14910][TimerWK6_AutoLSLoc][T0][YB42AC4D0007-00063C4977899C57-0-0] [lt=153763524][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1001) [2025-08-14 09:02:31.739528] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:951) [337][EvtHisUpdTask][T1][YB42AC4D0007-00063C4960299C78-0-0] [lt=1020][errcode=-4012] failed to close result(close_ret=-4012, ret=-4012) [2025-08-14 09:02:31.751272] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:31.745745] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:31.752623] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1748) [14910][TimerWK6_AutoLSLoc][T0][YB42AC4D0007-00063C4977899C57-0-0] [lt=18][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1753069619, tenant_id=1001) [2025-08-14 09:02:31.753883] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:31.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:31.753945] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:02:31.754256] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:31.756434] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [14910][TimerWK6_AutoLSLoc][T0][YB42AC4D0007-00063C4977899C57-0-0] [lt=17][errcode=-4012] query failed(ret=-4012, conn=0x7f98de5266f0, start=1755161994776488, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-08-14 09:02:31.756469] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [14910][TimerWK6_AutoLSLoc][T0][YB42AC4D0007-00063C4977899C57-0-0] [lt=33][errcode=-4012] read failed(ret=-4012) [2025-08-14 09:02:31.759145] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [491][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=9] ====== tenant freeze timer task ====== [2025-08-14 09:02:31.759176] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1786) [491][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=17] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=130023424, ctx.total_memstore_used_=230686720, ctx.total_memstore_hold_=230686720, memstore_freeze_trigger_limit_=165650000, tenant_id=1) [2025-08-14 09:02:31.759198] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1790) [491][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=16] [TenantFreezer] oldest frozen memtable(list="") [2025-08-14 09:02:31.759227] INFO [STORAGE] need_re_freeze (ob_tx_data_table.cpp:1349) [491][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=11] tx data table need refreeze(ls_id={id:1}, max_freeze_interval=300000000, last_request_ts="2025-08-14 08:56:58.760231", last_freeze_ts="2025-08-14 08:56:58.760231", current_time="2025-08-14 09:02:31.751011") [2025-08-14 09:02:31.759281] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:582) [491][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=22] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=0, Tenant Active TxData Memory(MB)=0, Freeze TxData Trigger Memory(MB)=102, Total TxDataTable Hold Memory(MB)=3, Total TxDataTable Memory Limit(MB)=409) [2025-08-14 09:02:31.760295] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:31.761286] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:31.761320] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:31.761367] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=16] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:31.762604] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:31.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:31.762657] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=51][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:31.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:31.762671] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919827b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:31.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:31.762685] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:31.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:31.763465] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:31.765793] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=16] gc stale ls task succ [2025-08-14 09:02:31.760342] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:31.760389] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:31.775153] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14733][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [381]OB_MALLOC COST TOO MUCH TIME, cost_time=954859, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=34, WASH_BLOCK_END=2997, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=11945], seq:[0, 1, 2, 3, 4] [634]OB_MALLOC COST TOO MUCH TIME, cost_time=943510, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=33, WASH_BLOCK_END=943440, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=36], seq:[0, 1, 2, 3, 4] [651]OB_MALLOC COST TOO MUCH TIME, cost_time=907791, tenant_id=500, label=CommonArray, ctx_id=0, prio=0, size=15872, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1], seq:[0, 1] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=940691, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=46, WASH_BLOCK_END=940644], seq:[0, 1, 2] [337]OB_MALLOC COST TOO MUCH TIME, cost_time=952679, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=952671], seq:[0] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=936580, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=34, WASH_BLOCK_END=936546], seq:[0, 1, 2] [832]OB_MALLOC COST TOO MUCH TIME, cost_time=935662, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=43, WASH_BLOCK_END=935619], seq:[0, 1, 2] [2025-08-14 09:02:31.775183] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:31.775193] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:31.775203] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:01:20.739, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):371", last click ts=09:01:12.762], idx=473, next_allow_print_ts="09:01:50.739") [2025-08-14 09:02:31.775219] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:31.775227] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:31.775339] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=104][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:31.775349] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:01:49.731, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:01:42.751], idx=659, next_allow_print_ts="09:02:19.731") [2025-08-14 09:02:31.775359] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:31.775373] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:31.777415] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:31.826297] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=402084976, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:31.826325] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.826934] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1001}, delta=407098345, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:31.826945] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.845764] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=11] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:31.848147] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=15] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:31.853936] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:31.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:31.945797] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:31.948207] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=12] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:31.954000] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:31.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:32.045887] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=14] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:32.048243] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=19] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:32.054053] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:31.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:32.145951] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:32.148304] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=12] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:32.248379] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=13] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:32.716707] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=467606] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:31.760536] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:32.716884] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=954900][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162151760532], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:32.716903] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162151760532], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:32.716925] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162151760532]}) [2025-08-14 09:02:32.716935] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162151760532]}) [2025-08-14 09:02:32.716946] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:32.716954] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:32.716970] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=5] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:32.716977] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162152716966], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:32.716984] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162152716966]}) [2025-08-14 09:02:32.246011] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:32.717307] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=470571] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:32.154118] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:32.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:32.717336] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=562363][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:32.154|207=563.22ms|210=0us|222=2us|total=563.25ms) [2025-08-14 09:02:32.717363] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4389] timer task use too much time(end=1755162152717363, start=1755162152154115, delta=563248) [2025-08-14 09:02:32.716990] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162152716966]}) [2025-08-14 09:02:32.720118] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=2943][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:32.720135] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:32.720158] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162152720112], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:32.720173] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162152720112]}) [2025-08-14 09:02:32.720183] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162152720112]}) [730]OB_MALLOC COST TOO MUCH TIME, cost_time=928562, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=43, WASH_BLOCK_END=928518], seq:[0, 1, 2] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=918264, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=42, WASH_BLOCK_END=918222], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=903785, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=44, WASH_BLOCK_END=903740], seq:[0, 1, 2] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=898188, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=44, WASH_BLOCK_END=898144], seq:[0, 1, 2] [571]OB_MALLOC COST TOO MUCH TIME, cost_time=869047, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=43, WASH_BLOCK_END=869004], seq:[0, 1, 2] [861]OB_MALLOC COST TOO MUCH TIME, cost_time=868471, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=868431], seq:[0, 1, 2] [822]OB_MALLOC COST TOO MUCH TIME, cost_time=866077, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=35, WASH_BLOCK_END=866042], seq:[0, 1, 2] [733]OB_MALLOC COST TOO MUCH TIME, cost_time=865900, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=42, WASH_BLOCK_END=865858], seq:[0, 1, 2] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=858786, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=44, WASH_BLOCK_END=858742], seq:[0, 1, 2] [466]OB_MALLOC COST TOO MUCH TIME, cost_time=826879, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=57, WASH_BLOCK_END=826822], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=773104, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=50, WASH_BLOCK_END=773054], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=770186, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=44, WASH_BLOCK_END=770141], seq:[0, 1, 2] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=898239, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [729]OB_MALLOC COST TOO MUCH TIME, cost_time=900091, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=690226, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=52, WASH_BLOCK_END=690174], seq:[0, 1, 2] [1031]OB_MALLOC COST TOO MUCH TIME, cost_time=491256, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=65, WASH_BLOCK_END=491191], seq:[0, 1, 2] [337]OB_MALLOC COST TOO MUCH TIME, cost_time=958639, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=952689], seq:[0, 1] [552]OB_MALLOC COST TOO MUCH TIME, cost_time=949727, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=41, WASH_BLOCK_END=940017, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=9664], seq:[0, 1, 2, 3, 4] [2025-08-14 09:01:13.744513] EDIAG [ELECT] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [473][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 78863103, owner: ObLog, click_count: 5, time dist:[FORMAT_END=16, ObMalloc#start=1, ObMalloc#end=62, ALLOC_END=78863024, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ea7c4 0x9818f9c 0xf9d631d 0xf9d4120 0xfa09ca2 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:10.766765] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2984350, owner: ObLog, click_count: 5, time dist:[FORMAT_END=13, ObMalloc#start=1, ObMalloc#end=2984331, ALLOC_END=5, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:31.783728] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=24] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:32.720869] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=935719] temporary file module use no blocks [2025-08-14 09:02:32.720877] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:32.720889] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:32.720901] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623053223425, flush_io_finished_ret_:0, flush_io_finished_round_:26292, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:31.801297] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=308][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:31.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:31.790983] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=18] gc stale ls task succ [2025-08-14 09:02:32.720989] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=918300][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:31.801|207=919.69ms|210=0us|222=1us|total=919.71ms) [2025-08-14 09:02:32.721012] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4389] timer task use too much time(end=1755162152721011, start=1755162151801295, delta=919716) [2025-08-14 09:02:32.721025] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=710992, expected_run_ts_us_=1755162152010031, ObClockGenerator::getRealClock()=1755162152721024, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:32.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:32.721041] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:32.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:31.850899] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [571][T1_ArcSender][T1][YB42AC4D0007-00063C495BA99C16-0-0] [lt=600391][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:02:31.853999] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:02:31.854256] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=609405][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:02:32.721551] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=858844][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:31.947326] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=5482][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:31.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:32.721658] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=773158][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:31.947|207=774.33ms|210=0us|222=2us|total=774.36ms) [2025-08-14 09:02:32.721688] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4389] timer task use too much time(end=1755162152721687, start=1755162151947324, delta=774363) [2025-08-14 09:02:32.721561] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:32.032674] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:02:32.724046] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=690284] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:02:32.724095] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:02:32.724132] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:02:32.724148] INFO [COMMON] print_io_status (ob_io_manager.cpp:2639) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:2418, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:688640, total:688644) [2025-08-14 09:02:32.724178] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, param_config_:{memory_limit:536870912, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=865568, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:02:32.724212] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2826) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=30] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:2418, [delay/us]: prepare:0, schedule:0, submit:600000000, device:688640, total:688644) [2025-08-14 09:02:32.232416] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:31.893402] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c0f70, caller:mds_tenant_service.cpp:mtl_start:117, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_145, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571c1250, time_wheel:0x7f9959fef990, thread_pool:0x7f9959fef170, is_running:1, time_interval:15.00s, expected_run_time:09:02:31.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:32.724724] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=830063][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:31.893|207=831.32ms|210=0us|222=2us|total=831.34ms) [2025-08-14 09:02:32.724742] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] timer task use too much time(end=1755162152724742, start=1755162151893399, delta=831343) [2025-08-14 09:02:32.724751] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=960617, expected_run_ts_us_=1755162151764132, ObClockGenerator::getRealClock()=1755162152724750, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c0b90, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571c0e70, time_wheel:0x7f9959fef990, thread_pool:0x7f9959fef170, is_running:1, time_interval:3.00s, expected_run_time:09:02:31.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:32.724766] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c0b90, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571c0e70, time_wheel:0x7f9959fef990, thread_pool:0x7f9959fef170, is_running:1, time_interval:3.00s, expected_run_time:09:02:31.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:32.724230] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:02:31.764295] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:981) [337][EvtHisUpdTask][T1][YB42AC4D0007-00063C4960299C78-0-0] [lt=9221][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1755161909716139), 'FAILURE_DETECTOR', 'clog disk hang event', 'FAILURE_MODULE', 'LOG', 'FAILURE_TYPE', 'PROCESS HANG', 'AUTO_RECOVER', 'False', '', '', '', '172.77.0.7', 2882)"}, record_ret=-4012, ret=-4012) [2025-08-14 09:02:31.775558] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=594] temporary file module use no blocks [2025-08-14 09:02:32.727017] WDIAG print_alloc_failed_msg (alloc_failed_reason.cpp:146) [473][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=78863103][errcode=-4013] [OOPS]: alloc failed reason is that server memory has reached the upper limit(server_hold: 4294754304, server_limit: 4294967296, alloc_size: 2097152). detailed info: tenant_id=500, ctx_id=16, ctx_name=RPC_CTX_ID, ctx_hold=2818048, ctx_limit=9223372036854775807, tenant_hold=699760640, tenant_limit=9223372036854775807, backtrace=0x26a1806d 0xf98a0db 0xf989b06 0x9327c08 0x9315ac5 0x27aa2827 0x27a872f9 0xfcf0b15 0xfc721d0 0xfd2bbd0 0xf9d16a8 0xf9d4150 0xfa09ca2 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:32.727034] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [473][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_LOG_ELECTION_PREPARE_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-08-14 09:02:32.727043] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:57) [473][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-08-14 09:02:32.727053] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [473][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4013] LogRpc post_request failed(ret=-4013, palf_id_=1, req={this:0x7f994aa55e20, BASE:{msg_type:"Prepare Request", id:1, sender:"172.77.0.7:2882", receiver:"172.77.0.7:2882", restart_counter:1, ballot_number:67, debug_ts:{src_construct_ts:"01:10.739599", src_serialize_ts:"00:00.0", dest_deserialize_ts:"00:00.0", dest_process_ts:"00:00.0", process_delay:-1755162070739599}, biggest_min_cluster_version_ever_seen:4.3.5.2}, role:"Follower", is_buffer_valid:true, inner_priority_seed:4096, membership_version:{proposal_id:50, config_seq:47}}, server="172.77.0.7:2882") [2025-08-14 09:02:32.632650] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571634a0, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957163780, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:5.00s, expected_run_time:09:02:32.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:32.727150] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=94349][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:32.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:32.727166] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [473][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571f8030, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8310, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:0, time_interval:10.00s, expected_run_time:09:02:38.730, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:32.727178] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [473][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:02:32.727|total=81.99s) [2025-08-14 09:02:32.727189] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [473][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] task executed too delay(delay_time=81978120, *p_task_={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:31.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:32.727249] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [473][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=44][errcode=-4016] task executed too delay(delay_time=81968100, *p_task_={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:0, time_interval:500.00ms, expected_run_time:09:02:32.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:32.727380] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=2984350] gc stale ls task succ [2025-08-14 09:02:32.730667] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=4135] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:32.731176] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] get wrs ts(ls_id={id:1001}, delta=408002826, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:32.731190] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:31.775384] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:32.731505] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=954670][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:32.731521] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:32.731690] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:32.731703] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:32.731715] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15438, timeout ts=08:59:37.777, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:27.777], idx=3150, next_allow_print_ts="09:00:07.777") [2025-08-14 09:02:32.724094] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=309][errcode=0] task scheduled too delay(scheduled_delay=936005, expected_run_ts_us_=1755162151788089, ObClockGenerator::getRealClock()=1755162152724094, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728ca30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7291590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:31.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.733166] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9057][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728ca30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7291590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:31.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.733190] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] task scheduled too delay(scheduled_delay=937677, expected_run_ts_us_=1755162151795512, ObClockGenerator::getRealClock()=1755162152733189, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739ee10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:31.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.733204] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739ee10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:31.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.733217] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=928552, expected_run_ts_us_=1755162151804664, ObClockGenerator::getRealClock()=1755162152733217, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ebd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:31.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.733229] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ebd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:31.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.734876] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1641][errcode=0] task scheduled too delay(scheduled_delay=930162, expected_run_ts_us_=1755162151804712, ObClockGenerator::getRealClock()=1755162152734875, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739ec90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:31.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.734892] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739ec90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:31.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.730627] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [14905][T1001_TimerWK8][T1001][YB42AC4D0007-00063C4977399C32-0-0] [lt=20][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f9931bb2ab8, timer:0x7f99375fb6f0, task:0x7f99357f8518, task_type:"N9oceanbase7storage29ObTenantCheckpointSlogHandler21ObWriteCheckpointTaskE", scheduled_time:1755162099766270, pushed_time:1755162099766575, delay:60000000}, delay_in_thread_pool=50984935, threashold=500000) [2025-08-14 09:02:32.734905] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=918138, expected_run_ts_us_=1755162151816767, ObClockGenerator::getRealClock()=1755162152734905, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72874b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:31.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.734917] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72874b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:31.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.734929] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=912345, expected_run_ts_us_=1755162151822584, ObClockGenerator::getRealClock()=1755162152734929, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:31.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.734940] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:31.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.734952] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=912143, expected_run_ts_us_=1755162151822809, ObClockGenerator::getRealClock()=1755162152734952, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739dc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:31.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.734963] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739dc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:31.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.734976] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=839801, expected_run_ts_us_=1755162151895174, ObClockGenerator::getRealClock()=1755162152734976, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a59b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:31.895, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:31.993641] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=574][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:31.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:32.734987] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a59b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:31.895, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.734991] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=740228][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:31.993|207=741.35ms|210=0us|222=2us|total=741.36ms) [2025-08-14 09:02:32.735008] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] timer task use too much time(end=1755162152735007, start=1755162151993638, delta=741369) [2025-08-14 09:02:32.735011] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:32.726, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.735033] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739eb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:32.721, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.735032] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=19][errcode=0] task scheduled too delay(scheduled_delay=490398, expected_run_ts_us_=1755162152244633, ObClockGenerator::getRealClock()=1755162152735032, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:32.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:32.733104] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:32.735117] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EEB3-0-0] [lt=11][errcode=0] cost too much time:mds_tenant_service.cpp:int oceanbase::storage::mds::ObTenantMdsTimer::get_tablet_oldest_scn_(oceanbase::storage::ObTablet &, share::SCN &), (this=|threshold=5.00ms|start at 09:02:32.726|344=0us|total=8.47ms) [2025-08-14 09:02:32.724486] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=491325] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3922) [2025-08-14 09:02:32.735289] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10784][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:32.735321] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=18] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:32.735332] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=11] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3923) [2025-08-14 09:02:32.735339] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:32.735521] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:32.735532] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=11] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3924) [2025-08-14 09:02:32.735539] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:32.725349] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=1110] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:32.736427] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11057] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:32.717058] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [14910][TimerWK6_AutoLSLoc][T0][YB42AC4D0007-00063C4977899C57-0-0] [lt=10][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1002, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-08-14 09:02:32.736438] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SERVER] net_in=1kB/s, net_out=1kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=1280000kB/s [2025-08-14 09:02:32.736626] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:763) [337][EvtHisUpdTask][T0][YB42AC4D0007-00063C4960299C78-0-0] [lt=960934] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1755161909716139), 'FAILURE_DETECTOR', 'clog disk hang event', 'FAILURE_MODULE', 'LOG', 'FAILURE_TYPE', 'PROCESS HANG', 'AUTO_RECOVER', 'False', '', '', '', '172.77.0.7', 2882)"}, process_time=123964117) [2025-08-14 09:02:32.736645] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1018) [337][EvtHisUpdTask][T0][YB42AC4D0007-00063C4960299C78-0-0] [lt=18][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1755161909716139), 'FAILURE_DETECTOR', 'clog disk hang event', 'FAILURE_MODULE', 'LOG', 'FAILURE_TYPE', 'PROCESS HANG', 'AUTO_RECOVER', 'False', '', '', '', '172.77.0.7', 2882)"}, aret=-4012, ret=-4012) [2025-08-14 09:02:32.731207] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [WRS] update tenant weak read server version(tenant_id=1002, server_version={version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162152726503}, version_delta=394509408) [2025-08-14 09:02:32.736771] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=5541] get_max_applied_scn(scn={val:1755161749741193000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:50, palf_committed_end_lsn_:{lsn:1604090090}, palf_committed_end_scn_:{val:1755161749741193000, v:0}, last_check_scn_:{val:1755161749741193001, v:0}, max_applied_cb_scn_:{val:1755161749741193000, v:0}}) [2025-08-14 09:02:32.736794] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get_min_unreplayed_log_info(lsn={lsn:1604088821}, scn={val:1755100948296325001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873479, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:12492179, ret_code_:0}}, next_to_submit_lsn_:{lsn:1604088821}, next_to_submit_scn_:{val:1755100948296325001, v:0}, base_lsn_:{lsn:1409200128}, base_scn_:{val:1753840802540914000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:1604088821}, end_lsn:{lsn:1604088821}, read_buf:{buf_len_:3747840, buf_:0x7f98f6a05000}, block_size:67104768, log_storage_:0x7f990bfebbf0, log_storage_:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:42, read_io_size:147709952, read_disk_cost_ts:1061450, start_lsn:{lsn:1409200128}}}}}}) [2025-08-14 09:02:32.736874] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=78] get wrs ts(ls_id={id:1}, delta=402993038, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:32.736889] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:32.737063] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:32.726739] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=949743] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:32.737122] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10364] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:32.737134] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623053214591, flush_io_finished_ret_:0, flush_io_finished_round_:26324, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:32.737464] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=408006407, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:32.737483] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 08:59:28.790721] EDIAG [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [15438][T1_L0_G1][T1][YB42AC4D0007-00063C496CE99C4F-0-0] [lt=19][errcode=0] LOGGER COST TOO MUCH TIME, cost: 183669373, owner: ObLog, click_count: 7, time dist:[FORMAT_END=16, ObMalloc#start=0, ObMalloc#end=8781, ObMalloc#start=183660569, ObMalloc#end=2, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0xfbf79e6 0xfbf6a36 0xfb36e2f 0x126942ee 0x984d055 0x9280839 0x926c154 0x12c5a0c9 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 08:59:28.790721] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [15438][T1_L0_G1][T1][YB42AC4D0007-00063C496CE99C4F-0-0] [lt=19] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.77.0.7:2882", child={server:"172.77.0.7:2882", region:"deault_region", register_time_us:1755161965774369, keepalive_ts:-1}, is_to_leader=true) [2025-08-14 09:02:32.738549] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12396] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:32.740545] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] EASY SLOW: thread: 7f99542fb300, lastrun: 1755162115.752503 cost: 36.988039 loop:26416, slowcnt: 312 [2025-08-14 09:02:32.741419] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:32.741467] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=42] temporary file module use no blocks [2025-08-14 09:02:32.741477] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:32.741496] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=20] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:32.741509] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623054181140, flush_io_finished_ret_:0, flush_io_finished_round_:26312, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:32.741885] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:32.742434] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:32.742496] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [14905][T1001_TimerWK8_PalfGC][T1001][Y0-0000000000000000-0-0] [lt=4279] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2764, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2764, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):325, sequence:1}) [2025-08-14 09:02:32.742536] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:97) [14905][T1001_TimerWK8_PalfGC][T1001][Y0-0000000000000000-0-0] [lt=28] BlockGCTimerTask success(ret=0, cost_time_us=48, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.77.0.7:2882", log_dir:"/root/ob/observer/store/clog/tenant_1001", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2764, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2764, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):325, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:1}}) [2025-08-14 09:02:32.742562] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [14905][T1001_TimerWK8][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f9931bb2818, timer:0x7f993410def0, task:0x7f99321fb398, task_type:"N9oceanbase4palf16BlockGCTimerTaskE", scheduled_time:1755162099771228, pushed_time:1755162099776675, delay:30000}, delay_in_thread_pool=52958248, threashold=500000) [2025-08-14 09:02:32.742593] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [14905][T1001_TimerWK8][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] timer task too much delay in priority queue(token={this:0x7f9931bb2658, timer:0x7f9934119ee0, task:0x7f9934405038, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr12TabletGCTaskE", scheduled_time:1755161981770099, pushed_time:1755162136772529, delay:20000}, delay_in_priority_queue=155002430, threashold=1000000) [2025-08-14 09:02:32.742607] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [14905][T1001_TimerWK8][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f9931bb2658, timer:0x7f9934119ee0, task:0x7f9934405038, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr12TabletGCTaskE", scheduled_time:1755161981770099, pushed_time:1755162136772529, delay:20000}, delay_in_thread_pool=15970058, threashold=500000) [2025-08-14 09:02:32.746556] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:32.746600] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:32.746905] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:32.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.746966] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5598] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:32.747095] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=11] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:32.750256] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:32.750458] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:32.751016] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=20] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:32.755698] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:32.757288] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:32.759771] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6234] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:32.738583] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=34] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:32.759960] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=21342] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:32.760318] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:32.750, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.760351] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:32.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.760364] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d80b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:32.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.760377] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:32.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.760390] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891992330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196d710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:32.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.760404] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:32.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.760596] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=14] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:32.759973] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:32.760711] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=735] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:32.760724] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:32.760734] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:32.760743] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:32.760752] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:32.741255] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EEB3-0-0] [lt=15] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-08-14 09:02:32.762690] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:32.763508] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:32.764558] WDIAG [PALF] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [15438][T1_L0_G1][T1][YB42AC4D0007-00063C496CE99C4F-0-0] [lt=183669373][errcode=0] cost too much time:log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false], (this=|threshold=50.00ms|start at 09:02:32.764|total=184.99s) [2025-08-14 09:02:32.764602] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1590) [337][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1755161909716139), 'FAILURE_DETECTOR', 'clog disk hang event', 'FAILURE_MODULE', 'LOG', 'FAILURE_TYPE', 'PROCESS HANG', 'AUTO_RECOVER', 'False', '', '', '', '172.77.0.7', 2882)) [2025-08-14 09:02:32.766045] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=14] gc stale ls task succ [2025-08-14 09:02:32.770507] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:32.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.770547] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:32.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.770621] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=209] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:32.768042] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:02:32.771396] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=3345] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:02:32.771417] INFO [COMMON] loop_running_dag_net_list (ob_tenant_dag_scheduler.cpp:3845) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=16] loop running dag net list(ret=0, running_dag_net_list_size=0, slow_dag_net_cnt=0, finished_dag_net_list_size=0) [2025-08-14 09:02:32.771429] INFO [COMMON] loop_finished_dag_net_list (ob_tenant_dag_scheduler.cpp:3868) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] loop finsihed dag net list(ret=0, finished_dag_net_list_size=0) [2025-08-14 09:02:32.764745] WDIAG [SQL] open_plan (ob_result_set.cpp:104) [1003][BlackListServic][T1][YB42AC4D0007-00063C495F09BD69-0-0] [lt=962122][errcode=-4012] query is timeout(ret=-4012, timeout_ts=1755162140765205, start_time=1755162139738333) [2025-08-14 09:02:32.773387] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=161] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:02:32.776434] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:32.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:32.779245] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:32.781311] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:32.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.781344] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72751b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:32.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.781361] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d31b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:32.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.781376] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73959b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:32.773, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.791046] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=928607] gc stale ls task succ [2025-08-14 09:02:32.791371] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:32.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.791399] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72856b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:32.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.791413] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:32.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.801438] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919920b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196d590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:32.790, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:32.806611] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:32.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:32.817408] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:32.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:32.821625] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=295] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:02:32.821665] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=39] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:02:32.821681] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:32.823520] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=904506] PNIO [ratelimit] time: 1755162152823517, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007606, add_bytes: 0 [658]OB_MALLOC COST TOO MUCH TIME, cost_time=898398, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=898388], seq:[0] [471]OB_MALLOC COST TOO MUCH TIME, cost_time=897311, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=32896, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=897304], seq:[0] [2025-08-14 09:02:33.727445] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=6] gc stale ls task succ [567]OB_MALLOC COST TOO MUCH TIME, cost_time=906644, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:02:33.733499] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=298] get_max_applied_scn(scn={val:1755161547738516001, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:50, palf_committed_end_lsn_:{lsn:5282737655}, palf_committed_end_scn_:{val:1755161547738516001, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161547738516001, v:0}}) [2025-08-14 09:02:33.733550] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=43] get_min_unreplayed_log_info(lsn={lsn:5282737655}, scn={val:1755100947825033001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161624848121, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:13063726, ret_code_:0}}, next_to_submit_lsn_:{lsn:5282737655}, next_to_submit_scn_:{val:1755100947825033001, v:0}, base_lsn_:{lsn:4831543296}, base_scn_:{val:1755063445733337001, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7605000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:5282737655}, end_lsn:{lsn:5282737655}, read_buf:{buf_len_:3747840, buf_:0x7f98f7605000}, block_size:67104768, log_storage_:0x7f993e529bf0, log_storage_:{log_tail:{lsn:5282737655}, readable_log_tail:{lsn:5282737655}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:5234171904}, min_scn:{val:1755152244802535004, v:0}, curr_block_id:78, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:206, min_block_id:42, max_block_id:79, curr_writable_block_id:78}, logical_block_size_:67104768, curr_block_writable_size_:18539017, block_header_serialize_buf_:0x7f993e52a8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:85, read_io_size:301367296, read_disk_cost_ts:2364604, start_lsn:{lsn:4831543296}}}}}}) [2025-08-14 09:02:33.734609] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=84] get wrs ts(ls_id={id:1}, delta=605995923, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:33.734705] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=76] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [857]OB_MALLOC COST TOO MUCH TIME, cost_time=896613, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:02:33.734862] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=403993246, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:33.734883] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:33.735604] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=31] get wrs ts(ls_id={id:1001}, delta=409006615, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:33.735622] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:33.735656] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119812272, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-08-14 09:02:33.735682] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=403993246, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:33.735695] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:33.737308] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:33.737346] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=21] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:33.737365] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] temporary file module use no blocks [2025-08-14 09:02:33.737375] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:33.737397] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=20] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:33.737412] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623055176925, flush_io_finished_ret_:0, flush_io_finished_round_:26325, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:33.741596] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:33.744981] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:33.745302] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:33.746496] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:33.746535] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=46] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:33.747210] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:33.750671] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] EASY SLOW: thread: 7f99542fb300, lastrun: 1755162115.752503 cost: 37.998164 loop:26416, slowcnt: 322 [2025-08-14 09:02:33.750742] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=8929][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fb3b0, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fb690, time_wheel:0x7f99589fd990, thread_pool:0x7f9959fff170, is_running:1, time_interval:2.00s, expected_run_time:09:02:33.698, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:33.752525] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=18] swc wakeup.(stat_period_=1000000, ready=false) [729]OB_MALLOC COST TOO MUCH TIME, cost_time=927725, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:02:33.754332] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:33.754580] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:33.756058] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=31] get wrs ts(ls_id={id:1001}, delta=409027151, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:33.756087] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:33.756332] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B927E-0-0] [lt=33][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162152825394) [2025-08-14 09:02:33.756362] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B927E-0-0] [lt=28][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162152825394}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162153733478}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162153733478}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [822]OB_MALLOC COST TOO MUCH TIME, cost_time=930951, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=31, WASH_BLOCK_END=930917, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1], seq:[0, 1, 2, 3, 4] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=833808, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=33, WASH_BLOCK_END=833775], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=833652, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=833612], seq:[0, 1, 2] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=727792, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=33, WASH_BLOCK_END=727758], seq:[0, 1, 2] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=719148, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=33, WASH_BLOCK_END=719114], seq:[0, 1, 2] [768]OB_MALLOC COST TOO MUCH TIME, cost_time=123704, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=44, WASH_BLOCK_END=123660], seq:[0, 1, 2] [2025-08-14 09:02:32.824318] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=866118] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:02:32.921877] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:32.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:33.756846] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=833704][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:32.921|207=834.97ms|210=0us|222=2us|total=834.99ms) [2025-08-14 09:02:33.756872] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4389] timer task use too much time(end=1755162153756871, start=1755162152921875, delta=834996) [2025-08-14 09:02:33.027980] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:33.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:33.756944] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=727860][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:33.279|207=728.96ms|210=1us|222=1us|total=728.98ms) [2025-08-14 09:02:33.756957] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task use too much time(end=1755162153756957, start=1755162153027978, delta=728979) [2025-08-14 09:02:33.756972] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=464621, expected_run_ts_us_=1755162153292349, ObClockGenerator::getRealClock()=1755162153756971, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:33.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:33.756990] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:33.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:33.036783] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:02:33.757326] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=930996][errcode=0] get_fetch_log_dst failed or invalid(ret=0, palf_id=1, self="172.77.0.7:2882", fetch_log_dst="172.77.0.7:2882") [2025-08-14 09:02:33.757346] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=18] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:02:33.757435] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:33.733098] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1986] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:33.734758] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=38] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=605996239, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=253108, succ_cluster_heartbeat_count=2201, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161000606550000, v:0}, local_cluster_delta=1153128205, force_self_check=true, weak_read_refresh_interval=100000) [2025-08-14 09:02:33.759042] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23683] get wrs ts(ls_id={id:1}, delta=606020214, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:33.759061] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:33.759655] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=606020214, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:33.759666] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [658]OB_MALLOC COST TOO MUCH TIME, cost_time=935861, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=898434], seq:[0, 1] [2025-08-14 09:02:32.823125] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=651][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:32.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:33.760447] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=935903][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:32.823|207=937.32ms|210=1us|222=1us|total=937.35ms) [2025-08-14 09:02:33.760479] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=-4389] timer task use too much time(end=1755162153760479, start=1755162152823123, delta=937356) [770]OB_MALLOC COST TOO MUCH TIME, cost_time=841765, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=34, WASH_BLOCK_END=841729], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=734389, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=35, WASH_BLOCK_END=734354], seq:[0, 1, 2] [381]OB_MALLOC COST TOO MUCH TIME, cost_time=539322, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=54, WASH_BLOCK_END=539268], seq:[0, 1, 2] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=528717, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=34, WASH_BLOCK_END=528683], seq:[0, 1, 2] [1031]OB_MALLOC COST TOO MUCH TIME, cost_time=525068, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=44, WASH_BLOCK_END=525024], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=243837, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=46, WASH_BLOCK_END=243790], seq:[0, 1, 2] [998]OB_MALLOC COST TOO MUCH TIME, cost_time=987026, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=987025], seq:[0] [998]OB_MALLOC COST TOO MUCH TIME, cost_time=991547, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=987038], seq:[0, 1] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=915202, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=34, WASH_BLOCK_END=907349, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=7818], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:33.771120] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:33.760343] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=10] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:03.746093] EDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:720) [999][T1002_L0_G0][T1002][YB42AC4D0007-00063C49651A3232-0-0] [lt=10044][errcode=0] LOGGER COST TOO MUCH TIME, cost: 12986811, owner: ObLog, click_count: 6, time dist:[FORMAT_END=12982754, ObMalloc#start=1, ObMalloc#end=2198, ObMalloc#start=27, ObMalloc#end=1707, FREE_END=124], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0xfd33b2e 0xfd33236 0xfd32f2b 0xfd32cd8 0x148db297 0x933f0ce 0x93363f3 0x932e8ab 0x92d9ed1 0x9828a7d 0x9298239 0x9285f13 0x927efad 0x926c154 0x12c5a0c9 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:33.772849] WDIAG ev_invoke_pending (ev.c:2136) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=12485][errcode=0] epoll handler too much time :12510 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:32.786305] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [726][T1001_ObXAHbWor][T1001][Y0-0000000000000000-0-0] [lt=12342] XA scheduler heartbeat task statistics(avg_time=1) [2025-08-14 09:02:33.721247] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=237] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:33.757114] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=719189] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:02:33.773388] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=52061][errcode=0] epoll handler too much time :52161 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:33.747074] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=15] current monitor number(seq_=-1) [2025-08-14 09:02:33.773424] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16249] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:02:33.773473] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:02:33.773491] INFO [COMMON] print_io_status (ob_io_manager.cpp:2639) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:3903, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:11907, total:11911) [2025-08-14 09:02:33.773496] WDIAG [SQL] start_stmt (ob_result_set.cpp:348) [999][T1002_L0_G0][T1002][YB42AC4D0007-00063C49651A3232-0-0] [lt=12986811][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:500395, schema_version:1753088287663064, object_type:1, is_db_explicit:false, is_existed:true}]) [2025-08-14 09:02:33.773513] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, param_config_:{memory_limit:536870912, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=865568, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:02:33.773558] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=14] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:02:32.764637] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [337][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] long time to hold one guard(schema_mgr=0x7f98ad7818f0, tenant_id=1, version=1753073979411448, cur_timestamp=1755162152764626, ref_timestamp=1755162049733347, lbt()="0x26a1806d 0x974f3e5 0x974e159 0x974e084 0x948f9df 0x97257c0 0x124e2fb7 0x97aa07c 0x124eac91 0x97a9944 0x261395f9 0x1f900517 0x1f8fe2bf 0x9a5f486 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3") [2025-08-14 09:02:33.773530] WDIAG [SQL] do_open_plan (ob_result_set.cpp:575) [999][T1002_L0_G0][T1002][YB42AC4D0007-00063C49651A3232-0-0] [lt=34][errcode=-4283] fail start stmt(ret=-4283) [2025-08-14 09:02:32.921677] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:33.776489] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=853517][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:33.773554] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2826) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=36] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:3903, [delay/us]: prepare:0, schedule:0, submit:600000000, device:11907, total:11911) [2025-08-14 09:02:33.776598] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3026] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:02:33.776613] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:33.776625] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:33.776636] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SERVER] net_in=0kB/s, net_out=0kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=1280000kB/s [2025-08-14 09:02:33.777852] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:33.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:33.772935] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:140) [14905][T1001_TimerWK8_CompactionRefre][T1001][YB42AC4D0007-00063C4977399C33-0-0] [lt=10] InfoPoolResizeTask(cost_ts=1018537) [2025-08-14 09:02:33.778661] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [14905][T1001_TimerWK8][T1001][YB42AC4D0007-00063C4977399C33-0-0] [lt=5716][errcode=-4389] timer task too much delay in priority queue(token={this:0x7f9931bb3618, timer:0x7f9966a59ba0, task:0x7f99333bfa58, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr18InfoPoolResizeTaskE", scheduled_time:1755162073718902, pushed_time:1755162137767272, delay:30000000}, delay_in_priority_queue=64048370, threashold=1000000) [2025-08-14 09:02:33.778692] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [14905][T1001_TimerWK8][T1001][YB42AC4D0007-00063C4977399C33-0-0] [lt=30][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f9931bb3618, timer:0x7f9966a59ba0, task:0x7f99333bfa58, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr18InfoPoolResizeTaskE", scheduled_time:1755162073718902, pushed_time:1755162137767272, delay:30000000}, delay_in_thread_pool=14975347, threashold=500000) [2025-08-14 09:02:33.779467] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=14] log compressor cycles once. (ret=0, cost_time=14379, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25937367040, fast_delete_log_mode=false) [2025-08-14 09:02:33.774599] WDIAG [SQL] open (ob_result_set.cpp:137) [999][T1002_L0_G0][T1002][YB42AC4D0007-00063C49651A3232-0-0] [lt=1064][errcode=-4283] execute plan failed(ret=-4283) [2025-08-14 09:02:33.722850] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] tx gc loop thread is running(MTL_ID()=1) [2025-08-14 09:02:33.780062] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=57123] try gc retain ctx [2025-08-14 09:02:33.780073] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=11] try to retry start_working [2025-08-14 09:02:33.750851] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:33.780640] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=29741][errcode=0] epoll handler too much time :29795 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:33.633279] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:33.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:33.780710] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=147204][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:33.633|207=147.43ms|210=0us|222=2us|total=147.45ms) [2025-08-14 09:02:33.780735] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4389] timer task use too much time(end=1755162153780735, start=1755162153633276, delta=147459) [2025-08-14 09:02:33.780785] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=59685] temporary file module use no blocks [2025-08-14 09:02:33.757450] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [316][pnio1][T1][YB42AC4D0007-00063C49607B927E-0-0] [lt=13][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1, delta=2422943, last_succ_cluster_heartbeat_tstamp_=1755162151334506) [2025-08-14 09:02:33.758502] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [979][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F6A-0-0] [lt=47][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1002, svr="172.77.0.7:2882", version={val:1755161758221798000, v:0}, valid_part_count=0, total_part_count=2, generate_timestamp=1755162152836832) [2025-08-14 09:02:33.781153] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [979][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F6A-0-0] [lt=22610][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1002, req={req_server:"172.77.0.7:2882", version:{val:1755161758221798000, v:0}, valid_part_count:0, total_part_count:2, generate_timestamp:1755162152836832}, twrs={inited:true, tenant_id:1002, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162153735674}, server_version_for_stat:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162153735674}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:33.781284] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [317][pnio1][T1002][YB42AC4D0007-00063C49612B8F6A-0-0] [lt=14][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1002, delta=2056252, last_succ_cluster_heartbeat_tstamp_=1755162151725031) [2025-08-14 09:02:33.781341] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:33.782072] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=404032981, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:33.782090] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:33.782600] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1001}, delta=409046350, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:33.782614] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:33.791119] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=21] gc stale ls task succ [2025-08-14 09:02:33.808045] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:33.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:33.831136] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=16] PNIO [ratelimit] time: 1755162153831133, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007616, add_bytes: 0 [2025-08-14 09:02:33.857370] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:02:33.859809] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=606120726, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:33.859837] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:33.876546] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:33.882152] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=404140191, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:33.882175] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:33.956984] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:33.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:33.976605] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:34.029293] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:34.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:34.057045] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:33.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:34.076642] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:34.076965] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:02:34.077013] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:02:34.157109] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:34.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:34.178253] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:34.257170] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=51][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:34.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:34.273409] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:34.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:34.278313] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:34.303604] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:34.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:34.357243] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:34.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:34.378353] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:34.457306] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:34.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:34.478430] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:34.524984] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:34.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:34.557377] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=50][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:34.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:34.578441] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:34.657437] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:34.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:34.678509] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:34.681243] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:34.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:34.684689] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:1, time_interval:3.00s, expected_run_time:09:02:34.554, task_priority:1, with_handle_protected:1}) [421]OB_MALLOC COST TOO MUCH TIME, cost_time=932017, tenant_id=500, label=IoControl, ctx_id=0, prio=0, size=65280, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=3], seq:[0, 1] [2025-08-14 09:02:33.780797] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 08:59:58.783186] EDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:720) [469][T1_Occam][T1][YB42AC4D0007-00063C495C5A1329-0-0] [lt=10356][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2948385, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=2, ALLOC_BLOCK#end=82, FORMAT_END=2948300, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0xfd33b2e 0xfd33236 0xfd32f2b 0xfd32cd8 0x148db297 0x933f0ce 0x93363f3 0x932e8ab 0x96ac524 0x96a672d 0x96a5033 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x9c34403 0x9c31cbc 0x122def07 0x123237cf 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [374]OB_MALLOC COST TOO MUCH TIME, cost_time=950021, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=32, WASH_BLOCK_END=949988], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=957864, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=7868, WASH_BLOCK_END=949996], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=957844, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=7863, WASH_BLOCK_END=949981], seq:[0, 1, 2] [381]OB_MALLOC COST TOO MUCH TIME, cost_time=957813, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=7840, WASH_BLOCK_END=949973], seq:[0, 1, 2] [771]OB_MALLOC COST TOO MUCH TIME, cost_time=957815, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=7843, WASH_BLOCK_END=949972], seq:[0, 1, 2] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=957054, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=7064, WASH_BLOCK_END=949990], seq:[0, 1, 2] [1031]OB_MALLOC COST TOO MUCH TIME, cost_time=957017, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=7034, WASH_BLOCK_END=949983], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=957000, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=7033, WASH_BLOCK_END=949967], seq:[0, 1, 2] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=956575, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=6616, WASH_BLOCK_END=949959], seq:[0, 1, 2] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=956359, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=6402, WASH_BLOCK_END=949957], seq:[0, 1, 2] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=954388, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=4392, WASH_BLOCK_END=949996], seq:[0, 1, 2] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=954398, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=4391, WASH_BLOCK_END=950007], seq:[0, 1, 2] [729]OB_MALLOC COST TOO MUCH TIME, cost_time=953161, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=3122, WASH_BLOCK_END=950039], seq:[0, 1, 2] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=945320, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=57, WASH_BLOCK_END=945262], seq:[0, 1, 2] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=945311, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=62, WASH_BLOCK_END=945249], seq:[0, 1, 2] [2025-08-14 09:02:33.220459] EDIAG [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1497675, owner: ObLog, click_count: 6, time dist:[FORMAT_END=21, ObMalloc#start=2, ObMalloc#end=539330, ObMalloc#start=28, ObMalloc#end=957820, FREE_END=474], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b537806 0x1b4f816d 0x1b4fbf53 0x1b5014e0 0x1b5009d1 0x1b50387c 0x1b8905de 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:33.235629] EDIAG [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1482642, owner: ObLog, click_count: 6, time dist:[FORMAT_END=22, ObMalloc#start=1, ObMalloc#end=525077, ObMalloc#start=18, ObMalloc#end=957024, FREE_END=500], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x18257d0c 0x1823b885 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:33.025069] EDIAG try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=773315][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1692943, owner: ObLog, click_count: 6, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=734397, ObMalloc#start=18, ObMalloc#end=957851, FREE_END=664], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x26499505 0x97074a3 0x26498fa9 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:34.077058] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:02:34.719050] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=640983] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:02:34.719116] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=35] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:02:34.719127] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=1, sync_total=0, async_cnt= 1 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:34.719138] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:34.719146] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SERVER] net_in=0kB/s, net_out=0kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=1280000kB/s [2025-08-14 09:02:34.721381] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [567]OB_MALLOC COST TOO MUCH TIME, cost_time=764795, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:02:33.774616] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb2658, timer:0x7f9934119ee0, task:0x7f9934405038, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr12TabletGCTaskE", scheduled_time:1755162152762618, pushed_time:0, delay:20000}, running_task_token={this:0x7f9931bb2498, timer:0x7f9934119ee0, task:0x7f9934405010, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr11TableGCTaskE", scheduled_time:1755162043738943, pushed_time:1755162152742822, delay:20000}, total_delay=1011997, threashold=1000000, tenant_id=1001) [2025-08-14 09:02:33.783793] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=21] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:02:33.762254] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:34.726950] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=963238][errcode=0] epoll handler too much time :964704 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:34.727507] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=28] gc stale ls task succ [737]OB_MALLOC COST TOO MUCH TIME, cost_time=1003230, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=43707, WASH_BLOCK_END=952235, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=7287], seq:[0, 1, 2, 3, 4] [1003]OB_MALLOC COST TOO MUCH TIME, cost_time=952661, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=52, WASH_BLOCK_END=945337, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=7271], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:33.755827] WDIAG [SQL] open (ob_result_set.cpp:137) [1003][BlackListServic][T1][YB42AC4D0007-00063C495F09BD69-0-0] [lt=8432][errcode=-4012] execute plan failed(ret=-4012) [2025-08-14 09:02:34.725737] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=606985546, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:34.727909] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2163] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:34.728494] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27] get wrs ts(ls_id={id:1}, delta=606988730, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:34.728511] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:34.727590] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [1003][BlackListServic][T1][YB42AC4D0007-00063C495F09BD69-0-0] [lt=970291][errcode=-4012] open result set failed(ret=-4012) [2025-08-14 09:02:34.730646] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:801) [1003][BlackListServic][T1][YB42AC4D0007-00063C495F09BD69-0-0] [lt=3046][errcode=-4012] result set open failed(ret=-4012, executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}) [2025-08-14 09:02:33.094259] EDIAG [ELECT] operator() (election_proposer.cpp:232) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=3950][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1634128, owner: ObLog, click_count: 7, time dist:[FORMAT_END=665414, ObMalloc#start=0, ObMalloc#end=957822, ObMalloc#start=27, ObMalloc#end=10858, ALLOC_END=3, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xfa07f77 0xfa07a14 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [387]OB_MALLOC COST TOO MUCH TIME, cost_time=969251, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=7893, WASH_BLOCK_END=950116, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=7072], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:34.719456] WDIAG [SQL] start_stmt (ob_result_set.cpp:348) [469][T1_Occam][T1][YB42AC4D0007-00063C495C5A1329-0-0] [lt=2948385][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:117, schema_version:1753069637394160, object_type:1, is_db_explicit:false, is_existed:true}]) [2025-08-14 09:02:33.764291] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=16] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:34.730768] WDIAG [SQL] do_open_plan (ob_result_set.cpp:575) [469][T1_Occam][T1][YB42AC4D0007-00063C495C5A1329-0-0] [lt=11293][errcode=-4283] fail start stmt(ret=-4283) [2025-08-14 09:02:34.730800] WDIAG [SQL] open (ob_result_set.cpp:137) [469][T1_Occam][T1][YB42AC4D0007-00063C495C5A1329-0-0] [lt=32][errcode=-4283] execute plan failed(ret=-4283) [2025-08-14 09:02:34.730819] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [469][T1_Occam][T1][YB42AC4D0007-00063C495C5A1329-0-0] [lt=18][errcode=-4283] open result set failed(ret=-4283) [2025-08-14 09:02:34.730829] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:801) [469][T1_Occam][T1][YB42AC4D0007-00063C495C5A1329-0-0] [lt=8][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='172.77.0.7' and svr_port=2882"}) [2025-08-14 09:02:33.757551] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24415][errcode=0] epoll handler too much time :24462 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:34.730897] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1634128][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:0, time_interval:500.00ms, expected_run_time:09:02:33.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:34.730934] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:02:34.730|total=1.64s) [2025-08-14 09:02:34.730952] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] task executed too delay(delay_time=1516560, *p_task_={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:0, time_interval:500.00ms, expected_run_time:09:02:33.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:34.730913] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=963836] get_min_unreplayed_log_info(lsn={lsn:8461098908}, scn={val:1755100947600130001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161920778323, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:23084275, ret_code_:0}}, next_to_submit_lsn_:{lsn:8461098908}, next_to_submit_scn_:{val:1755100947600130001, v:0}, base_lsn_:{lsn:7515734016}, base_scn_:{val:1753840867171901045, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6e05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:8461098908}, end_lsn:{lsn:8461098908}, read_buf:{buf_len_:3747840, buf_:0x7f98f6e05000}, block_size:67104768, log_storage_:0x7f99297a3bf0, log_storage_:{log_tail:{lsn:8461098908}, readable_log_tail:{lsn:8461098908}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8455200768}, min_scn:{val:1755160384851871002, v:0}, curr_block_id:126, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:215, min_block_id:93, max_block_id:127, curr_writable_block_id:126}, logical_block_size_:67104768, curr_block_writable_size_:61206628, block_header_serialize_buf_:0x7f99297a48b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:198, read_io_size:699822080, read_disk_cost_ts:4178035, start_lsn:{lsn:7515734016}}}}}}) [2025-08-14 09:02:34.731009] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4016] task executed too delay(delay_time=1466307, *p_task_={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:33.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:34.731025] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] task executed too delay(delay_time=1466317, *p_task_={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:0, time_interval:250.00ms, expected_run_time:09:02:33.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:33.746563] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:34.731503] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=81] get wrs ts(ls_id={id:1}, delta=434954509, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:34.731520] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:34.715805] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=933593] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:34.732142] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16312] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623055160703, flush_io_finished_ret_:0, flush_io_finished_round_:26293, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:33.722790] EDIAG [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1008102, owner: ObLog, click_count: 6, time dist:[FORMAT_END=14, ObMalloc#start=1, ObMalloc#end=1003253, ObMalloc#start=26, ObMalloc#end=4745, FREE_END=63], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b51d070 0x986d718 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:34.732453] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=1008102] try gc retain ctx [2025-08-14 09:02:34.732476] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=23] try to retry start_working [2025-08-14 09:02:34.720423] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=1497675][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162153220444], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:34.732519] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12074][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162153220444], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:34.732541] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162153220444]}) [2025-08-14 09:02:34.720547] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=1482642] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3925) [2025-08-14 09:02:34.732568] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=12001][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:34.732598] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:34.732609] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=11] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3926) [2025-08-14 09:02:34.732618] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:34.730607] INFO [STORAGE] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:123) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] wead read timestamp is too old(timestamp={val:1755161749741193000, v:0}, gts_scn={val:1755161762721798002, v:0}, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1755161749741193000, v:0}}) [2025-08-14 09:02:33.231893] EDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] LOGGER COST TOO MUCH TIME, cost: 1501160, owner: ObLog, click_count: 8, time dist:[FORMAT_END=19, ObMalloc#start=1, ObMalloc#end=528734, ObMalloc#start=39, ObMalloc#end=957061, ObMalloc#start=601, ObMalloc#end=14545, FREE_END=160], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x13b06ffb 0x958a39e 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:33.860559] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:33.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:34.735275] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=873390][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:33.860|207=874.72ms|210=0us|222=2us|total=874.74ms) [2025-08-14 09:02:34.735298] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4389] timer task use too much time(end=1755162154735297, start=1755162153860556, delta=874741) [2025-08-14 09:02:34.735353] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=1501160][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:02:34.735377] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:34.735389] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:34.735408] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:34.735420] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:32.917445] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1816226, owner: ObLog, click_count: 8, time dist:[FORMAT_END=47, ObMalloc#start=0, ObMalloc#end=841777, ObMalloc#start=24, ObMalloc#end=957873, ObMalloc#start=719, ObMalloc#end=15720, FREE_END=65], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:34.736448] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1816226][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:32.917|207=1.82s|210=0us|222=2us|total=1.82s) [2025-08-14 09:02:34.736482] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4389] timer task use too much time(end=1755162154736481, start=1755162152917443, delta=1819038) [2025-08-14 09:02:34.736881] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=4586] get wrs ts(ls_id={id:1001}, delta=410003852, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:34.736903] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:34.735439] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:34.737043] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=1600][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:01:49.731, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:01:42.751], idx=659, next_allow_print_ts="09:02:19.731") [2025-08-14 09:02:34.737083] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:34.737099] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:34.737112] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:34.737124] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:34.737138] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:34.737341] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:34.737356] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:34.737664] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1076) [999][T1002_L0_G0][T1002][YB42AC4D0007-00063C49651A3232-0-0] [lt=5267][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:false, stmt_retry_times:1, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:2, client_ret:-4283}, need_retry=true) [2025-08-14 09:02:34.737694] WDIAG [SERVER] response_result (ob_sync_plan_driver.cpp:71) [999][T1002_L0_G0][T1002][YB42AC4D0007-00063C49651A3232-0-0] [lt=27][errcode=-4283] result set open failed, check if need retry(ret=-4283, cli_ret=-4283, retry_ctrl_.need_retry()=1) [2025-08-14 09:02:34.737719] WDIAG [SERVER] response_result (ob_sync_plan_driver.cpp:81) [999][T1002_L0_G0][T1002][YB42AC4D0007-00063C49651A3232-0-0] [lt=13][errcode=-4283] close result set fail(cret=-4283) [2025-08-14 09:02:32.851735] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=39][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1883642, owner: ObLog, click_count: 8, time dist:[FORMAT_END=23, ObMalloc#start=1, ObMalloc#end=915219, ObMalloc#start=22, ObMalloc#end=950044, ObMalloc#start=769, ObMalloc#end=17399, FREE_END=165], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:34.738262] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1883642][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:32.851|207=1.89s|210=0us|222=3us|total=1.89s) [2025-08-14 09:02:34.738283] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4389] timer task use too much time(end=1755162154738282, start=1755162152851733, delta=1886549) [2025-08-14 09:02:34.738293] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1892168, expected_run_ts_us_=1755162152846124, ObClockGenerator::getRealClock()=1755162154738293, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d54b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ecd90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:32.846, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.738311] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d54b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ecd90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:32.846, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.738324] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1890270, expected_run_ts_us_=1755162152848053, ObClockGenerator::getRealClock()=1755162154738324, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ecc10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:32.848, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.738336] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ecc10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:32.848, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.738361] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=1016184, expected_run_ts_us_=1755162153722177, ObClockGenerator::getRealClock()=1755162154738361, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.722, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.738374] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.722, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.738387] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1007505, expected_run_ts_us_=1755162153730881, ObClockGenerator::getRealClock()=1755162154738387, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.738411] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.738498] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:946) [1003][BlackListServic][T1][YB42AC4D0007-00063C495F09BD69-0-0] [lt=56][errcode=-4012] execute failed(ret=-4012, tenant_id=1, executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, retry_cnt=0, local_sys_schema_version=1753073979411448, local_tenant_schema_version=1753073979411448) [2025-08-14 09:02:34.731866] WDIAG [RPC] handle_resp (ob_poc_rpc_proxy.cpp:192) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=23610][errcode=0] async_cb handler cost too much time(total_time=23661, decode_time=7, process_time=23651, session_destroy_time=3, ret=0, pcode=1815) [2025-08-14 09:02:34.741760] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:34.742276] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:34.743153] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1076) [1003][BlackListServic][T1][YB42AC4D0007-00063C495F09BD69-0-0] [lt=17][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4012, err_:"OB_TIMEOUT", retry_type:0, client_ret:-4012}, need_retry=false) [2025-08-14 09:02:34.738424] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1006897, expected_run_ts_us_=1755162153731526, ObClockGenerator::getRealClock()=1755162154738424, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891826cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891918190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.743218] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=4784][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891826cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891918190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.743279] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=57][errcode=0] task scheduled too delay(scheduled_delay=995940, expected_run_ts_us_=1755162153747338, ObClockGenerator::getRealClock()=1755162154743279, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.743298] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.743313] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=997221, expected_run_ts_us_=1755162153746092, ObClockGenerator::getRealClock()=1755162154743313, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919845b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.737734] WDIAG [SERVER] do_process (obmp_query.cpp:1141) [999][T1002_L0_G0][T1002][YB42AC4D0007-00063C49651A3232-0-0] [lt=11][errcode=-4283] execute query fail(ret=-4283, timeout_timestamp=1755164952736878) [2025-08-14 09:02:34.743183] WDIAG [SQL] do_close_plan (ob_result_set.cpp:844) [1003][BlackListServic][T1][YB42AC4D0007-00063C495F09BD69-0-0] [lt=27][errcode=-4006] exec result is null(ret=-4006) [2025-08-14 09:02:34.743473] WDIAG [SQL] do_close (ob_result_set.cpp:935) [1003][BlackListServic][T1][YB42AC4D0007-00063C495F09BD69-0-0] [lt=287][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-08-14 09:02:34.743488] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [1003][BlackListServic][T1][YB42AC4D0007-00063C495F09BD69-0-0] [lt=12][errcode=-4012] result set close failed(ret=-4012) [2025-08-14 09:02:34.743499] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:199) [1003][BlackListServic][T1][YB42AC4D0007-00063C495F09BD69-0-0] [lt=9][errcode=-4012] result set close failed(ret=-4012) [2025-08-14 09:02:34.732486] INFO [STORAGE.TRANS] refresh_elr_tenant_config_ (ob_tx_elr_util.cpp:42) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=8] refresh tenant config success(tenant_id=1001, *this={last_refresh_ts:1755162154731676, can_tenant_elr:true}) [2025-08-14 09:02:34.743329] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919845b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.743580] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=249][errcode=0] task scheduled too delay(scheduled_delay=989061, expected_run_ts_us_=1755162153754518, ObClockGenerator::getRealClock()=1755162154743580, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.743596] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.743609] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=988886, expected_run_ts_us_=1755162153754723, ObClockGenerator::getRealClock()=1755162154743609, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.743623] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.743943] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:33.783609] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=32][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a5b58, timer:0x7f994b3feee0, task:0x7f994b3fef08, task_type:"N9oceanbase8observer18ObTableLoadService17ObCheckTenantTaskE", scheduled_time:1755162032742702, pushed_time:0, delay:1000000}, running_task_token={this:0x7f994e1a5df8, timer:0x7f994b3feee0, task:0x7f994b3fef50, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162023758648, pushed_time:1755162122757495, delay:1000000}, total_delay=121040905, threashold=1000000, tenant_id=1) [2025-08-14 09:02:34.739389] WDIAG stat_cleanup (debug.c:35) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=7510][errcode=0] PNIO [delay_warn] cost too much time: 982007us, procedure: pn_pktc_resp_cb [2025-08-14 09:02:34.744387] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=404990483, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:34.744407] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:34.745034] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1001}, delta=410003852, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:34.745050] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:34.746488] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:34.746522] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:34.742457] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=97, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:02:34.747498] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5030][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:34.747519] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=20] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:34.747555] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=25] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:34.747572] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162154747550], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:34.747593] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162154747550]}) [2025-08-14 09:02:34.747609] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] gts statistics(tenant_id=1001, gts_rpc_cnt=0, get_gts_cache_cnt=70, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:02:34.747778] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:34.747786] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:34.747854] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=13] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:34.743508] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:951) [1003][BlackListServic][T1][YB42AC4D0007-00063C495F09BD69-0-0] [lt=9][errcode=-4012] failed to close result(close_ret=-4012, ret=-4012) [2025-08-14 09:02:33.798097] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=34] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:02:34.743636] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=988232, expected_run_ts_us_=1755162153755403, ObClockGenerator::getRealClock()=1755162154743636, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.748022] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=4377][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.748046] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] task scheduled too delay(scheduled_delay=988669, expected_run_ts_us_=1755162153759376, ObClockGenerator::getRealClock()=1755162154748045, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.748060] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.748073] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=990487, expected_run_ts_us_=1755162153757586, ObClockGenerator::getRealClock()=1755162154748073, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eacb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:33.791415] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=20] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:02:34.748130] WDIAG eloop_delay_warn (debug.h:93) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=4941][errcode=0] PNIO [delay_warn] eloop handle events delay high: 987007, malloc=0/0 write=0/0 read=0/1 server_process=0/0 client_cb=982007/1 [2025-08-14 09:02:34.749626] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:34.749746] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=434974564, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=252257, succ_cluster_heartbeat_count=1896, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161003593738002, v:0}, local_cluster_delta=1151156005, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:02:34.755285] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=51] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:34.756254] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=24] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:34.756297] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:33.517302] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1238179, owner: ObLog, click_count: 9, time dist:[FORMAT_END=43, ObMalloc#start=1, ObMalloc#end=243844, ObMalloc#start=17, ObMalloc#end=957010, ObMalloc#start=588, ObMalloc#end=36666, ALLOC_END=5, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:33.517302] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:33.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:34.757402] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1238179][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:33.517|207=1.24s|210=1us|222=1us|total=1.24s) [2025-08-14 09:02:34.757426] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4389] timer task use too much time(end=1755162154757426, start=1755162153517299, delta=1240127) [2025-08-14 09:02:34.757439] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=1251951, expected_run_ts_us_=1755162153505487, ObClockGenerator::getRealClock()=1755162154757439, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:33.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:34.757481] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=0] task scheduled too delay(scheduled_delay=1168164, expected_run_ts_us_=1755162153589316, ObClockGenerator::getRealClock()=1755162154757481, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:33.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:34.757505] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=0] task scheduled too delay(scheduled_delay=1053207, expected_run_ts_us_=1755162153704297, ObClockGenerator::getRealClock()=1755162154757505, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:33.704, task_priority:1, with_handle_protected:1}) [2025-08-14 08:59:59.745979] EDIAG [STORAGE] runTimerTask (ob_checkpoint_service.cpp:243) [15295][T1002_TimerWK9_Flush][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 78898715, owner: ObLog, click_count: 7, time dist:[FORMAT_END=9, ObMalloc#start=0, ObMalloc#end=25411, ObMalloc#start=78873283, ObMalloc#end=3, ALLOC_END=1, APPEND_END=8], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x1c3fe35c 0x1c3fcfe9 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:34.731103] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=983052][errcode=0] epoll handler too much time :984546 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:34.757955] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26799] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:34.747796] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=19] get number(ret=-4038, service_type_=0, range=1, base_id=1755162154731676000, start_id=0, end_id=140295968341544) [2025-08-14 09:02:34.758336] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=10521][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:34.758355] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162154747772], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:34.720454] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=999154][errcode=0] epoll handler too much time :1000685 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:34.758437] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=37899] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:34.754431] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] EASY SLOW: thread: 7f99545f7300, lastrun: 1755162153.746556 cost: 1.007872 loop:26524, slowcnt: 10 [2025-08-14 09:02:34.748087] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eacb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.758950] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10843][errcode=0] task scheduled too delay(scheduled_delay=990959, expected_run_ts_us_=1755162153767990, ObClockGenerator::getRealClock()=1755162154758949, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.767, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.758971] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.767, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.758987] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=979586, expected_run_ts_us_=1755162153779400, ObClockGenerator::getRealClock()=1755162154758987, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.758385] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=27] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162154747772]}) [2025-08-14 09:02:34.760403] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=2013] gts statistics(tenant_id=1002, gts_rpc_cnt=0, get_gts_cache_cnt=84, get_gts_with_stc_cnt=99, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:02:34.759000] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.760845] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1829][errcode=0] task scheduled too delay(scheduled_delay=982511, expected_run_ts_us_=1755162153778332, ObClockGenerator::getRealClock()=1755162154760844, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.760866] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.760879] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=982297, expected_run_ts_us_=1755162153778581, ObClockGenerator::getRealClock()=1755162154760878, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.760889] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.760898] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=990743, expected_run_ts_us_=1755162153770154, ObClockGenerator::getRealClock()=1755162154760898, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.761323] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=423][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.761354] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] task scheduled too delay(scheduled_delay=977702, expected_run_ts_us_=1755162153783651, ObClockGenerator::getRealClock()=1755162154761354, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72716b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a4910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.761368] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72716b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a4910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.761380] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=973352, expected_run_ts_us_=1755162153788027, ObClockGenerator::getRealClock()=1755162154761379, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.763796] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=2410][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.763820] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] task scheduled too delay(scheduled_delay=975552, expected_run_ts_us_=1755162153788267, ObClockGenerator::getRealClock()=1755162154763820, ret=0, ret="OB_SUCCESS", *this={this:0x7f98918100b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.763832] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918100b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.763842] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=982630, expected_run_ts_us_=1755162153781212, ObClockGenerator::getRealClock()=1755162154763842, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.741804] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=21] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:34.764280] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=22437] temporary file module use no blocks [2025-08-14 09:02:34.764288] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:34.764298] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:34.764309] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623056181407, flush_io_finished_ret_:0, flush_io_finished_round_:26314, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:34.763852] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.764448] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=590][errcode=0] task scheduled too delay(scheduled_delay=970652, expected_run_ts_us_=1755162153793794, ObClockGenerator::getRealClock()=1755162154764447, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.793, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.764472] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.793, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.764483] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=972083, expected_run_ts_us_=1755162153792400, ObClockGenerator::getRealClock()=1755162154764483, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180fe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.764501] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180fe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:33.818741] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] [thread pool stat](pool_stat={pool:0x564d6e8922c0, last_check_time:1755162153818740, last_idle_time:102477764955, last_thread_count:12}, *pool={name:TimerWK, this:0x564d6e8922c0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:10, threads_idle_time:102477764955, tenant_id:500}, qsize=10, current_idle_time=102477764955) [2025-08-14 09:02:34.764511] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=958764, expected_run_ts_us_=1755162153805747, ObClockGenerator::getRealClock()=1755162154764511, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.805, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.766362] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1846][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.805, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.766394] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] task scheduled too delay(scheduled_delay=953839, expected_run_ts_us_=1755162153812555, ObClockGenerator::getRealClock()=1755162154766394, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73927b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.812, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.766406] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73927b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.812, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.766417] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=950950, expected_run_ts_us_=1755162153815466, ObClockGenerator::getRealClock()=1755162154766417, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7399090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.815, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:31.748975] EDIAG [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1531) [354][EvtHisUpdTask][T1][YB42AC4D0007-00063C495FE9A745-0-0] [lt=4208][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3015016, owner: ObLog, click_count: 5, time dist:[FORMAT_END=3015008, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x9ace3c3 0x9acda90 0xfad1d79 0xfad1b46 0x148e2b50 0x93d7bb4 0x9394489 0x9725f91 0x124dfd77 0x96a5a4d 0x97aa16f 0x124eac91 0x97a9944 0x261395f9 0x1f900517 0x1f8fe2bf 0x9a5f486 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 08:59:30.770794] EDIAG [SHARE] set_default_timeout_ctx (ob_share_util.cpp:150) [341][LSSysTblUp0][T0][YB42AC4D0007-00063C495B49B120-0-0] [lt=7][errcode=-4012] LOGGER COST TOO MUCH TIME, cost: 183720188, owner: ObLog, click_count: 7, time dist:[FORMAT_END=2549, ObMalloc#start=1, ObMalloc#end=3986, ALLOC_END=183713649, APPEND_END=3, ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=1], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x1fd0b986 0x9814c98 0x1eed1d13 0x1eecf030 0x1eecdd21 0x1eeb346c 0x1fcca226 0x1fcc9df1 0x211cb1a5 0x1eed018e 0x1eeb2659 0x9a05aa8 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:31.748985] EDIAG [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1531) [373][Occam][T1][YB42AC4D0007-00063C4960599C9C-0-0] [lt=3972][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3015978, owner: ObLog, click_count: 3, time dist:[FORMAT_END=3015972, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2], BACKTRACE: 0x9ace3c3 0x9acda90 0xfad1d79 0xfad1b46 0x148e2b50 0x93d7bb4 0x9394489 0x9725f91 0x124dfd77 0x96a5a4d 0x97aa16f 0x124eac91 0x97a9944 0x261395f9 0xfa0e430 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:34.766112] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=945935] [thread pool stat](pool_stat={pool:0x7f99667cbcf0, last_check_time:1755162154766110, last_idle_time:34763902764, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x7f99667cbcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:34763902764, tenant_id:500}, qsize=-1, current_idle_time=34763902764) [2025-08-14 09:02:34.768447] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=2329] [thread pool stat](pool_stat={pool:0x7f99667ccaf0, last_check_time:1755162154768446, last_idle_time:26538376076, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x7f99667ccaf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:26538376076, tenant_id:500}, qsize=-1, current_idle_time=26538376076) [2025-08-14 09:02:34.768503] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=55] [thread pool stat](pool_stat={pool:0x7f9969de56b0, last_check_time:1755162154768503, last_idle_time:102938681541, last_thread_count:17}, *pool={name:TimerWK, this:0x7f9969de56b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:16, threads_idle_time:102938681541, tenant_id:1}, qsize=10, current_idle_time=102938681541) [2025-08-14 09:02:34.768514] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] [thread pool stat](pool_stat={pool:0x7f99356b0e70, last_check_time:1755162154768514, last_idle_time:102366343177, last_thread_count:15}, *pool={name:TimerWK, this:0x7f99356b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:12, threads_idle_time:102366343177, tenant_id:1001}, qsize=9, current_idle_time=102366343177) [2025-08-14 09:02:34.768525] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] [thread pool stat](pool_stat={pool:0x7f99230b0e70, last_check_time:1755162154768524, last_idle_time:102534969158, last_thread_count:17}, *pool={name:TimerWK, this:0x7f99230b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:16, threads_idle_time:102534969158, tenant_id:1002}, qsize=15, current_idle_time=102534969158) [2025-08-14 09:02:31.748975] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1531) [354][EvtHisUpdTask][T1][YB42AC4D0007-00063C495FE9A745-0-0] [lt=4208] end stmt(ret=0, tx_id=21195869, plain_select=false, stmt_type=2, savepoint={branch:0, seq:2}, tx_desc={this:0x7f98b2d87140, tx_id:{txid:21195869}, state:1, addr:"172.77.0.7:2882", tenant_id:1, session_id:1, assoc_session_id:1, client_sid:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:3, alloc_ts:1755162008738469, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999989, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1753069619, cluster_version:17180067074, seq_base:1755162007854427, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:2}, trans_result={incomplete:false, parts:[], touched_ls_list:[], conflict_txs:[]}, rollback=true, need_rollback=true, session={this:0x7f984c604038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f98b2d87140}, exec_ctx.get_errcode()=-4038) [2025-08-14 08:59:30.770794] WDIAG [SHARE] set_default_timeout_ctx (ob_share_util.cpp:150) [341][LSSysTblUp0][T0][YB42AC4D0007-00063C495B49B120-0-0] [lt=7][errcode=-4012] timeouted(ret=-4012, ret="OB_TIMEOUT", abs_timeout_ts=1755161965774613, ctx_timeout_ts=1755161965774613, worker_timeout_ts=9223372036854775807, default_timeout=2000000) [2025-08-14 09:02:31.748985] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1531) [373][Occam][T1][YB42AC4D0007-00063C4960599C9C-0-0] [lt=3972] end stmt(ret=0, tx_id=21195867, plain_select=false, stmt_type=2, savepoint={branch:0, seq:40095}, tx_desc={this:0x7f98c909c5c0, tx_id:{txid:21195867}, state:1, addr:"172.77.0.7:2882", tenant_id:1, session_id:1, assoc_session_id:1, client_sid:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:3, alloc_ts:1755162005734943, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999989, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1753069619, cluster_version:17180067074, seq_base:1755162005731220, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:2}, trans_result={incomplete:false, parts:[], touched_ls_list:[], conflict_txs:[]}, rollback=true, need_rollback=true, session={this:0x7f985b410038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f98c909c5c0}, exec_ctx.get_errcode()=-4038) [2025-08-14 09:02:34.758807] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:981) [1003][BlackListServic][T1][YB42AC4D0007-00063C495F09BD69-0-0] [lt=4375][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, record_ret=-4012, ret=-4012) [2025-08-14 09:02:34.771017] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:763) [1003][BlackListServic][T0][YB42AC4D0007-00063C495F09BD69-0-0] [lt=12185] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, process_time=15032682) [2025-08-14 09:02:34.771028] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1018) [1003][BlackListServic][T0][YB42AC4D0007-00063C495F09BD69-0-0] [lt=12][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, aret=-4012, ret=-4012) [2025-08-14 09:02:34.771041] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1808) [1003][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2025-08-14 09:02:34.766427] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7399090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.815, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.771126] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=4688][errcode=0] task scheduled too delay(scheduled_delay=954827, expected_run_ts_us_=1755162153816298, ObClockGenerator::getRealClock()=1755162154771126, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.771150] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.771166] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] task scheduled too delay(scheduled_delay=943836, expected_run_ts_us_=1755162153827329, ObClockGenerator::getRealClock()=1755162154771166, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73954b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.827, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.771182] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73954b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.827, task_priority:1, with_handle_protected:0}) [2025-08-14 09:00:02.763000] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:763) [826][T1002_CdcSrv][T1002][YB42AC4D0007-00063C495D29A61B-0-0] [lt=6989] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"select dest_no, value from __all_log_archive_dest_parameter where tenant_id=1002 and name='path' order by dest_no asc"}, process_time=30984196) [2025-08-14 09:00:02.763000] EDIAG [SERVER] process_final (ob_inner_sql_connection.cpp:763) [826][T1002_CdcSrv][T1002][YB42AC4D0007-00063C495D29A61B-0-0] [lt=6989][errcode=0] LOGGER COST TOO MUCH TIME, cost: 151779809, owner: ObLog, click_count: 7, time dist:[FORMAT_END=11, ObMalloc#start=0, ObMalloc#end=31, ObMalloc#start=151765704, ObMalloc#end=14057, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1275158c 0x96a79f6 0x96a5754 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x1f5a5003 0x1236d9da 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:34.771053] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1079) [1003][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2025-08-14 09:02:34.774439] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1748) [1003][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=3377][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1753069619, tenant_id=1) [2025-08-14 09:02:34.774454] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [1003][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4012] query failed(ret=-4012, conn=0x7f98bfa9c050, start=1755162138765205, sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2025-08-14 09:02:34.762476] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=31] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119797572, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755162138745740) [2025-08-14 09:02:34.775069] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11999] get wrs ts(ls_id={id:1}, delta=434988972, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:34.775086] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:34.771198] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] task scheduled too delay(scheduled_delay=897514, expected_run_ts_us_=1755162153873684, ObClockGenerator::getRealClock()=1755162154771198, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.873, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.775127] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=3921][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:33.873, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.775155] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] task scheduled too delay(scheduled_delay=526982, expected_run_ts_us_=1755162154248172, ObClockGenerator::getRealClock()=1755162154775155, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:34.248, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.775167] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:34.248, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.775193] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7270030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a3b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:34.735, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.775349] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1018) [826][T1002_CdcSrv][T1002][YB42AC4D0007-00063C495D29A61B-0-0] [lt=151779809][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"select dest_no, value from __all_log_archive_dest_parameter where tenant_id=1002 and name='path' order by dest_no asc"}, aret=-4012, ret=-4012) [2025-08-14 09:02:34.775386] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1808) [826][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1001, sql=select dest_no, value from __all_log_archive_dest_parameter where tenant_id=1002 and name='path' order by dest_no asc) [2025-08-14 09:02:34.775392] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1079) [826][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1001) [2025-08-14 09:02:34.775399] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1748) [826][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=4][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1753069619, tenant_id=1001) [2025-08-14 09:02:34.775404] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [826][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=-4012] query failed(ret=-4012, conn=0x7f98f673f780, start=1755161970749487, sql=select dest_no, value from __all_log_archive_dest_parameter where tenant_id=1002 and name='path' order by dest_no asc) [2025-08-14 09:02:34.775418] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [826][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4012] read failed(ret=-4012) [2025-08-14 09:02:34.767188] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=963952] gc stale ls task succ [2025-08-14 09:02:34.735143] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=2325] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:34.775866] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=40660] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3927) [2025-08-14 09:02:34.775872] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:34.738529] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:946) [469][T1_Occam][T1][YB42AC4D0007-00063C495C5A1329-0-0] [lt=16][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='172.77.0.7' and svr_port=2882"}, retry_cnt=10, local_sys_schema_version=1753073979411448, local_tenant_schema_version=1753073979411448) [2025-08-14 09:02:34.775900] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1076) [469][T1_Occam][T1][YB42AC4D0007-00063C495C5A1329-0-0] [lt=37311][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:10, local_retry_times:10, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2025-08-14 09:02:34.775922] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [469][T1_Occam][T1][YB42AC4D0007-00063C495C5A1329-0-0] [lt=12][errcode=-4283] result set close failed(ret=-4283) [2025-08-14 09:02:34.775929] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:199) [469][T1_Occam][T1][YB42AC4D0007-00063C495C5A1329-0-0] [lt=5][errcode=-4283] result set close failed(ret=-4283) [2025-08-14 09:02:34.775934] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:951) [469][T1_Occam][T1][YB42AC4D0007-00063C495C5A1329-0-0] [lt=5][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2025-08-14 09:02:34.775951] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:981) [469][T1_Occam][T1][YB42AC4D0007-00063C495C5A1329-0-0] [lt=9][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='172.77.0.7' and svr_port=2882"}, record_ret=-4012, ret=-4012) [2025-08-14 09:02:34.775958] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:763) [469][T1_Occam][T1][YB42AC4D0007-00063C495C5A1329-0-0] [lt=6] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='172.77.0.7' and svr_port=2882"}, process_time=177009449) [2025-08-14 09:02:34.775965] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1018) [469][T1_Occam][T1][YB42AC4D0007-00063C495C5A1329-0-0] [lt=6][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='172.77.0.7' and svr_port=2882"}, aret=-4012, ret=-4012) [2025-08-14 09:02:34.775971] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1808) [469][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT zone FROM __all_server where svr_ip='172.77.0.7' and svr_port=2882) [2025-08-14 09:02:34.775977] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1079) [469][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2025-08-14 09:02:34.775982] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1748) [469][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1753069619, tenant_id=1) [2025-08-14 09:02:34.775988] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [469][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4012] query failed(ret=-4012, conn=0x7f98d486c050, start=1755161977766489, sql=SELECT zone FROM __all_server where svr_ip='172.77.0.7' and svr_port=2882) [2025-08-14 09:02:34.776009] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:114) [1003][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4012] read failed(ret=-4012) [2025-08-14 09:02:34.775203] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ecb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:34.739, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.776099] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=891][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7353730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7396990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:34.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.776118] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:34.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.776132] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ea30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:34.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.776145] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919afbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:34.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.776274] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=26304] current monitor number(seq_=-1) [2025-08-14 09:02:34.775993] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [469][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4012] read failed(ret=-4012) [2025-08-14 09:02:34.744275] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=971867] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:34.776783] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32457][errcode=0] epoll handler too much time :32513 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:34.778410] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=32] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:34.776159] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919854b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:34.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.778649] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=2482][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:34.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.778673] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:34.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.778687] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d45b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:34.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.778700] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72747b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:34.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.778713] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:34.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.782884] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:34.783263] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:34.785243] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:34.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.785277] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:34.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.785293] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:34.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.785308] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:34.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.791185] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=22] gc stale ls task succ [2025-08-14 09:02:34.795304] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:34.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.795339] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:34.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.801007] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=2448][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:34.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:34.815403] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919836b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:34.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.835376] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:34.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:34.836563] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:34.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:34.837018] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=405095180, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:34.837041] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:34.837655] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1001}, delta=410109605, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:34.837668] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:34.838754] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=23] PNIO [ratelimit] time: 1755162154838751, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007618, add_bytes: 0 [2025-08-14 09:02:34.845595] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:34.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.845628] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:34.831, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.848058] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c0b90, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571c0e70, time_wheel:0x7f9959fef990, thread_pool:0x7f9959fef170, is_running:1, time_interval:3.00s, expected_run_time:09:02:34.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:34.850457] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=435074896, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:34.850489] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=29] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:34.860724] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=199] LogLoopThread round_cost_time(us)(round_cost_time=225) [2025-08-14 09:02:34.875792] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:34.862, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.883338] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:34.935437] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:34.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:34.936624] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:34.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:34.958882] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:34.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:34.983388] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:35.016835] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:35.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.022537] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:35.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.035506] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:34.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.036686] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:34.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.058961] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:34.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.083556] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:35.119596] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:02:35.119649] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:02:35.119688] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:02:35.119727] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:02:35.119745] INFO [COMMON] print_io_status (ob_io_manager.cpp:2639) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:5, norm_iops:1, bw:20446, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:12736, total:12741) [2025-08-14 09:02:35.119766] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, param_config_:{memory_limit:536870912, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=865568, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=1, ops=0, iops=1, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:02:35.119804] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2826) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=34] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:5, avg_bw:20444, [delay/us]: prepare:0, schedule:0, submit:600000000, device:12736, total:12741) [2025-08-14 09:02:35.119822] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:02:35.119835] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:35.119853] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:35.119863] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SERVER] net_in=0kB/s, net_out=0kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=1280000kB/s [2025-08-14 09:02:35.135568] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:35.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.136747] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:35.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.159056] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:35.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.183800] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:35.235630] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:35.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.236828] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:35.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.237529] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:35.237555] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:02:35.237575] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:35.237587] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:35.237606] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:35.237619] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:35.259120] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:35.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.260331] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:35.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.260360] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:35.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.260685] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=13] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:35.260706] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162155260681], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:35.260734] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162155260681], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:35.260750] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162155260681]}) [2025-08-14 09:02:35.260776] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=11] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:35.260791] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162155260763], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:35.260803] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162155260763]}) [2025-08-14 09:02:35.261015] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:35.261027] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:35.261041] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162155261012], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:35.261053] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162155261012]}) [2025-08-14 09:02:35.270394] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:35.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.283868] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:35.300585] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:35.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.335661] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:35.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.359192] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:35.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.383926] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:35.435686] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:35.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.459273] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:35.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.483982] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:35.512040] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:35.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.512088] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:35.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.522035] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:35.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.535719] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=52][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:35.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.559321] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:35.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.584050] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:35.592595] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:35.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.635787] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:35.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.659394] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:35.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.683742] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:35.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.684099] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:35.717969] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:35.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.720734] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:35.721551] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [276]OB_MALLOC COST TOO MUCH TIME, cost_time=540953, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:02:34.782962] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=8] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:02:35.237638] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:35.336890] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:35.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.726287] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=388803][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:35.336|207=389.39ms|210=0us|222=2us|total=389.41ms) [2025-08-14 09:02:35.726307] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task use too much time(end=1755162155726307, start=1755162155336889, delta=389418) [2025-08-14 09:02:35.730715] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=15] gc stale ls task succ [2025-08-14 09:02:35.730872] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=965036] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:35.732293] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:35.184398] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc4738, timer:0x7f9970f40f80, task:0x564d6cd5b940, task_type:"N9oceanbase3sql15ObSQLSessionMgrE", scheduled_time:1755161613781775, pushed_time:0, delay:5000000}, running_task_token={this:0x7f9970fc5fb8, timer:0x7f9970f40f80, task:0x564d727182f0, task_type:"N9oceanbase5share16ObAshRefreshTaskE", scheduled_time:1755161611717613, pushed_time:1755161614757742, delay:120000000}, total_delay=541402621, threashold=1000000, tenant_id=500) [2025-08-14 09:02:35.726259] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=487875][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:01:49.731, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:01:42.751], idx=659, next_allow_print_ts="09:02:19.731") [2025-08-14 09:02:35.733710] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7435][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:35.733732] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:35.733745] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:35.733767] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:35.733782] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:35.733986] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:35.734003] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:35.736414] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:35.732319] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=21] temporary file module use no blocks [2025-08-14 09:02:35.739324] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7319] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:35.739362] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=38] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:35.739375] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:35.739385] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:35.739392] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:35.739403] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:35.739416] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:35.739425] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:35.739430] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:35.739442] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:02:35.739449] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:02:35.739465] INFO [COMMON] loop_running_dag_net_list (ob_tenant_dag_scheduler.cpp:3845) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=14] loop running dag net list(ret=0, running_dag_net_list_size=0, slow_dag_net_cnt=0, finished_dag_net_list_size=0) [2025-08-14 09:02:35.739486] INFO [COMMON] loop_finished_dag_net_list (ob_tenant_dag_scheduler.cpp:3868) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=21] loop finsihed dag net list(ret=0, finished_dag_net_list_size=0) [2025-08-14 09:02:35.741255] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:35.742256] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [729]OB_MALLOC COST TOO MUCH TIME, cost_time=791680, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=914731, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [857]OB_MALLOC COST TOO MUCH TIME, cost_time=807088, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:02:35.743453] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=435966782, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:35.743490] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=33] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:35.744103] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] get wrs ts(ls_id={id:1}, delta=435966782, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:35.744117] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:35.744747] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=608003445, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:35.744761] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:35.745425] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=406000768, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:35.745443] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:35.746032] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=411014137, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:35.746044] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:35.746062] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=397524262, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=251693, succ_cluster_heartbeat_count=1494, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161006924036004, v:0}, local_cluster_delta=1148822024, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:02:35.746108] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=37] get wrs ts(ls_id={id:1}, delta=406000768, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:35.746119] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:35.746844] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1001}, delta=411014137, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:35.746858] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:00:01.744365] EDIAG [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:185) [14776][T1_TimerWK4_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=995348][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1997230, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1997229, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1b3f0f1a 0x9aaaf7e 0x1c3fa49f 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:35.748718] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=608003445, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:35.748736] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:35.756307] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=23] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:35.757585] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=18] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:35.757597] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:35.764633] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=899] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:35.747943] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:35.766322] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18347][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918114b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:35.766351] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728aa90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:35.766366] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:35.755877] WDIAG [SHARE] get_valid_dest_pairs (ob_archive_persist_helper.cpp:455) [826][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4012] failed to exec sql(ret=-4012, sql=select dest_no, value from __all_log_archive_dest_parameter where tenant_id=1002 and name='path' order by dest_no asc) [2025-08-14 09:02:35.767344] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [826][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=11422][errcode=0] long time to hold one guard(schema_mgr=0x7f98ad7818f0, tenant_id=1, version=1753073979411448, cur_timestamp=1755162155767335, ref_timestamp=1755161999795739, lbt()="0x26a1806d 0x974f3e5 0x974e159 0x974e084 0x948f9df 0x97257c0 0x97256f4 0x9725691 0x1f5a5221 0x1236d9da 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3") [2025-08-14 09:02:35.767363] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [826][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] long time to hold one guard(schema_mgr=0x7f988fae9820, tenant_id=1001, version=1753074886385104, cur_timestamp=1755162155767359, ref_timestamp=1755161999795739, lbt()="0x26a1806d 0x974f3e5 0x974e159 0x974e084 0x948fab3 0x97257c0 0x97256f4 0x9725691 0x1f5a5221 0x1236d9da 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3") [2025-08-14 09:02:35.769295] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=8638] gc stale ls task succ [2025-08-14 09:02:35.766378] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:35.769408] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=3022][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:35.769432] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:35.769442] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:35.769452] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:35.771578] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119812272, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-08-14 09:02:35.771607] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=406027849, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:35.771625] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:35.772208] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1001}, delta=411041218, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:35.772220] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:35.772338] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [316][pnio1][T1002][YB42AC4D0007-00063C49612B8F6D-0-0] [lt=14][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1002, delta=1991054, last_succ_cluster_heartbeat_tstamp_=1755162153781283) [2025-08-14 09:02:35.769463] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fc830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:35.773751] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=4278][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:35.773773] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a54b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:35.726154] INFO [ARCHIVE] produce_log_fetch_task_ (ob_archive_sequencer.cpp:166) [731][T1001_ArcSeq][T1001][YB42AC4D0007-00063C495C999C16-0-0] [lt=994460] archive round not in doing status, just skip(key={incarnation:-1, dest_id:-1, round:-1}, state={status:"INVALID"}) [2025-08-14 09:02:35.759383] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=4073][errcode=0] EASY SLOW: thread: 7f99542fb300, lastrun: 1755162115.752503 cost: 40.006868 loop:26416, slowcnt: 342 [2025-08-14 09:02:35.764807] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:35.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.778267] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:34.806253] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=13] [sql nio session](*s={this:0x7f98f733f030, session_id:3221589417, trace_id:YB42AC4D0007-00063C496359B268-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:252}, err:0, last_decode_time:1755161813752329, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1778, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:35.776689] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11860][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:35.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.779561] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72edc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:35.779585] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72edb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:35.779596] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:35.780662] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=543] update_check_range(min_check_version=2540, max_check_version=2540, global_version=2641) [2025-08-14 09:02:35.780520] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=263][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:35.780822] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=301] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:35.780842] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=10] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:35.780853] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162155780839], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:35.780868] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162155780839]}) [2025-08-14 09:02:35.780878] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162155780839]}) [2025-08-14 09:02:35.781073] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:35.781086] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:35.781099] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162155781070], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:35.781112] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162155781070]}) [2025-08-14 09:02:35.781122] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162155781070]}) [2025-08-14 09:02:35.778490] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=970768] [sql nio session](*s={this:0x7f98f733e030, session_id:3221589980, trace_id:YB42AC4D0007-00063C496359B261-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:251}, err:0, last_decode_time:1755161748778070, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1681, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:35.784164] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:35.808671] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=337][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:35.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.826409] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:35.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.846385] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=27] PNIO [ratelimit] time: 1755162155846383, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007632, add_bytes: 0 [749]OB_MALLOC COST TOO MUCH TIME, cost_time=966845, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=28440], seq:[0, 1] [2025-08-14 09:02:35.779606] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.716314] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=935284][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:35.779|207=936.70ms|210=1us|222=2us|total=936.74ms) [2025-08-14 09:02:36.716353] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4389] timer task use too much time(end=1755162156716352, start=1755162155779605, delta=936747) [2025-08-14 09:02:35.748024] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [749][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB90B7-0-0] [lt=24][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1001, svr="172.77.0.7:2882", version={val:1755161719775179002, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162154949847) [2025-08-14 09:02:36.716370] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=955370, expected_run_ts_us_=1755162155760998, ObClockGenerator::getRealClock()=1755162156716369, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.716401] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.716381] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [749][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB90B7-0-0] [lt=966891][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1001, req={req_server:"172.77.0.7:2882", version:{val:1755161719775179002, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162154949847}, twrs={inited:true, tenant_id:1001, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162155743532}, server_version_for_stat:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162155743532}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:35.781315] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=2820] [sql nio session](*s={this:0x7f98f7332030, session_id:3221590132, trace_id:YB42AC4D0007-00063C496359B265-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:266}, err:0, last_decode_time:1755161768783802, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:2125, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:36.718615] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1421) [771][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13732] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1001, is_allowed_broadcast=true) [770]OB_MALLOC COST TOO MUCH TIME, cost_time=791050, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=43, WASH_BLOCK_END=791005, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1, 2, 3, 4] [381]OB_MALLOC COST TOO MUCH TIME, cost_time=436777, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=52, WASH_BLOCK_END=436725], seq:[0, 1, 2] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=483814, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=58, WASH_BLOCK_END=483756], seq:[0, 1, 2] [2025-08-14 09:02:35.926478] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:35.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:36.718784] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=791105][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:35.926|207=792.30ms|210=0us|222=2us|total=792.33ms) [2025-08-14 09:02:36.718807] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4389] timer task use too much time(end=1755162156718807, start=1755162155926476, delta=792331) [2025-08-14 09:02:36.011577] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1919][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:36.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:36.234189] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:36.718846] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=483918][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:02:36.718876] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:36.716417] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=945582, expected_run_ts_us_=1755162155770834, ObClockGenerator::getRealClock()=1755162156716417, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.719325] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=2901][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.719349] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] task scheduled too delay(scheduled_delay=940009, expected_run_ts_us_=1755162155779339, ObClockGenerator::getRealClock()=1755162156719349, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728ac10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.719379] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728ac10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.779, task_priority:1, with_handle_protected:0}) [473]OB_MALLOC COST TOO MUCH TIME, cost_time=969839, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=12150, WASH_BLOCK_END=8983, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=9089], seq:[0, 1, 2, 3, 4] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=942281, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=34, WASH_BLOCK_END=942211, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=36], seq:[0, 1, 2, 3, 4] [373]OB_MALLOC COST TOO MUCH TIME, cost_time=958459, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=958458], seq:[0] [729]OB_MALLOC COST TOO MUCH TIME, cost_time=889341, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=888989, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:02:36.718840] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=706192][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:36.115|207=707.26ms|210=1us|222=1us|total=707.27ms) [2025-08-14 09:02:36.718888] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:36.727737] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8826][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:36.727751] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:36.727774] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:36.719394] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] task scheduled too delay(scheduled_delay=916489, expected_run_ts_us_=1755162155802905, ObClockGenerator::getRealClock()=1755162156719394, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.802, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.727812] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8403][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.802, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.727829] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=926354, expected_run_ts_us_=1755162155801474, ObClockGenerator::getRealClock()=1755162156727828, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.727792] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:01:49.731, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:01:42.751], idx=659, next_allow_print_ts="09:02:19.731") [2025-08-14 09:02:36.730505] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=2705][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:36.730522] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:36.730532] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:35.276254] EDIAG [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1458670, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=7, ALLOC_BLOCK#end=3, FORMAT_END=1458654, ObMalloc#start=1, ObMalloc#end=4, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x18257d0c 0x1823b885 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:35.733919] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1003364, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1003363, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:35.736024] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1001301, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=8, ALLOC_BLOCK#end=3, FORMAT_END=989864, ALLOC_END=11426, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [472]OB_MALLOC COST TOO MUCH TIME, cost_time=778125, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=34, WASH_BLOCK_END=778045, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=45], seq:[0, 1, 2, 3, 4] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=717781, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=48, WASH_BLOCK_END=717733], seq:[0, 1, 2] [857]OB_MALLOC COST TOO MUCH TIME, cost_time=868151, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=44, WASH_BLOCK_END=868062, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=45], seq:[0, 1, 2, 3, 4] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=855692, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=34, WASH_BLOCK_END=855657], seq:[0, 1, 2] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=749213, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=36, WASH_BLOCK_END=749177], seq:[0, 1, 2] [373]OB_MALLOC COST TOO MUCH TIME, cost_time=979823, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=958473], seq:[0, 1] [2025-08-14 09:02:34.820701] EDIAG try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=1692943][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1928234, owner: ObLog, click_count: 8, time dist:[ALLOC_BLOCK#start=11, ALLOC_BLOCK#end=3, FORMAT_END=1926837, ObMalloc#start=1, ObMalloc#end=79, ObMalloc#start=21, ObMalloc#end=48, FREE_END=1234], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x26499505 0x97074a3 0x26498fa9 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:36.503566] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=4][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fe030, caller:ob_table_lock_service.cpp:start:276, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE6$_1273, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe310, time_wheel:0x7f99535f3990, thread_pool:0x7f99535f3170, is_running:1, time_interval:10.00s, expected_run_time:09:02:36.426, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.276254] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:36.737163] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=1458670] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3928) [2025-08-14 09:02:36.737172] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:36.737195] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:36.737202] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=6] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3929) [2025-08-14 09:02:36.737207] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:36.737397] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:36.737405] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3930) [2025-08-14 09:02:36.737410] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:36.727840] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.738633] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10773][errcode=0] task scheduled too delay(scheduled_delay=934322, expected_run_ts_us_=1755162155804310, ObClockGenerator::getRealClock()=1755162156738633, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728f9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.738658] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728f9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.738671] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=920130, expected_run_ts_us_=1755162155818540, ObClockGenerator::getRealClock()=1755162156738671, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eae10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:35.733919] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:35.736024] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:35.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:36.721595] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:36.730549] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:35.791249] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=21] gc stale ls task succ [2025-08-14 09:02:36.735043] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=231122][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:36.503|207=231.47ms|210=0us|222=1us|total=231.50ms) [2025-08-14 09:02:36.738827] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1003364][errcode=0] epoll handler too much time :1004915 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:36.752487] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13618] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:36.738861] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1001301][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:35.736|207=1.00s|210=0us|222=2us|total=1.00s) [2025-08-14 09:02:36.752508] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=13626][errcode=-4389] timer task use too much time(end=1755162156752508, start=1755162155736022, delta=1016486) [2025-08-14 09:02:36.738689] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eae10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.752691] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13976][errcode=0] task scheduled too delay(scheduled_delay=929783, expected_run_ts_us_=1755162155822907, ObClockGenerator::getRealClock()=1755162156752691, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.752710] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.752720] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=887362, expected_run_ts_us_=1755162155865358, ObClockGenerator::getRealClock()=1755162156752720, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.735761] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=608996246, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:36.734945] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=436959583, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:36.742191] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=355][errcode=-4389] timer task use too much time(end=1755162156742191, start=1755162155990985, delta=751206) [2025-08-14 09:02:36.753149] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:951) [354][EvtHisUpdTask][T1][YB42AC4D0007-00063C495FE9A745-0-0] [lt=1253][errcode=-4012] failed to close result(close_ret=-4038, ret=-4012) [2025-08-14 09:02:36.281326] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:36.756571] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=474525][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162156281322], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:36.756591] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162156281322], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:36.756607] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162156281322]}) [2025-08-14 09:02:36.756628] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=10] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:36.756636] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162156756624], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:36.756647] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162156756624]}) [2025-08-14 09:02:36.756813] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:36.756830] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:36.756844] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162156756810], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:36.752454] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=17383][errcode=-4389] timer task use too much time(end=1755162156752453, start=1755162156503562, delta=248891) [2025-08-14 09:02:36.752729] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.757011] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=4272][errcode=0] task scheduled too delay(scheduled_delay=890237, expected_run_ts_us_=1755162155866773, ObClockGenerator::getRealClock()=1755162156757011, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.866, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.757033] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:35.866, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.757053] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=427954, expected_run_ts_us_=1755162156329099, ObClockGenerator::getRealClock()=1755162156757053, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.757065] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.757082] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72756b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.756855] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162156756810]}) [2025-08-14 09:02:36.748603] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=960405] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:02:35.767482] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [373][Occam][T1][YB42AC4D0007-00063C4960599C9C-0-0] [lt=3015978][errcode=-4038] result set close failed(ret=-4038) [2025-08-14 09:02:36.761832] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:199) [373][Occam][T1][YB42AC4D0007-00063C4960599C9C-0-0] [lt=992844][errcode=-4038] result set close failed(ret=-4038) [2025-08-14 09:02:36.730813] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=17] gc stale ls task succ [2025-08-14 09:02:36.761058] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=3967][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.724, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.765093] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=4025][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a36b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.765110] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919bfbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aa690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.765127] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919911b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196cc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.757289] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:981) [354][EvtHisUpdTask][T1][YB42AC4D0007-00063C495FE9A745-0-0] [lt=11][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_rootservice_event_history (gmt_create, module, event, name1, value1, name2, value2, value3, value4, value5, value6, rs_svr_ip, rs_svr_port) VALUES (usec_to_time(1755161441756996), 'server', 'load_servers', 'ret', 0, 'has_build', 1, '', '', '', '', '172.77.0.7', 2882)"}, record_ret=-4012, ret=-4012) [2025-08-14 09:02:36.775770] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:763) [354][EvtHisUpdTask][T0][YB42AC4D0007-00063C495FE9A745-0-0] [lt=18448] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"INSERT INTO __all_rootservice_event_history (gmt_create, module, event, name1, value1, name2, value2, value3, value4, value5, value6, rs_svr_ip, rs_svr_port) VALUES (usec_to_time(1755161441756996), 'server', 'load_servers', 'ret', 0, 'has_build', 1, '', '', '', '', '172.77.0.7', 2882)"}, process_time=169007303) [2025-08-14 09:02:36.775785] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1018) [354][EvtHisUpdTask][T0][YB42AC4D0007-00063C495FE9A745-0-0] [lt=16][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"INSERT INTO __all_rootservice_event_history (gmt_create, module, event, name1, value1, name2, value2, value3, value4, value5, value6, rs_svr_ip, rs_svr_port) VALUES (usec_to_time(1755161441756996), 'server', 'load_servers', 'ret', 0, 'has_build', 1, '', '', '', '', '172.77.0.7', 2882)"}, aret=-4012, ret=-4012) [2025-08-14 09:02:36.775800] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1590) [354][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=INSERT INTO __all_rootservice_event_history (gmt_create, module, event, name1, value1, name2, value2, value3, value4, value5, value6, rs_svr_ip, rs_svr_port) VALUES (usec_to_time(1755161441756996), 'server', 'load_servers', 'ret', 0, 'has_build', 1, '', '', '', '', '172.77.0.7', 2882)) [2025-08-14 09:02:36.775842] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [354][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] long time to hold one guard(schema_mgr=0x7f98ad7818f0, tenant_id=1, version=1753073979411448, cur_timestamp=1755162156775833, ref_timestamp=1755162008738469, lbt()="0x26a1806d 0x974f3e5 0x974e159 0x974e084 0x948f9df 0x97257c0 0x124e2fb7 0x97aa07c 0x124eac91 0x97a9944 0x261395f9 0x1f900517 0x1f8fe2bf 0x9a5f486 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3") [2025-08-14 09:02:36.775865] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1079) [354][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2025-08-14 09:02:35.757605] EDIAG [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1020401, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1, FORMAT_END=1020394, ObMalloc#start=1, ObMalloc#end=1, ALLOC_END=0, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x99721a1 0x997a72e 0x2006ed17 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:36.779497] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:35.757605] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:36.779578] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1020401] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:36.779588] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:36.779597] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:36.779605] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:36.779614] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:36.779622] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:36.779633] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:02:36.779643] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:02:36.742373] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:36.779673] WDIAG ev_invoke_pending (ev.c:2136) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=37242][errcode=0] epoll handler too much time :37308 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:36.765136] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d40b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.780129] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14967][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.780149] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.780162] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.780173] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.775873] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1543) [354][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4012] execute_write failed(ret=-4012, tenant_id=1, sql=INSERT INTO __all_rootservice_event_history (gmt_create, module, event, name1, value1, name2, value2, value3, value4, value5, value6, rs_svr_ip, rs_svr_port) VALUES (usec_to_time(1755161441756996), 'server', 'load_servers', 'ret', 0, 'has_build', 1, '', '', '', '', '172.77.0.7', 2882), is_user_sql=false) [2025-08-14 09:02:36.780186] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.744, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.782044] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1852][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aaf90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.782076] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aba10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.782094] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.782110] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196cb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.782119] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.782136] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.751, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.782152] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.782168] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919acf10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.782184] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.792207] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919abe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.792260] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=50][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.792273] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199fe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919acd90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.792290] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e80b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.792306] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a31b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ede10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.794246] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=956704] gc stale ls task succ [2025-08-14 09:02:36.812337] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.806, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.812372] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.812391] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.812407] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.800, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.812419] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728e330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7290690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.805, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.812436] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.803, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.819244] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:36.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:36.832463] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.824, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.842519] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.832, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.842901] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1296][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:36.844403] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=944][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:36.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:36.852584] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=5][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:36.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:36.853984] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=25] PNIO [ratelimit] time: 1755162156853981, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007598, add_bytes: 0 [2025-08-14 09:02:36.872632] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ade10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.864, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.872676] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.872699] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919adc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.872711] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919adb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.882697] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72865b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.878, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.902834] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728de30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7290b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.899, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.912895] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:36.902, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:36.919315] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:36.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:36.942954] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:36.944468] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:36.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:36.952637] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:36.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.019370] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:36.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.040407] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=547912][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc4ff8, timer:0x7f9970f40f80, task:0x564d6d07c170, task_type:"N9oceanbase8observer8ObServer25ObRefreshNetworkSpeedTaskE", scheduled_time:1755161615749643, pushed_time:0, delay:1000000}, running_task_token={this:0x7f9970fc5fb8, timer:0x7f9970f40f80, task:0x564d727182f0, task_type:"N9oceanbase5share16ObAshRefreshTaskE", scheduled_time:1755161611717613, pushed_time:1755161614757742, delay:120000000}, total_delay=541290763, threashold=1000000, tenant_id=500) [2025-08-14 09:02:37.042976] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:37.044529] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:36.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.052698] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:36.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.119434] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=51][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:37.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.124085] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.111, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.143036] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:37.144568] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:37.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.152767] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:37.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.219504] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:37.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.243097] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:37.244632] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:37.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.246745] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=2338][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:37.246776] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:02:37.246796] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:37.246812] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:37.246842] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:37.246858] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:37.246881] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:37.246897] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:01:49.731, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:01:42.751], idx=659, next_allow_print_ts="09:02:19.731") [2025-08-14 09:02:37.246907] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:37.246926] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:37.246937] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:37.246952] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:37.246969] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:37.247159] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:37.247178] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:37.252838] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:37.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.261346] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=4230] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:37.261372] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162157261342], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:37.261389] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162157261342], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:37.261407] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162157261342]}) [2025-08-14 09:02:37.261430] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:37.261442] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:37.261455] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=11] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:37.261464] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162157261454], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:37.261484] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162157261454]}) [2025-08-14 09:02:37.261669] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:37.261687] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:37.319566] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:37.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.343171] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:37.344643] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:37.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.352882] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:37.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.419599] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:37.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.443243] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:37.444708] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:37.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.452945] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:37.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.519661] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:37.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.543303] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:37.544768] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:37.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.553007] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:37.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.619723] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:37.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.643375] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:37.644817] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:37.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.686244] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=56789][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571634a0, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957163780, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:5.00s, expected_run_time:09:02:37.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.686279] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:37.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:36.716549] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [723][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=28] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=16, total_destructed_detector=16, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f99333562b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [332]OB_MALLOC COST TOO MUCH TIME, cost_time=956049, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=39, WASH_BLOCK_END=956010], seq:[0, 1, 2] [2025-08-14 09:02:35.804216] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [412][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=3670] [sql nio session](*s={this:0x7f98f733a030, session_id:3221590255, trace_id:YB42AC4D0007-00063C49651A3232-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:257}, err:0, last_decode_time:1755161952736711, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1716, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:36.758403] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=12] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:35.804216] EDIAG [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [412][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=3670][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1908787, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1908776, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=7], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a5bbf2 0x94da542 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:37.731267] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=34211] gc stale ls task succ [401]OB_MALLOC COST TOO MUCH TIME, cost_time=970537, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=3106, WASH_BLOCK_END=960572, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=6858], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:36.762796] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:951) [373][Occam][T1][YB42AC4D0007-00063C4960599C9C-0-0] [lt=962][errcode=-4012] failed to close result(close_ret=-4038, ret=-4012) [2025-08-14 09:02:35.770190] EDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16104][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1960278, owner: ObLog, click_count: 11, time dist:[ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=99, ALLOC_BLOCK#start=10098, ALLOC_BLOCK#end=99, ALLOC_BLOCK#start=10052, ALLOC_BLOCK#end=1, FORMAT_END=1939918, ObMalloc#start=0, ObMalloc#end=2, ALLOC_END=0, APPEND_END=6], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:35.777964] EDIAG [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=6443][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1952895, owner: ObLog, click_count: 9, time dist:[ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=83, ALLOC_BLOCK#start=10059, ALLOC_BLOCK#end=3, FORMAT_END=1924755, ObMalloc#start=0, ObMalloc#end=17985, ALLOC_END=2, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1d17bb52 0x1d17ada0 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [567]OB_MALLOC COST TOO MUCH TIME, cost_time=975276, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=34, WASH_BLOCK_END=956794, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=13545], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:35.770190] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16104][errcode=0] epoll handler too much time :16136 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:37.733467] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1960278] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:35.777964] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=6443] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623057204100, flush_io_finished_ret_:0, flush_io_finished_round_:26315, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:37.237451] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:37.733262] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:37.733646] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:981) [373][Occam][T1][YB42AC4D0007-00063C4960599C9C-0-0] [lt=967594][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1755161991751967), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.77.0.7:2882", reason:the only request', 'HAPPENED_TIME', '2025-08-14 08:44:54.846391', '', '', '172.77.0.7', 2882)"}, record_ret=-4012, ret=-4012) [2025-08-14 09:02:37.733994] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:763) [373][Occam][T0][YB42AC4D0007-00063C4960599C9C-0-0] [lt=344] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1755161991751967), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.77.0.7:2882", reason:the only request', 'HAPPENED_TIME', '2025-08-14 08:44:54.846391', '', '', '172.77.0.7', 2882)"}, process_time=165978850) [2025-08-14 09:02:37.734014] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1018) [373][Occam][T0][YB42AC4D0007-00063C4960599C9C-0-0] [lt=19][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1755161991751967), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.77.0.7:2882", reason:the only request', 'HAPPENED_TIME', '2025-08-14 08:44:54.846391', '', '', '172.77.0.7', 2882)"}, aret=-4012, ret=-4012) [2025-08-14 09:02:37.734034] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1590) [373][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1755161991751967), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.77.0.7:2882", reason:the only request', 'HAPPENED_TIME', '2025-08-14 08:44:54.846391', '', '', '172.77.0.7', 2882)) [2025-08-14 09:02:36.757611] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=440] get wrs ts(ls_id={id:1}, delta=609016857, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:37.734406] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=975315] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:37.734500] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=970431][errcode=0] epoll handler too much time :971912 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:35.748586] EDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1983962, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1983954, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0xfc0e73d 0x993f560 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:37.653068] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:37.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:35.748586] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:8607757, max_lsn:1604090331, max_scn:{val:1755161756730847001, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:1556737504}, reuse_lsn:{lsn:1604090331}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:1562147291}}, last_submit_lsn:{lsn:1604090203}, last_submit_end_lsn:{lsn:1604090331}, last_submit_log_id:8607757, last_submit_log_pid:50, max_flushed_lsn:{lsn:1604090203}, max_flushed_end_lsn:{lsn:1604090331}, max_flushed_log_pid:50, committed_end_lsn:{lsn:1604090090}, last_slide_log_id:8607755, last_slide_scn:{val:1755161749741193000, v:0}, last_slide_lsn:{lsn:1604089968}, last_slide_end_lsn:{lsn:1604090090}, last_slide_log_pid:50, last_slide_log_accum_checksum:1446576759, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f990bfe4530}, [StateMgr]={this:0x7f990bfe89b0, self:"172.77.0.7:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"PENDING", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:51}, leader:"0.0.0.0:0", leader_epoch:-1, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:1604090331}, scan_disk_log_finished:true, last_check_start_id:8607756, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161901828039, palf_role_change_cb:0x7f990bfef470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:50, prev:{version:1, config:{config_version:{proposal_id:49, config_seq:45}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:50, config_seq:46}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_mode_pid:50}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_end_lsn:{lsn:1604088821}, prev_mode_pid:50}, reconfig_barrier:{prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_end_lsn:{lsn:1604088821}, prev_mode_pid:50}, persistent_config_version:{proposal_id:50, config_seq:46}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755161738854289, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755161738854289, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f990bfe5c70}, [ModeMgr]={palf_id:1, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:50, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:1543409664}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:51}, log_config_meta:{version:3, proposal_id:50, prev:{version:1, config:{config_version:{proposal_id:49, config_seq:45}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:50, config_seq:46}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_mode_pid:50}, log_snapshot_meta:{version:2, base_lsn:{lsn:1543409664}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:712704}, readable_log_tail:{lsn:712704}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:1610514432}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/meta", dir_fd:225, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66392064, block_header_serialize_buf_:0x7f990bfeabb8, flashback_version:0}, log_storage:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1755161901828095, this:0x7f990bfe90f0}, [Reconfirm]={palf_id:1, self:"172.77.0.7:2882", state:"RECONFIRM_MODE_META", new_proposal_id:51, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"172.77.0.7:2882", majority_max_accept_pid:50, majority_max_lsn:{lsn:1604090331}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:1755161901828373, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f990bfe8b30}) [2025-08-14 09:02:37.734068] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [373][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] long time to hold one guard(schema_mgr=0x7f98ad7818f0, tenant_id=1, version=1753073979411448, cur_timestamp=1755162157734061, ref_timestamp=1755162005772383, lbt()="0x26a1806d 0x974f3e5 0x974e159 0x974e084 0x948f9df 0x97257c0 0x124e2fb7 0x97aa07c 0x124eac91 0x97a9944 0x261395f9 0xfa0e430 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3") [2025-08-14 09:02:37.735595] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1079) [373][Occam][T0][Y0-0000000000000000-0-0] [lt=1514][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2025-08-14 09:02:37.735608] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1543) [373][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4012] execute_write failed(ret=-4012, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1755161991751967), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.77.0.7:2882", reason:the only request', 'HAPPENED_TIME', '2025-08-14 08:44:54.846391', '', '', '172.77.0.7', 2882), is_user_sql=false) [2025-08-14 09:02:37.735620] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:159) [373][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4012] execute sql failed(ret=-4012, conn=0x7f98b26f83a0, start=1755161991752016, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1755161991751967), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.77.0.7:2882", reason:the only request', 'HAPPENED_TIME', '2025-08-14 08:44:54.846391', '', '', '172.77.0.7', 2882)) [2025-08-14 09:02:34.764198] EDIAG [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=28735][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2975439, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2975435, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b51df0f 0x986bcde 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:34.764536] EDIAG [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=17816][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2975837, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2975835, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b51df0f 0x986bcde 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 08:58:56.719844] EDIAG [STORAGE.TRANS] handle_trans_commit_request (ob_trans_service_v4.cpp:1982) [951][T1002_L0_G0][T1002][YB42AC4D0007-00063C4979499C21-0-0] [lt=944863][errcode=-4023] LOGGER COST TOO MUCH TIME, cost: 1056135, owner: ObLog, click_count: 9, time dist:[ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=98, ALLOC_BLOCK#start=10141, ALLOC_BLOCK#end=64, ALLOC_BLOCK#start=11102, ALLOC_BLOCK#end=66, FORMAT_END=1034658, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b83b27d 0x1b7ec938 0x984d055 0x9280839 0x926c154 0x12c5a0c9 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:36.727289] EDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8434][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 1018088, owner: ObLog, click_count: 9, time dist:[ALLOC_BLOCK#start=7, ALLOC_BLOCK#end=13737, ALLOC_BLOCK#start=10449, ALLOC_BLOCK#end=87, ALLOC_BLOCK#start=10188, ALLOC_BLOCK#end=17, FORMAT_END=983602, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1b53d414 0x97ad9aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [385]OB_MALLOC COST TOO MUCH TIME, cost_time=988461, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=3964, WASH_BLOCK_END=984497], seq:[0, 1, 2] [729]OB_MALLOC COST TOO MUCH TIME, cost_time=984690, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=30, WASH_BLOCK_END=984660], seq:[0, 1, 2] [857]OB_MALLOC COST TOO MUCH TIME, cost_time=989389, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=3910, WASH_BLOCK_END=985479], seq:[0, 1, 2] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=980667, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=980627], seq:[0, 1, 2] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=980706, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=54, WASH_BLOCK_END=980652], seq:[0, 1, 2] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=988333, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=31, WASH_BLOCK_END=988302], seq:[0, 1, 2] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=982784, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=47, WASH_BLOCK_END=982683, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=54], seq:[0, 1, 2, 3, 4] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=977335, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=39, WASH_BLOCK_END=977296], seq:[0, 1, 2] [2025-08-14 09:02:35.750922] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2000653, owner: ObLog, click_count: 8, time dist:[ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=18, FORMAT_END=1011423, ObMalloc#start=0, ObMalloc#end=988341, ObMalloc#start=21, ObMalloc#end=54, FREE_END=792], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [392]OB_MALLOC COST TOO MUCH TIME, cost_time=992424, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=3964, WASH_BLOCK_END=988413, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=47], seq:[0, 1, 2, 3, 4] [552]OB_MALLOC COST TOO MUCH TIME, cost_time=997297, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=30, WASH_BLOCK_END=982038, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=15224], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:35.753585] EDIAG [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2006165, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1008839, ObMalloc#start=0, ObMalloc#end=997323, ALLOC_END=2, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1d189030 0x1d18285e 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:36.739158] EDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17535][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1023271, owner: ObLog, click_count: 9, time dist:[ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=133, ALLOC_BLOCK#start=10111, ALLOC_BLOCK#end=78, ALLOC_BLOCK#start=10301, ALLOC_BLOCK#end=20424, FORMAT_END=982220, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:36.761312] EDIAG [RS.COMPACTION] try_idle (ob_major_merge_scheduler.cpp:226) [1275][T1001_MergeSche][T1001][YB42AC4D0007-00063C4963E99CBE-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1001248, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1001248, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1125d502 0x1125cf81 0x1125a4c3 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:36.726499] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1036902, owner: ObLog, click_count: 5, time dist:[FORMAT_END=987844, ObMalloc#start=2, ObMalloc#end=35712, ALLOC_END=13344, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:37.743745] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=801] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:37.743772] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=286] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:37.743960] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.743997] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.744008] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891991430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ce10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:34.764198] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=28735] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136826952869001, v:0}, clog_base_lsn:{lsn:5099962368}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1755071646732636000, v:0}, all_id_meta:{id_meta:[{limited_id:1755161555795565000, latest_log_ts:{val:1755161545796642001, v:0}}, {limited_id:22000001, latest_log_ts:{val:1755135645787806003, v:0}}, {limited_id:60000001, latest_log_ts:{val:1755135646400653001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:50, palf_env_:0x7f9950bf4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-08-14 09:02:37.744270] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:37.744826] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:34.764536] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=17816] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136829134139001, v:0}, clog_base_lsn:{lsn:8186781696}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1753840867372367034, v:0}, all_id_meta:{id_meta:[{limited_id:1755161729775179000, latest_log_ts:{val:1755161719775179001, v:0}}, {limited_id:18000001, latest_log_ts:{val:1755157440846556001, v:0}}, {limited_id:60000001, latest_log_ts:{val:1755135657266496001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:46, palf_env_:0x7f99321fa030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-08-14 09:02:37.744946] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=10331][errcode=-4389] timer task use too much time(end=1755162157744945, start=1755162156753134, delta=991811) [2025-08-14 09:02:37.744967] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=761435, expected_run_ts_us_=1755162156983531, ObClockGenerator::getRealClock()=1755162157744966, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:36.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.746830] WDIAG [COORDINATOR] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [471][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] cost too much time:ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_(), (this=|threshold=1.00s|start at 09:02:37.746|total=1.89s) [2025-08-14 09:02:37.746855] WDIAG [COORDINATOR] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [471][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=22][errcode=0] cost too much time:ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_failure(), (this=|threshold=1.00s|start at 09:02:37.746|total=1.89s) [2025-08-14 09:02:37.746872] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [471][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:0, time_interval:100.00ms, expected_run_time:09:02:37.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.746887] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [471][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:02:37.746|total=1.89s) [2025-08-14 09:02:36.727289] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8434][errcode=-4389] timer task use too much time(end=1755162156727287, start=1755162156011575, delta=715712) [2025-08-14 09:02:37.746947] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1018088][errcode=0] task scheduled too delay(scheduled_delay=1737812, expected_run_ts_us_=1755162156009134, ObClockGenerator::getRealClock()=1755162157746946, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:36.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.746961] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:36.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.746983] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=0] task scheduled too delay(scheduled_delay=1657667, expected_run_ts_us_=1755162156089316, ObClockGenerator::getRealClock()=1755162157746983, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:36.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.746994] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:36.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.747006] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1542709, expected_run_ts_us_=1755162156204297, ObClockGenerator::getRealClock()=1755162157747006, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:36.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.747016] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:36.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.747035] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=991548, expected_run_ts_us_=1755162156755487, ObClockGenerator::getRealClock()=1755162157747035, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:36.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.747047] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:36.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:36.780418] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:159) [354][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=4536][errcode=-4012] execute sql failed(ret=-4012, conn=0x7f98eb1610e0, start=1755161987755805, sql=INSERT INTO __all_rootservice_event_history (gmt_create, module, event, name1, value1, name2, value2, value3, value4, value5, value6, rs_svr_ip, rs_svr_port) VALUES (usec_to_time(1755161441756996), 'server', 'load_servers', 'ret', 0, 'has_build', 1, '', '', '', '', '172.77.0.7', 2882)) [2025-08-14 09:02:37.748391] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4021] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:36.858303] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=1928234] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:02:37.687584] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571630c0, caller:mds_tenant_service.cpp:mtl_start:117, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_145, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571633a0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.748780] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=61099][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:1, time_interval:3.00s, expected_run_time:09:02:37.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.725015] INFO [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:171) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=6] handle gts request statistics(total_rt=12091, total_cnt=11, avg_rt=1.007583333333333371e+03, avg_cnt=1.100000000000000089e+00) [2025-08-14 09:02:37.750622] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:37.750645] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:02:37.750662] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:37.750671] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:37.750687] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:37.750696] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:37.750713] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:37.750722] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:01:49.731, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:01:42.751], idx=659, next_allow_print_ts="09:02:19.731") [2025-08-14 09:02:37.750732] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:37.750745] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:37.750756] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:37.750766] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:37.750776] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:37.750881] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=198] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:37.750955] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:37.751676] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=712][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:37.753443] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=26780] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:02:37.754434] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aac90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.719788] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:37.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.754456] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.759461] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=4985][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.759491] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e62b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.759503] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73940b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.760816] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=610022016, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:37.760832] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:37.760973] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [648][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9283-0-0] [lt=26][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162157744847) [2025-08-14 09:02:37.760986] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [648][T1_L0_G0][T1][YB42AC4D0007-00063C49607B9283-0-0] [lt=13][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162157744847}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162157759754}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162157759754}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:37.761470] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [316][pnio1][T1][YB42AC4D0007-00063C49607B9283-0-0] [lt=10][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1, delta=4004020, last_succ_cluster_heartbeat_tstamp_=1755162153757449) [2025-08-14 09:02:35.753585] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=31] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:37.762817] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=2006165] temporary file module use no blocks [2025-08-14 09:02:37.762826] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:37.762839] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:37.762854] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623057183752, flush_io_finished_ret_:0, flush_io_finished_round_:26327, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:37.763334] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=4007] get_max_applied_scn(scn={val:1755161749741193000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:50, palf_committed_end_lsn_:{lsn:1604090090}, palf_committed_end_scn_:{val:1755161749741193000, v:0}, last_check_scn_:{val:1755161749741193001, v:0}, max_applied_cb_scn_:{val:1755161749741193000, v:0}}) [2025-08-14 09:02:37.763356] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get_min_unreplayed_log_info(lsn={lsn:1604088821}, scn={val:1755100948296325001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873479, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:12492179, ret_code_:0}}, next_to_submit_lsn_:{lsn:1604088821}, next_to_submit_scn_:{val:1755100948296325001, v:0}, base_lsn_:{lsn:1409200128}, base_scn_:{val:1753840802540914000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:1604088821}, end_lsn:{lsn:1604088821}, read_buf:{buf_len_:3747840, buf_:0x7f98f6a05000}, block_size:67104768, log_storage_:0x7f990bfebbf0, log_storage_:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:42, read_io_size:147709952, read_disk_cost_ts:1061450, start_lsn:{lsn:1409200128}}}}}}) [2025-08-14 09:02:37.763426] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=68] get wrs ts(ls_id={id:1}, delta=408020537, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:37.763436] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:37.763726] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162157759754}, version_delta=610025209) [2025-08-14 09:02:37.763958] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=413033906, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:36.739158] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17535][errcode=0] epoll handler too much time :17582 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:37.763971] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:36.754799] EDIAG [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=965496][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1024955, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1024953, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1fce4e91 0x1fce455d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:37.764014] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1023271] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:36.770310] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=11] gc stale ls task succ [2025-08-14 09:02:37.764694] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=610023214, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:37.764713] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:37.715751] WDIAG [DETECT] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [723][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=997689][errcode=0] cost too much time:ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t), (this=|threshold=100.00ms|start at 09:02:37.764|total=1.98s) [2025-08-14 09:02:36.726499] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:37.764996] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1036902][errcode=0] epoll handler too much time :1038507 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:37.765013] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:37.728288] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=256] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:37.765063] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=36716] temporary file module use no blocks [2025-08-14 09:02:37.765070] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:37.765078] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=8] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:37.765087] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=8] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623059168030, flush_io_finished_ret_:0, flush_io_finished_round_:26295, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:37.765140] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eeb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.765158] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73936b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eec90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.765171] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.765952] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:37.767216] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:37.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.767250] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:37.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.770557] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=992228] gc stale ls task succ [2025-08-14 09:02:37.729091] WDIAG print_alloc_failed_msg (alloc_failed_reason.cpp:146) [341][LSSysTblUp0][T0][YB42AC4D0007-00063C495B49B120-0-0] [lt=183720188][errcode=-4013] [OOPS]: alloc failed reason is that server memory has reached the upper limit(server_hold: 4292911104, server_limit: 4294967296, alloc_size: 2097152). detailed info: tenant_id=500, ctx_id=16, ctx_name=RPC_CTX_ID, ctx_hold=2859008, ctx_limit=9223372036854775807, tenant_hold=697917440, tenant_limit=9223372036854775807, backtrace=0x26a1806d 0xf98a0db 0xf989b06 0x9327c08 0x9315ac5 0x27aa2827 0x27a872f9 0x20e143d0 0x1eed36b5 0x1eecf030 0x1eecdd21 0x1eeb346c 0x1fcca226 0x1fcc9df1 0x211cb1a5 0x1eed018e 0x1eeb2659 0x9a05aa8 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:37.770617] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [341][LSSysTblUp0][T0][YB42AC4D0007-00063C495B49B120-0-0] [lt=41452][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_DETECT_MASTER_RS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-08-14 09:02:37.774842] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=10] [sql nio session](*s={this:0x7f98f72b3030, session_id:3221562808, trace_id:YB42AC4D0007-00063C4965DA1F2E-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:267}, err:0, last_decode_time:1755161748766816, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1890, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:37.774866] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=24] [sql nio session](*s={this:0x7f98f72ab030, session_id:3221589435, trace_id:YB42AC4D0007-00063C4965DA1F35-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:246}, err:0, last_decode_time:1755161762721798, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:2456, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:37.774877] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=10] [sql nio session](*s={this:0x7f98f72a9030, session_id:3221590528, trace_id:YB42AC4D0007-00063C4965DA1F34-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:250}, err:0, last_decode_time:1755161762715415, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:205, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:37.774886] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=9] [sql nio session](*s={this:0x7f98f72b5030, session_id:3221590961, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:10, sql_initiative_shutdown:false, reader:{fd:278}, err:5, last_decode_time:1755162075745307, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:37.776413] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.776440] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72765b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.776456] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.776466] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.776476] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.779328] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1003][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:37.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:36.754799] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=965496] blacklist_loop exec finished(cost_time=17, is_enabled=true, send_cnt=0) [2025-08-14 09:02:37.781388] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=2975439] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:37.781415] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=14] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:37.767295] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1983962] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161974728707, last_flushed_end_lsn:{lsn:1604090331}}]) [2025-08-14 09:02:37.781451] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14126] current log_task status(palf_id=1, self="172.77.0.7:2882", log_id=8607756, log_task={header:{begin_lsn:{lsn:1604090090}, end_lsn:{lsn:1604090203}, log_id:8607756, min_scn:{val:1755161749741193001, v:0}, max_scn:{val:1755161749741193001, v:0}, data_len:57, proposal_id:50, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:1604090090}, data_checksum:2031283194, accum_checksum:759186326, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1755161750716165, freeze_ts:1755161750716173, submit_ts:1755161750716183, flushed_ts:1755161853765770, gen_to_freeze cost time:8, gen_to_submit cost time:18, submit_to_flush cost time:103049587}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161974728707, last_flushed_end_lsn:{lsn:1604090331}}]) [2025-08-14 09:02:37.769872] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=804] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:37.781651] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11759][errcode=0] epoll handler too much time :11787 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:37.781990] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=2975837] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:37.782016] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=18] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:37.761761] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2978] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:37.782314] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20515] [IO STATUS SERVER] net_in=0kB/s, net_out=0kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=1280000kB/s [2025-08-14 09:02:37.779803] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:37.764753] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=10882][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fb3b0, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fb690, time_wheel:0x7f99589fd990, thread_pool:0x7f9959fff170, is_running:1, time_interval:2.00s, expected_run_time:09:02:37.698, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.754763] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=997066] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119797572, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755162138745740) [2025-08-14 09:02:37.786076] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=30693] get wrs ts(ls_id={id:1}, delta=438010512, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:37.786094] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=2, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:37.786670] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=438010512, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:37.786680] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:37.787018] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aab10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.774, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.787051] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739ef90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.787066] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a45b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.787080] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eee10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.787093] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73931b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.760157] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4452] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:37.791644] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=31438] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3933) [2025-08-14 09:02:37.791654] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:37.795253] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=23] gc stale ls task succ [2025-08-14 09:02:37.797165] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.797203] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.797220] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7273c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.797244] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d27b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.797263] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d94b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.797282] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.780, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.797300] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739bbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.780, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.797310] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fa10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.797320] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f727dc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f729ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.797346] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f727d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f729af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.797364] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.800410] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:37.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.803540] INFO handle (memory_dump.cpp:671) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=19] statistics: tenant_cnt: 6, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 168 270 0 1 2 110 1 0 1 3 7 0 0 1 4 1 1 0 1 5 86 6 0 1 6 13 0 0 1 7 22 2 0 1 8 53 1 0 1 12 17 3 0 1 13 10 1 0 1 14 1 1 0 1 15 1 1 0 1 16 6 8 0 1 18 4 3 0 1 20 2 1 0 1 22 1 1 0 1 24 1 0 0 1 25 6 0 0 500 0 555 275 0 500 7 194 7 0 500 8 41 1 0 500 9 2 1 0 500 10 9 2 0 500 16 52 4 0 500 17 54 7 0 500 22 38 64 0 500 23 27 10 0 508 0 2 7 0 508 8 5 1 0 509 0 2 7 0 509 8 5 1 0 1001 0 147 253 0 1001 2 26 1 0 1001 3 5 0 0 1001 4 1 1 0 1001 5 85 6 0 1001 6 11 0 0 1001 7 30 2 0 1001 8 36 1 0 1001 12 17 3 0 1001 13 3 1 0 1001 14 1 1 0 1001 16 6 3 0 1001 18 4 3 0 1001 20 4 1 0 1001 25 8 0 0 1002 0 263 278 0 1002 2 198 1 0 1002 3 4 0 0 1002 4 1 1 0 1002 5 101 6 0 1002 6 14 0 0 1002 7 6 2 0 1002 8 73 1 0 1002 12 18 3 0 1002 13 7 1 0 1002 14 1 2 0 1002 16 8 8 0 1002 18 9 3 0 1002 20 2 1 0 1002 21 1 0 0 1002 24 1 0 0 1002 25 7 0 0 cost_time: 2025949 [2025-08-14 09:02:37.803583] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=21] [MEMORY] tenant: 1, limit: 2147483648 hold: 1011077120 cache_hold: 56623104 cache_used: 56623104 cache_item_count: 27 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 444993536 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 230686720 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 1163264 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 483328 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 53714944 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 1458176 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 14204928 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 111149056 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35651584 limit= 429496720 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 8003584 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= PS_CACHE_CTX_ID hold_bytes= 65536 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 10633216 limit= 9223372036854775807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 3481600 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3964928 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 212992 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 884736 limit= 9223372036854775807 [2025-08-14 09:02:37.803896] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=15] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 444993536 used= 389625696 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 70 washed_blocks= 287 washed_size= 70221824 [MEMORY] request_cached_chunk_cnt= 4 [MEMORY] hold= 41963520 used= 41943072 count= 1 avg_used= 41943072 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 41963520 used= 41943040 count= 1 avg_used= 41943040 block_cnt= 1 chunk_cnt= 1 mod=LogGroupBuffer [MEMORY] hold= 29130752 used= 29116416 count= 14 avg_used= 2079744 block_cnt= 14 chunk_cnt= 14 mod=CACHE_MAP_NODE [MEMORY] hold= 25640960 used= 25394048 count= 37 avg_used= 686325 block_cnt= 37 chunk_cnt= 27 mod=MysqlRequesReco [MEMORY] hold= 20615168 used= 20545536 count= 4 avg_used= 5136384 block_cnt= 4 chunk_cnt= 4 mod=ServerObjecPool [MEMORY] hold= 16908288 used= 16777728 count= 16 avg_used= 1048608 block_cnt= 16 chunk_cnt= 16 mod=DiagnosticInfos [MEMORY] hold= 16797696 used= 16777336 count= 1 avg_used= 16777336 block_cnt= 1 chunk_cnt= 1 mod=MrgMemPoolChk [MEMORY] hold= 16083712 used= 15926225 count= 112 avg_used= 142198 block_cnt= 38 chunk_cnt= 9 mod=OmtTenant [MEMORY] hold= 10510336 used= 10491392 count= 1 avg_used= 10491392 block_cnt= 1 chunk_cnt= 1 mod=ASHListBuffer [MEMORY] hold= 9457664 used= 9437784 count= 1 avg_used= 9437784 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 8777728 used= 8760064 count= 2 avg_used= 4380032 block_cnt= 2 chunk_cnt= 2 mod=FreeTbltStream [MEMORY] hold= 8540160 used= 8519680 count= 1 avg_used= 8519680 block_cnt= 1 chunk_cnt= 1 mod=ArcFetchQueue [MEMORY] hold= 8540160 used= 8519680 count= 1 avg_used= 8519680 block_cnt= 1 chunk_cnt= 1 mod=RCSrv [MEMORY] hold= 6856704 used= 6788160 count= 9 avg_used= 754240 block_cnt= 9 chunk_cnt= 9 mod=ResourceGroup [MEMORY] hold= 6365184 used= 5636096 count= 89 avg_used= 63326 block_cnt= 89 chunk_cnt= 30 mod=[T]char [MEMORY] hold= 5505024 used= 4822272 count= 168 avg_used= 28704 block_cnt= 168 chunk_cnt= 31 mod=DiagnosticInfo [MEMORY] hold= 5386240 used= 5368320 count= 1 avg_used= 5368320 block_cnt= 1 chunk_cnt= 1 mod=RespTimeColl [MEMORY] hold= 4943872 used= 4915456 count= 2 avg_used= 2457728 block_cnt= 2 chunk_cnt= 2 mod=HashBuckDmId [MEMORY] hold= 4739072 used= 4718712 count= 1 avg_used= 4718712 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmChe [MEMORY] hold= 4472832 used= 2529072 count= 546 avg_used= 4632 block_cnt= 132 chunk_cnt= 55 mod=WorkerMap [MEMORY] hold= 4235264 used= 4194304 count= 2 avg_used= 2097152 block_cnt= 2 chunk_cnt= 2 mod=SHARE_BLOCK [MEMORY] hold= 3865600 used= 3705600 count= 800 avg_used= 4632 block_cnt= 800 chunk_cnt= 13 mod=CkptDgnMem [MEMORY] hold= 3865600 used= 3705600 count= 800 avg_used= 4632 block_cnt= 800 chunk_cnt= 13 mod=CkptDgnMemCU [MEMORY] hold= 3772416 used= 3751944 count= 1 avg_used= 3751944 block_cnt= 1 chunk_cnt= 1 mod=IteratorStorage [MEMORY] hold= 3723264 used= 3698704 count= 2 avg_used= 1849352 block_cnt= 2 chunk_cnt= 2 mod=LogDIOAligned [MEMORY] hold= 3162112 used= 3145808 count= 2 avg_used= 1572904 block_cnt= 2 chunk_cnt= 2 mod=TenantDLMgr [MEMORY] hold= 3145728 used= 2228224 count= 128 avg_used= 17408 block_cnt= 128 chunk_cnt= 6 mod=SqlDtlQueue [MEMORY] hold= 3047424 used= 3013468 count= 10 avg_used= 301346 block_cnt= 10 chunk_cnt= 2 mod=LDIOSetup [MEMORY] hold= 3031040 used= 3013952 count= 1 avg_used= 3013952 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2850816 used= 2832384 count= 12 avg_used= 236032 block_cnt= 12 chunk_cnt= 10 mod=DiagnosticSum [MEMORY] hold= 2379776 used= 2359608 count= 1 avg_used= 2359608 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMonMap [MEMORY] hold= 2379776 used= 2359608 count= 1 avg_used= 2359608 block_cnt= 1 chunk_cnt= 1 mod=FastFrezCkr [MEMORY] hold= 2379776 used= 2359608 count= 1 avg_used= 2359608 block_cnt= 1 chunk_cnt= 1 mod=MediumTabletMap [MEMORY] hold= 2375680 used= 2359536 count= 2 avg_used= 1179768 block_cnt= 2 chunk_cnt= 2 mod=HashBuckLCSta [MEMORY] hold= 2248704 used= 2228224 count= 1 avg_used= 2228224 block_cnt= 1 chunk_cnt= 1 mod=LogIOCb [MEMORY] hold= 2198720 used= 416700 count= 25277 avg_used= 16 block_cnt= 272 chunk_cnt= 3 mod=Number [MEMORY] hold= 2197056 used= 2174312 count= 77 avg_used= 28237 block_cnt= 77 chunk_cnt= 2 mod=TabletMap [MEMORY] hold= 2064384 used= 1835008 count= 28 avg_used= 65536 block_cnt= 28 chunk_cnt= 11 mod=MajorMergeMgr [MEMORY] hold= 1843200 used= 1770696 count= 9 avg_used= 196744 block_cnt= 9 chunk_cnt= 5 mod=DagMap [MEMORY] hold= 1374912 used= 1370880 count= 21 avg_used= 65280 block_cnt= 21 chunk_cnt= 10 mod=ResultSet [MEMORY] hold= 1294336 used= 1280384 count= 2 avg_used= 640192 block_cnt= 2 chunk_cnt= 2 mod=TransTimeWheel [MEMORY] hold= 1294336 used= 1280384 count= 2 avg_used= 640192 block_cnt= 2 chunk_cnt= 1 mod=XATimeWheel [MEMORY] hold= 1187840 used= 1179768 count= 1 avg_used= 1179768 block_cnt= 1 chunk_cnt= 1 mod=RewriteRuleMap [MEMORY] hold= 1187840 used= 1179768 count= 1 avg_used= 1179768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsCache [MEMORY] hold= 1187840 used= 1179768 count= 1 avg_used= 1179768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPlanCac [MEMORY] hold= 1187840 used= 1179768 count= 1 avg_used= 1179768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsInfo [MEMORY] hold= 1130496 used= 1126688 count= 6 avg_used= 187781 block_cnt= 6 chunk_cnt= 6 mod=IoControl [MEMORY] hold= 1114112 used= 1079296 count= 131 avg_used= 8238 block_cnt= 80 chunk_cnt= 44 mod=ParamCharsType [MEMORY] hold= 1015808 used= 983040 count= 4 avg_used= 245760 block_cnt= 4 chunk_cnt= 3 mod=SqlExecutor [MEMORY] hold= 958464 used= 950272 count= 1 avg_used= 950272 block_cnt= 1 chunk_cnt= 1 mod=IOWorkerLQ [MEMORY] hold= 933888 used= 931072 count= 1 avg_used= 931072 block_cnt= 1 chunk_cnt= 1 mod=TransService [MEMORY] hold= 933888 used= 931072 count= 1 avg_used= 931072 block_cnt= 1 chunk_cnt= 1 mod=ArcSenderQueue [MEMORY] hold= 884736 used= 856896 count= 18 avg_used= 47605 block_cnt= 12 chunk_cnt= 9 mod=CommSysVarFac [MEMORY] hold= 829632 used= 821248 count= 2 avg_used= 410624 block_cnt= 2 chunk_cnt= 2 mod=LSMap [MEMORY] hold= 802816 used= 800000 count= 1 avg_used= 800000 block_cnt= 1 chunk_cnt= 1 mod=SqlFltSpanRec [MEMORY] hold= 798112 used= 789624 count= 3 avg_used= 263208 block_cnt= 2 chunk_cnt= 2 mod=PartitTableTask [MEMORY] hold= 786688 used= 524352 count= 33 avg_used= 15889 block_cnt= 33 chunk_cnt= 4 mod=LogAlloc [MEMORY] hold= 786432 used= 386304 count= 48 avg_used= 8048 block_cnt= 48 chunk_cnt= 20 mod=[T]ObPerfEventR [MEMORY] hold= 679872 used= 658688 count= 83 avg_used= 7936 block_cnt= 10 chunk_cnt= 7 mod=TltMemtablMgr [MEMORY] hold= 671744 used= 663552 count= 1 avg_used= 663552 block_cnt= 1 chunk_cnt= 1 mod=ApplySrv [MEMORY] hold= 666880 used= 404416 count= 89 avg_used= 4544 block_cnt= 66 chunk_cnt= 32 mod=[T]ObTraceEvent [MEMORY] hold= 663552 used= 659200 count= 1 avg_used= 659200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 663552 used= 655360 count= 1 avg_used= 655360 block_cnt= 1 chunk_cnt= 1 mod=FetchLog [MEMORY] hold= 655360 used= 626816 count= 4 avg_used= 156704 block_cnt= 4 chunk_cnt= 4 mod=SQLSessionInfo [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=OBJLockGC [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=MultiVersionGC [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=MdsT [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=DupTbLease [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=CoordTR [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=DetectorTimer [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=CoordTF [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=FrzTrigger [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=ElectTimer [MEMORY] hold= 598016 used= 590232 count= 1 avg_used= 590232 block_cnt= 1 chunk_cnt= 1 mod=DagNetIdMap [MEMORY] hold= 516096 used= 507904 count= 1 avg_used= 507904 block_cnt= 1 chunk_cnt= 1 mod=CommitVersions [MEMORY] hold= 483328 used= 477216 count= 1 avg_used= 477216 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMon [MEMORY] hold= 475136 used= 458816 count= 2 avg_used= 229408 block_cnt= 2 chunk_cnt= 2 mod=ts_token [MEMORY] hold= 442368 used= 393216 count= 6 avg_used= 65536 block_cnt= 6 chunk_cnt= 6 mod=MinorMergeMgr [MEMORY] hold= 409600 used= 401408 count= 1 avg_used= 401408 block_cnt= 1 chunk_cnt= 1 mod=ReplaySrv [MEMORY] hold= 401408 used= 393504 count= 1 avg_used= 393504 block_cnt= 1 chunk_cnt= 1 mod=ClogGe [MEMORY] hold= 385024 used= 375520 count= 2 avg_used= 187760 block_cnt= 2 chunk_cnt= 2 mod=bf_queue [MEMORY] hold= 376832 used= 223376 count= 23 avg_used= 9712 block_cnt= 23 chunk_cnt= 17 mod=[T]ObDSActionAr [MEMORY] hold= 311104 used= 303904 count= 6 avg_used= 50650 block_cnt= 6 chunk_cnt= 3 mod=PoolFreeList [MEMORY] hold= 303104 used= 294936 count= 1 avg_used= 294936 block_cnt= 1 chunk_cnt= 1 mod=DmlStatHashMap [MEMORY] hold= 303104 used= 294936 count= 1 avg_used= 294936 block_cnt= 1 chunk_cnt= 1 mod=ColUsagHashMap [MEMORY] hold= 303104 used= 294936 count= 1 avg_used= 294936 block_cnt= 1 chunk_cnt= 1 mod=RSCompStMap [MEMORY] hold= 303104 used= 294936 count= 1 avg_used= 294936 block_cnt= 1 chunk_cnt= 1 mod=RSCompactMap [MEMORY] hold= 286720 used= 280192 count= 1 avg_used= 280192 block_cnt= 1 chunk_cnt= 1 mod=MetaMemMgr [MEMORY] hold= 278528 used= 270468 count= 2 avg_used= 135234 block_cnt= 2 chunk_cnt= 2 mod=CsSstableReader [MEMORY] hold= 247120 used= 3318 count= 3061 avg_used= 1 block_cnt= 31 chunk_cnt= 1 mod=CharsetUtil [MEMORY] hold= 212992 used= 211072 count= 1 avg_used= 211072 block_cnt= 1 chunk_cnt= 1 mod=TimerWK [MEMORY] hold= 204800 used= 196616 count= 1 avg_used= 196616 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 204800 used= 196616 count= 1 avg_used= 196616 block_cnt= 1 chunk_cnt= 1 mod=T3MBucket [MEMORY] hold= 204800 used= 196744 count= 1 avg_used= 196744 block_cnt= 1 chunk_cnt= 1 mod=DagNetMap [MEMORY] hold= 204800 used= 196616 count= 1 avg_used= 196616 block_cnt= 1 chunk_cnt= 1 mod=TenDLBucket [MEMORY] hold= 155648 used= 150784 count= 19 avg_used= 7936 block_cnt= 19 chunk_cnt= 15 mod=LinearHashMapNo [MEMORY] hold= 147456 used= 139264 count= 1 avg_used= 139264 block_cnt= 1 chunk_cnt= 1 mod=RFLTaskQueue [MEMORY] hold= 139264 used= 134912 count= 17 avg_used= 7936 block_cnt= 10 chunk_cnt= 7 mod=SqlSession [MEMORY] hold= 131072 used= 130560 count= 2 avg_used= 65280 block_cnt= 2 chunk_cnt= 2 mod=MemTblObj [MEMORY] hold= 122880 used= 110608 count= 2 avg_used= 55304 block_cnt= 2 chunk_cnt= 2 mod=TabletIDSetBkt [MEMORY] hold= 114688 used= 111096 count= 1 avg_used= 111096 block_cnt= 1 chunk_cnt= 1 mod=DbmsSched_Job [MEMORY] hold= 114688 used= 111096 count= 1 avg_used= 111096 block_cnt= 1 chunk_cnt= 1 mod=AsyncFreezeTabl [MEMORY] hold= 114688 used= 111104 count= 14 avg_used= 7936 block_cnt= 13 chunk_cnt= 10 mod=QueryAllocator [MEMORY] hold= 114688 used= 106496 count= 1 avg_used= 106496 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlMgr [MEMORY] hold= 102512 used= 27016 count= 370 avg_used= 73 block_cnt= 74 chunk_cnt= 31 mod=Coro [MEMORY] hold= 98304 used= 95520 count= 1 avg_used= 95520 block_cnt= 1 chunk_cnt= 1 mod=LSSvr [MEMORY] hold= 98304 used= 91968 count= 1 avg_used= 91968 block_cnt= 1 chunk_cnt= 1 mod=tenant_mf_mgr [MEMORY] hold= 88000 used= 85888 count= 11 avg_used= 7808 block_cnt= 11 chunk_cnt= 1 mod=TabletIDSetNode [MEMORY] hold= 81920 used= 73896 count= 1 avg_used= 73896 block_cnt= 1 chunk_cnt= 1 mod=RSCompPairCache [MEMORY] hold= 81824 used= 73472 count= 2 avg_used= 36736 block_cnt= 2 chunk_cnt= 2 mod=ts_run_set [MEMORY] hold= 81824 used= 73472 count= 2 avg_used= 36736 block_cnt= 2 chunk_cnt= 2 mod=ts_cancel_set [MEMORY] hold= 81824 used= 73472 count= 2 avg_used= 36736 block_cnt= 2 chunk_cnt= 2 mod=ts_queue [MEMORY] hold= 81536 used= 78080 count= 10 avg_used= 7808 block_cnt= 10 chunk_cnt= 8 mod=HashNodeLCSta [MEMORY] hold= 73728 used= 72736 count= 1 avg_used= 72736 block_cnt= 1 chunk_cnt= 1 mod=LogSharedQueueT [MEMORY] hold= 73728 used= 72736 count= 1 avg_used= 72736 block_cnt= 1 chunk_cnt= 1 mod=LogSubmitLogQue [MEMORY] hold= 73728 used= 66176 count= 1 avg_used= 66176 block_cnt= 1 chunk_cnt= 1 mod=Rpc [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=FlushLog [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=LockMemObj [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=TxDataMemObj [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=TxCtxMemObj [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=DEVICE_MANAGER [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=StorMetaCacheIO [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=FlushMeta [MEMORY] hold= 57344 used= 51328 count= 1 avg_used= 51328 block_cnt= 1 chunk_cnt= 1 mod=palf_env [MEMORY] hold= 56448 used= 48384 count= 14 avg_used= 3456 block_cnt= 11 chunk_cnt= 9 mod=CACHE_INST [MEMORY] hold= 52672 used= 34944 count= 52 avg_used= 672 block_cnt= 36 chunk_cnt= 17 mod=[T]ObWarningBuf [MEMORY] hold= 49152 used= 45184 count= 1 avg_used= 45184 block_cnt= 1 chunk_cnt= 1 mod=IO_MGR [MEMORY] hold= 49152 used= 47616 count= 6 avg_used= 7936 block_cnt= 6 chunk_cnt= 5 mod=SqlSessiVarMap [MEMORY] hold= 49152 used= 47616 count= 6 avg_used= 7936 block_cnt= 6 chunk_cnt= 5 mod=PlanVaIdx [MEMORY] hold= 49152 used= 47616 count= 2 avg_used= 23808 block_cnt= 2 chunk_cnt= 1 mod=IOUsageInfo [MEMORY] hold= 48960 used= 44840 count= 2 avg_used= 22420 block_cnt= 2 chunk_cnt= 2 mod=DynamicFactor [MEMORY] hold= 48112 used= 5016 count= 209 avg_used= 24 block_cnt= 74 chunk_cnt= 29 mod=[T]MemoryContex [MEMORY] hold= 46400 used= 32160 count= 67 avg_used= 480 block_cnt= 25 chunk_cnt= 8 mod=OMT_Worker [MEMORY] hold= 42096 used= 28792 count= 59 avg_used= 488 block_cnt= 30 chunk_cnt= 12 mod=[T]ObSQLMockedT [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CTBriefMap [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletPtrMap [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CliTaskMap [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=DagWarnHisBkt [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=TLD_TblCtxMap [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoBkt [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=Autoincrement [MEMORY] hold= 40960 used= 39680 count= 5 avg_used= 7936 block_cnt= 5 chunk_cnt= 5 mod=LSIter [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=HB_SERVICE [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=HTblRkMapBkt [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=HashBucCliInFo [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=TableAQueryBkt [MEMORY] hold= 40896 used= 39680 count= 5 avg_used= 7936 block_cnt= 5 chunk_cnt= 5 mod=MdsTableMgr [MEMORY] hold= 40768 used= 39040 count= 5 avg_used= 7808 block_cnt= 5 chunk_cnt= 5 mod=HashNodePlanCac [MEMORY] hold= 40640 used= 39680 count= 5 avg_used= 7936 block_cnt= 5 chunk_cnt= 1 mod=ObDMMDL [MEMORY] hold= 36096 used= 33216 count= 3 avg_used= 11072 block_cnt= 3 chunk_cnt= 3 mod=DI_CONTAINER [MEMORY] hold= 32768 used= 24688 count= 2 avg_used= 12344 block_cnt= 2 chunk_cnt= 2 mod=TLD_AssignedMgr [MEMORY] hold= 32768 used= 27960 count= 3 avg_used= 9320 block_cnt= 3 chunk_cnt= 3 mod=PlGdbHelper [MEMORY] hold= 32768 used= 15872 count= 4 avg_used= 3968 block_cnt= 4 chunk_cnt= 4 mod=CkpDgn [MEMORY] hold= 32704 used= 25191 count= 2 avg_used= 12595 block_cnt= 2 chunk_cnt= 2 mod=CGReadInfoMgr [MEMORY] hold= 24576 used= 16392 count= 1 avg_used= 16392 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 24576 used= 19776 count= 1 avg_used= 19776 block_cnt= 1 chunk_cnt= 1 mod=LogApplyStatus [MEMORY] hold= 24576 used= 20136 count= 2 avg_used= 10068 block_cnt= 2 chunk_cnt= 2 mod=TabletStats [MEMORY] hold= 17024 used= 8448 count= 3 avg_used= 2816 block_cnt= 2 chunk_cnt= 2 mod=BaseLogWriter [MEMORY] hold= 16384 used= 8448 count= 1 avg_used= 8448 block_cnt= 1 chunk_cnt= 1 mod=PalfEnv [MEMORY] hold= 16384 used= 15584 count= 2 avg_used= 7792 block_cnt= 2 chunk_cnt= 2 mod=CkptDgnCUNode [MEMORY] hold= 16384 used= 14016 count= 1 avg_used= 14016 block_cnt= 1 chunk_cnt= 1 mod=LogReplayStatus [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletMap [MEMORY] hold= 16384 used= 9792 count= 1 avg_used= 9792 block_cnt= 1 chunk_cnt= 1 mod=TSQLSessionMgr [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=MaBlkLoadMap [MEMORY] hold= 16384 used= 13896 count= 1 avg_used= 13896 block_cnt= 1 chunk_cnt= 1 mod=task_cache [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=MedCheckSet [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=backupTaskSched [MEMORY] hold= 16384 used= 8200 count= 1 avg_used= 8200 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=TLD_ResourceMgr [MEMORY] hold= 16384 used= 10248 count= 1 avg_used= 10248 block_cnt= 1 chunk_cnt= 1 mod=TabletSvrBucket [MEMORY] hold= 12864 used= 12288 count= 3 avg_used= 4096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapDi [MEMORY] hold= 12864 used= 12288 count= 3 avg_used= 4096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapCn [MEMORY] hold= 11008 used= 10624 count= 2 avg_used= 5312 block_cnt= 1 chunk_cnt= 1 mod=Scheduler [MEMORY] hold= 9456 used= 9064 count= 2 avg_used= 4532 block_cnt= 2 chunk_cnt= 2 mod=GROUP_INDEX_MAP [MEMORY] hold= 8192 used= 7792 count= 1 avg_used= 7792 block_cnt= 1 chunk_cnt= 1 mod=CkptDgnMemNode [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=RSCompPairs [MEMORY] hold= 8192 used= 7728 count= 1 avg_used= 7728 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsCache [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=GetMediumList [MEMORY] hold= 8192 used= 7712 count= 1 avg_used= 7712 block_cnt= 1 chunk_cnt= 1 mod=DmlStatNode [MEMORY] hold= 8192 used= 7792 count= 1 avg_used= 7792 block_cnt= 1 chunk_cnt= 1 mod=ColUsagNode [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=GetKeptShotInfo [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=FillCache [MEMORY] hold= 8192 used= 7632 count= 1 avg_used= 7632 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoNode [MEMORY] hold= 8192 used= 7352 count= 1 avg_used= 7352 block_cnt= 1 chunk_cnt= 1 mod=mrg_prog_cker [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=MediumList [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=HandleSubmit [MEMORY] hold= 8192 used= 7632 count= 1 avg_used= 7632 block_cnt= 1 chunk_cnt= 1 mod=CheckMapNode [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=LsElectRefInfo [MEMORY] hold= 8192 used= 7808 count= 1 avg_used= 7808 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsInfo [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=ls_id [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=LCLSender [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=HTableLockMap [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=DASIDCache [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=CPUUSage [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=MatchLsnMap [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=APPLY_STATUS [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=leakChecker [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=REPLAY_STATUS [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=ShareBlksMap [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=MdsDebugMap [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileMap [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=LSInfo [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=PurgeThrottle [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileBlkMgr [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=LockWaitMgr [MEMORY] hold= 6880 used= 5600 count= 7 avg_used= 800 block_cnt= 7 chunk_cnt= 6 mod=DICollector [MEMORY] hold= 5216 used= 1976 count= 13 avg_used= 152 block_cnt= 2 chunk_cnt= 2 mod=ObTxLSLogCb [MEMORY] hold= 5056 used= 4656 count= 2 avg_used= 2328 block_cnt= 2 chunk_cnt= 2 mod=MViewService [MEMORY] hold= 4832 used= 4632 count= 1 avg_used= 4632 block_cnt= 1 chunk_cnt= 1 mod=ExTabletCntMap [MEMORY] hold= 4832 used= 4632 count= 1 avg_used= 4632 block_cnt= 1 chunk_cnt= 1 mod=PxPoolBkt [MEMORY] hold= 4832 used= 4632 count= 1 avg_used= 4632 block_cnt= 1 chunk_cnt= 1 mod=ResRuleIdMap [MEMORY] hold= 4832 used= 4632 count= 1 avg_used= 4632 block_cnt= 1 chunk_cnt= 1 mod=ResGrpIdMap [MEMORY] hold= 4608 used= 2544 count= 6 avg_used= 424 block_cnt= 6 chunk_cnt= 5 mod=SequenceIdMap [MEMORY] hold= 4288 used= 4096 count= 1 avg_used= 4096 block_cnt= 1 chunk_cnt= 1 mod=MacroFile [MEMORY] hold= 4224 used= 3072 count= 6 avg_used= 512 block_cnt= 5 chunk_cnt= 4 mod=SqlSessiQuerSql [MEMORY] hold= 4160 used= 3520 count= 4 avg_used= 880 block_cnt= 3 chunk_cnt= 3 mod=PoolArenaArray [MEMORY] hold= 4096 used= 2328 count= 1 avg_used= 2328 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoMap [MEMORY] hold= 3840 used= 2544 count= 6 avg_used= 424 block_cnt= 6 chunk_cnt= 5 mod=ContextsMap [MEMORY] hold= 3840 used= 2544 count= 6 avg_used= 424 block_cnt= 6 chunk_cnt= 5 mod=SequenceMap [MEMORY] hold= 3840 used= 2544 count= 6 avg_used= 424 block_cnt= 6 chunk_cnt= 5 mod=PackStateMap [MEMORY] hold= 3328 used= 1728 count= 4 avg_used= 432 block_cnt= 4 chunk_cnt= 4 mod=NegParamIdex [MEMORY] hold= 3328 used= 1792 count= 8 avg_used= 224 block_cnt= 1 chunk_cnt= 1 mod=LogIOTask [MEMORY] hold= 3312 used= 3112 count= 1 avg_used= 3112 block_cnt= 1 chunk_cnt= 1 mod=TFileFLCtx [MEMORY] hold= 3312 used= 3112 count= 1 avg_used= 3112 block_cnt= 1 chunk_cnt= 1 mod=PendingFreeMap [MEMORY] hold= 3200 used= 3008 count= 1 avg_used= 3008 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlDfc [MEMORY] hold= 2528 used= 2328 count= 1 avg_used= 2328 block_cnt= 1 chunk_cnt= 1 mod=IRMMemHashBuck [MEMORY] hold= 2528 used= 2328 count= 1 avg_used= 2328 block_cnt= 1 chunk_cnt= 1 mod=VecIdxLSMgr [MEMORY] hold= 2528 used= 2328 count= 1 avg_used= 2328 block_cnt= 1 chunk_cnt= 1 mod=DASMemHashBuck [MEMORY] hold= 2528 used= 2328 count= 1 avg_used= 2328 block_cnt= 1 chunk_cnt= 1 mod=MedCheckMap [MEMORY] hold= 2240 used= 2048 count= 1 avg_used= 2048 block_cnt= 1 chunk_cnt= 1 mod=CreateContext [MEMORY] hold= 2048 used= 1272 count= 1 avg_used= 1272 block_cnt= 1 chunk_cnt= 1 mod=LSCheckMap [MEMORY] hold= 2048 used= 1856 count= 1 avg_used= 1856 block_cnt= 1 chunk_cnt= 1 mod=LogIOWS [MEMORY] hold= 2016 used= 1824 count= 1 avg_used= 1824 block_cnt= 1 chunk_cnt= 1 mod=PxResMgr [MEMORY] hold= 1920 used= 1728 count= 1 avg_used= 1728 block_cnt= 1 chunk_cnt= 1 mod=timer_service [MEMORY] hold= 1808 used= 1424 count= 2 avg_used= 712 block_cnt= 2 chunk_cnt= 2 mod=DetectManager [MEMORY] hold= 1744 used= 1544 count= 1 avg_used= 1544 block_cnt= 1 chunk_cnt= 1 mod=TXBFRetryId [MEMORY] hold= 1744 used= 1544 count= 1 avg_used= 1544 block_cnt= 1 chunk_cnt= 1 mod=TabStatMgrLock [MEMORY] hold= 1472 used= 1272 count= 1 avg_used= 1272 block_cnt= 1 chunk_cnt= 1 mod=DUP_LS_SET [MEMORY] hold= 1472 used= 1272 count= 1 avg_used= 1272 block_cnt= 1 chunk_cnt= 1 mod=HashBucApiGroup [MEMORY] hold= 1472 used= 1272 count= 1 avg_used= 1272 block_cnt= 1 chunk_cnt= 1 mod=GCMemtableMap [MEMORY] hold= 1280 used= 1088 count= 1 avg_used= 1088 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 1248 used= 1056 count= 1 avg_used= 1056 block_cnt= 1 chunk_cnt= 1 mod=LOG_HASH_MAP [MEMORY] hold= 1232 used= 1032 count= 1 avg_used= 1032 block_cnt= 1 chunk_cnt= 1 mod=BFLoadTask [MEMORY] hold= 1200 used= 1000 count= 1 avg_used= 1000 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletMapLk [MEMORY] hold= 1120 used= 120 count= 5 avg_used= 24 block_cnt= 2 chunk_cnt= 2 mod=FreezeTask [MEMORY] hold= 1024 used= 832 count= 1 avg_used= 832 block_cnt= 1 chunk_cnt= 1 mod=tenant_rm_mgr [MEMORY] hold= 976 used= 776 count= 1 avg_used= 776 block_cnt= 1 chunk_cnt= 1 mod=DepTabletSet [MEMORY] hold= 896 used= 696 count= 1 avg_used= 696 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfo [MEMORY] hold= 752 used= 552 count= 1 avg_used= 552 block_cnt= 1 chunk_cnt= 1 mod=LSFreeze [MEMORY] hold= 704 used= 208 count= 1 avg_used= 208 block_cnt= 1 chunk_cnt= 1 mod=[T]ObMacroBlock [MEMORY] hold= 672 used= 472 count= 1 avg_used= 472 block_cnt= 1 chunk_cnt= 1 mod=Election [MEMORY] hold= 640 used= 224 count= 2 avg_used= 112 block_cnt= 1 chunk_cnt= 1 mod=Coordinator [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=MSTXCTX [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=HAScheduler [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=VectorIndexVsag [MEMORY] hold= 544 used= 144 count= 2 avg_used= 72 block_cnt= 2 chunk_cnt= 2 mod=TntSrvObjPool [MEMORY] hold= 512 used= 120 count= 2 avg_used= 60 block_cnt= 1 chunk_cnt= 1 mod=UserResourceMgr [MEMORY] hold= 448 used= 64 count= 2 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfFSCbNode [MEMORY] hold= 416 used= 2 count= 2 avg_used= 1 block_cnt= 2 chunk_cnt= 2 mod=ObLogEXTHandler [MEMORY] hold= 400 used= 200 count= 1 avg_used= 200 block_cnt= 1 chunk_cnt= 1 mod=ExTabletMapLk [MEMORY] hold= 384 used= 102 count= 1 avg_used= 102 block_cnt= 1 chunk_cnt= 1 mod=TxLSLogBuf [MEMORY] hold= 304 used= 104 count= 1 avg_used= 104 block_cnt= 1 chunk_cnt= 1 mod=RedoLogBuckLock [MEMORY] hold= 256 used= 48 count= 1 avg_used= 48 block_cnt= 1 chunk_cnt= 1 mod=[T]TSITabletTab [MEMORY] hold= 256 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=[T]TSILastOper [MEMORY] hold= 256 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 240 used= 41 count= 1 avg_used= 41 block_cnt= 1 chunk_cnt= 1 mod=KeepAliveBuf [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=RebuildCbNode [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfRCCbNode [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=logservice [MEMORY] hold= 389625696 used= 379379459 count= 32871 avg_used= 11541 mod=SUMMARY [2025-08-14 09:02:37.803955] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=48] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 230686720 used= 228884480 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 block_cnt= 110 chunk_cnt= 110 mod=Memstore [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:02:37.803968] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= EXECUTE_CTX_ID hold= 1163264 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 15 washed_size= 13516800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:37.803980] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 block_cnt= 1 chunk_cnt= 1 mod=PartTranCtxMgr [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:02:37.804010] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 53714944 used= 47487424 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 84 washed_blocks= 334 washed_size= 126640128 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 40274240 used= 36480727 count= 2830 avg_used= 12890 block_cnt= 2830 chunk_cnt= 79 mod=SqlPhyPlan [MEMORY] hold= 3768320 used= 3427728 count= 321 avg_used= 10678 block_cnt= 84 chunk_cnt= 49 mod=SqlPlanCache [MEMORY] hold= 1524288 used= 1400877 count= 158 avg_used= 8866 block_cnt= 158 chunk_cnt= 11 mod=SqlPhyPlObj [MEMORY] hold= 1382080 used= 1263616 count= 617 avg_used= 2048 block_cnt= 223 chunk_cnt= 58 mod=CreateContext [MEMORY] hold= 536448 used= 523776 count= 66 avg_used= 7936 block_cnt= 66 chunk_cnt= 24 mod=OB_SQLSTAT_OBJ [MEMORY] hold= 2048 used= 1664 count= 2 avg_used= 832 block_cnt= 1 chunk_cnt= 1 mod=PlanCache [MEMORY] hold= 47487424 used= 43098388 count= 3994 avg_used= 10790 mod=SUMMARY [2025-08-14 09:02:37.804045] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=21] [MEMORY] tenant_id= 1 ctx_id= WORK_AREA hold= 1458176 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 13 washed_blocks= 66 washed_size= 25804800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:37.804066] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 14204928 used= 10970224 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 277 washed_size= 31932416 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10953744 used= 6065548 count= 18570 avg_used= 326 block_cnt= 372 chunk_cnt= 12 mod=PlJit [MEMORY] hold= 16480 used= 13376 count= 3 avg_used= 4458 block_cnt= 2 chunk_cnt= 2 mod=PlCodeGen [MEMORY] hold= 10970224 used= 6078924 count= 18573 avg_used= 327 mod=SUMMARY [2025-08-14 09:02:37.804086] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 111149056 used= 108896256 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 block_cnt= 211 chunk_cnt= 53 mod=CoStack [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 mod=SUMMARY [2025-08-14 09:02:37.804117] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=19] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 56623104 used= 56623104 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 56623104 used= 0 count= 27 avg_used= 0 block_cnt= 0 chunk_cnt= 0 mod=KvstorCacheMb [MEMORY] hold= 56623104 used= 0 count= 27 avg_used= 0 mod=SUMMARY [2025-08-14 09:02:37.804140] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 34365376 limit= 429496720 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 29491200 used= 26236400 count= 400 avg_used= 65591 block_cnt= 400 chunk_cnt= 15 mod=L_TabletPool [MEMORY] hold= 4784128 used= 4765440 count= 73 avg_used= 65280 block_cnt= 73 chunk_cnt= 3 mod=N_TabletPool [MEMORY] hold= 90048 used= 84768 count= 2 avg_used= 42384 block_cnt= 2 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 34365376 used= 31086608 count= 475 avg_used= 65445 mod=SUMMARY [2025-08-14 09:02:37.804160] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 8003584 used= 4947392 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 10 washed_blocks= 23 washed_size= 12967936 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4947392 used= 4793344 count= 604 avg_used= 7936 block_cnt= 66 chunk_cnt= 8 mod=MemtableCallbac [MEMORY] hold= 4947392 used= 4793344 count= 604 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:37.804182] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=LobPersist [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:37.804203] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 65536 used= 31024 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 2031616 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 31024 used= 27058 count= 20 avg_used= 1352 block_cnt= 6 chunk_cnt= 1 mod=SqlPsCache [MEMORY] hold= 31024 used= 27058 count= 20 avg_used= 1352 mod=SUMMARY [2025-08-14 09:02:37.804225] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= RPC_CTX_ID hold= 10633216 used= 3219456 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3022848 used= 1999488 count= 123 avg_used= 16256 block_cnt= 123 chunk_cnt= 2 mod=OB_WRS_CLUSTER_ [MEMORY] hold= 98304 used= 65024 count= 4 avg_used= 16256 block_cnt= 4 chunk_cnt= 2 mod=OB_LOG_REGISTER [MEMORY] hold= 24576 used= 16256 count= 1 avg_used= 16256 block_cnt= 1 chunk_cnt= 1 mod=OB_TENANT_MGR [MEMORY] hold= 24576 used= 16256 count= 1 avg_used= 16256 block_cnt= 1 chunk_cnt= 1 mod=OB_LOG_GET_STAT [MEMORY] hold= 16384 used= 7936 count= 2 avg_used= 3968 block_cnt= 2 chunk_cnt= 1 mod=[L]OB_WRS_CLUST [MEMORY] hold= 16384 used= 7936 count= 2 avg_used= 3968 block_cnt= 1 chunk_cnt= 1 mod=[L]OB_DETECT_MA [MEMORY] hold= 8192 used= 3968 count= 1 avg_used= 3968 block_cnt= 1 chunk_cnt= 1 mod=[L]OB_LOG_REGIS [MEMORY] hold= 8192 used= 3968 count= 1 avg_used= 3968 block_cnt= 1 chunk_cnt= 1 mod=[L]OB_TENANT_MG [MEMORY] hold= 3219456 used= 2120832 count= 135 avg_used= 15709 mod=SUMMARY [2025-08-14 09:02:37.804515] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=275] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 3481600 used= 2391232 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 10 washed_size= 4907008 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2121728 used= 2105152 count= 3 avg_used= 701717 block_cnt= 3 chunk_cnt= 3 mod=MEMTABLE_ARENA [MEMORY] hold= 261376 used= 253952 count= 32 avg_used= 7936 block_cnt= 21 chunk_cnt= 2 mod=TX_DATA_SLICE [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=TX_DATA_ARENA [MEMORY] hold= 2391232 used= 2367040 count= 36 avg_used= 65751 mod=SUMMARY [2025-08-14 09:02:37.804539] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=15] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 3964928 used= 3080192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 3 washed_size= 229376 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3080192 used= 2983936 count= 376 avg_used= 7936 block_cnt= 24 chunk_cnt= 2 mod=MdsTable [MEMORY] hold= 3080192 used= 2983936 count= 376 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:37.804562] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1 ctx_id= SCHEMA_SERVICE hold= 212992 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1884160 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=NonPartTenCac [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:37.804583] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=14] [MEMORY] tenant_id= 1 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:37.804602] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1 ctx_id= MERGE_NORMAL_CTX_ID hold= 884736 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 6 washed_blocks= 9 washed_size= 11698176 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:37.807229] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891814fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.793, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.807275] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.817290] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7facb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:37.819861] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35018][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:37.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.821436] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [571][T1_ArcSender][T1][YB42AC4D0007-00063C495BA99C16-0-0] [lt=869088][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:02:37.825133] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=872166][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:02:37.828319] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=865935][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:02:37.834807] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=951989][errcode=0] EASY SLOW: thread: 7f99542fb300, lastrun: 1755162115.752503 cost: 42.082302 loop:26416, slowcnt: 353 [2025-08-14 09:02:37.840733] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EEB9-0-0] [lt=188] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-08-14 09:02:37.844887] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:37.848912] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc49d8, timer:0x7f9970f40f80, task:0x564d6ef827d0, task_type:"N9oceanbase7storage20ObDDLCtrlSpeedHandle22RefreshSpeedHandleTaskE", scheduled_time:1755161615757725, pushed_time:0, delay:1000000}, running_task_token={this:0x7f9970fc5fb8, timer:0x7f9970f40f80, task:0x564d727182f0, task_type:"N9oceanbase5share16ObAshRefreshTaskE", scheduled_time:1755161611717613, pushed_time:1755161614757742, delay:120000000}, total_delay=542091186, threashold=1000000, tenant_id=500) [2025-08-14 09:02:37.854657] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=82316][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:37.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.860286] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c0b90, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571c0e70, time_wheel:0x7f9959fef990, thread_pool:0x7f9959fef170, is_running:1, time_interval:3.00s, expected_run_time:09:02:37.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.861597] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=23] PNIO [ratelimit] time: 1755162157861594, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007613, add_bytes: 0 [2025-08-14 09:02:37.863420] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=408121595, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:37.863443] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:37.864055] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1001}, delta=413134964, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:37.864076] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:37.864357] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=610125600, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:37.864382] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:37.882080] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=8] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:37.882253] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=5] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:37.886787] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1}, delta=438111405, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:37.886809] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:37.919936] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:37.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.944987] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:37.949067] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=889092] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:02:37.954718] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:37.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:37.963502] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=29] get wrs ts(ls_id={id:1}, delta=408221746, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:37.963529] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:37.964141] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25] get wrs ts(ls_id={id:1001}, delta=413235115, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:37.964158] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:37.964391] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=610225671, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:37.964410] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:37.982140] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=22] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:37.982323] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=14] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:37.986818] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=438211430, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:37.986839] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:37.986878] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [758][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB90BB-0-0] [lt=481][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1001, svr="172.77.0.7:2882", version={val:1755161719775179002, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162157986287) [2025-08-14 09:02:37.986901] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [758][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB90BB-0-0] [lt=18][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1001, req={req_server:"172.77.0.7:2882", version:{val:1755161719775179002, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162157986287}, twrs={inited:true, tenant_id:1001, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162157986325}, server_version_for_stat:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162157986325}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:37.987520] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=438211430, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:37.987533] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:38.008834] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199fbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919acc10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:37.995, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:38.011906] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:38.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.011940] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:38.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.020002] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:37.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.021971] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:38.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.044991] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:38.054780] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:37.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.063550] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=408321982, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:38.063574] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:38.064197] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1001}, delta=413335351, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:38.064215] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:38.064405] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=610325754, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:38.064427] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:38.082199] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=21] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:38.082384] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=14] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:38.087583] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=438311426, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:38.120075] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:38.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.145040] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:38.154844] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:38.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.163544] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=29] get wrs ts(ls_id={id:1}, delta=408421556, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:38.163571] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:38.164185] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] get wrs ts(ls_id={id:1001}, delta=413434925, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:38.164210] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:38.182443] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=14] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:38.182703] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:02:38.182754] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:02:38.182794] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:02:38.182832] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:02:38.182868] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:02:38.182879] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=1, sync_total=0, async_cnt= 1 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:38.182890] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:38.182898] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SERVER] net_in=0kB/s, net_out=0kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=1280000kB/s [2025-08-14 09:02:38.220140] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:38.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.245100] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:38.251859] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:38.251885] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:02:38.251903] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:38.251919] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:38.251942] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:38.251952] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:38.251970] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:38.251981] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:01:49.731, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:01:42.751], idx=659, next_allow_print_ts="09:02:19.731") [2025-08-14 09:02:38.251992] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:38.252006] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:38.252019] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:38.252030] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:38.252042] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:38.252202] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:38.252215] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:38.254883] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:38.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.257100] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=344] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:38.257123] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162158257088], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:38.257148] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162158257088], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:38.257170] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162158257088]}) [2025-08-14 09:02:38.257190] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=16] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:38.257212] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162158257189], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:38.257226] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162158257189]}) [2025-08-14 09:02:38.257441] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:38.257467] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:38.257494] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162158257437], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:38.257512] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162158257437]}) [2025-08-14 09:02:38.273535] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:38.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.283290] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:38.303715] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:38.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.320229] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:38.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.345162] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:38.354945] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:38.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.383352] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=22] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:38.420328] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:38.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.445227] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:38.454990] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:38.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.488396] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=22] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:38.520401] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=68][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:38.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.525057] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:38.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.545246] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:38.555050] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=52][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:38.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.591261] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=33] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:38.620465] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=67][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:38.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.645319] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:38.655117] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:38.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.687376] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:38.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.691325] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=29] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:38.716014] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=956112] swc wakeup.(stat_period_=1000000, ready=false) [567]OB_MALLOC COST TOO MUCH TIME, cost_time=554764, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:02:38.063614] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [979][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F73-0-0] [lt=30][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1002, svr="172.77.0.7:2882", version={val:1755161758221798000, v:0}, valid_part_count=0, total_part_count=2, generate_timestamp=1755162158063494) [2025-08-14 09:02:38.087603] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:38.720203] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=610980647, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:38.720224] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=7, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:38.720541] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=62][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:38.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.720831] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=35] get wrs ts(ls_id={id:1}, delta=610981709, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:38.720846] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:38.720873] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=20] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:38.721978] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:38.291747] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=13] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:38.720894] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=12] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:38.725760] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant: 500, limit: 9223372036854775807 hold: 697688064 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 340643840 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 39305216 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 85983232 limit= 9223372036854775807 [MEMORY] ctx_id= LIBEASY hold_bytes= 3145728 limit= 9223372036854775807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 10485760 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 2859008 limit= 9223372036854775807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 37355520 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 78925824 limit= 9223372036854775807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 98983936 limit= 9223372036854775807 [857]OB_MALLOC COST TOO MUCH TIME, cost_time=463039, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [2025-08-14 09:02:38.719990] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:38.723593] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728bb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:38.718, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:38.727817] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=4212][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180fbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:38.719, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:38.732250] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=23] gc stale ls task succ [2025-08-14 09:02:38.733917] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=1952895] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:38.733942] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=21] temporary file module use no blocks [2025-08-14 09:02:38.733953] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:38.733974] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=19] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:38.733988] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623060173607, flush_io_finished_ret_:0, flush_io_finished_round_:26316, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:38.736138] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=623] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:38.092420] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:38.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.736335] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=642937][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:38.924|207=643.91ms|210=0us|222=2us|total=643.94ms) [2025-08-14 09:02:38.736358] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4389] timer task use too much time(end=1755162158736358, start=1755162158092417, delta=643941) [2025-08-14 09:02:38.736373] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=532075, expected_run_ts_us_=1755162158204297, ObClockGenerator::getRealClock()=1755162158736372, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:38.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.736390] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:38.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.736404] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=477269, expected_run_ts_us_=1755162158259134, ObClockGenerator::getRealClock()=1755162158736404, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:38.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.736418] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:38.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.736430] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=480942, expected_run_ts_us_=1755162158255487, ObClockGenerator::getRealClock()=1755162158736429, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:38.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.736449] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:38.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.723688] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=431286] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3934) [2025-08-14 09:02:38.737893] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=14181][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:38.737919] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:38.737932] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=12] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3935) [2025-08-14 09:02:38.737940] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:38.738124] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:38.738137] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=13] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3936) [2025-08-14 09:02:38.738145] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:38.719925] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:57) [341][LSSysTblUp0][T0][YB42AC4D0007-00063C495B49B120-0-0] [lt=9][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-08-14 09:02:38.719562] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [979][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F73-0-0] [lt=654954][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1002, req={req_server:"172.77.0.7:2882", version:{val:1755161758221798000, v:0}, valid_part_count:0, total_part_count:2, generate_timestamp:1755162158063494}, twrs={inited:true, tenant_id:1002, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162158163528}, server_version_for_stat:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162158163528}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:38.742030] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] [COMPACTION DAG STATUS] (start_time=1755162148725925, end_time=1755162158742029, dag_status={}) [2025-08-14 09:02:38.744281] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:38.747607] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:38.748279] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2442] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:38.748367] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [317][pnio1][T1002][YB42AC4D0007-00063C49612B8F73-0-0] [lt=1073][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1002, delta=2976029, last_succ_cluster_heartbeat_tstamp_=1755162155772337) [2025-08-14 09:02:38.733689] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=717] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:38.748693] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14980][errcode=0] epoll handler too much time :15014 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:38.751568] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:38.727401] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=408985585, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:38.752271] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24826] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:38.737302] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:38.752385] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15059][errcode=0] epoll handler too much time :15092 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:38.752440] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:38.752462] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:00:01.751, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=08:59:51.751], idx=373, next_allow_print_ts="09:00:31.751") [2025-08-14 09:02:38.752497] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:38.752509] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:38.752527] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:38.752539] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:38.752559] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:38.752571] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:01:49.731, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:01:42.751], idx=659, next_allow_print_ts="09:02:19.731") [2025-08-14 09:02:38.752584] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:38.752599] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:38.752613] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:38.752625] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:38.752638] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:38.752809] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:37.777148] EDIAG [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1014638, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1014637, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1af3d850 0x1af3c294 0x20071394 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:38.740081] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=631083] get wrs ts(ls_id={id:1}, delta=438958988, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:38.754456] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14348] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=2, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:38.754833] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:02:38.754894] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1369] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:38.754923] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=981829] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:38.758567] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=464] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:38.758878] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=478] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:38.766360] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=984270] current monitor number(seq_=-1) [2025-08-14 09:02:38.768592] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:38.768630] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [409][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] [sql nio session](*s={this:0x7f98f7334030, session_id:3221557375, trace_id:YB42AC4D0007-00063C4962CA3040-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:262}, err:0, last_decode_time:1755161121788142, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:2389, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:38.768652] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [409][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] [sql nio session](*s={this:0x7f98f7333030, session_id:3221589989, trace_id:YB42AC4D0007-00063C4962CA3068-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:273}, err:0, last_decode_time:1755161766744050, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1903, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:38.768662] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [409][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] [sql nio session](*s={this:0x7f98f7339030, session_id:3221590023, trace_id:YB42AC4D0007-00063C4962CA3069-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:254}, err:0, last_decode_time:1755161784744265, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1683, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:38.768672] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [409][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] [sql nio session](*s={this:0x7f98f733d030, session_id:3221590940, trace_id:YB42AC4D0007-00063C4962CA306E-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:247}, err:0, last_decode_time:1755162096770219, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1286, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:38.771816] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] get wrs ts(ls_id={id:1001}, delta=414038837, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:38.771834] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:38.774025] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=409025468, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:38.774039] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:38.774529] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1001}, delta=414038837, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:38.774538] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:38.774554] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1}, delta=409025468, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:38.774561] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:38.774739] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=8] gc stale ls task succ [2025-08-14 09:02:38.775032] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8] get wrs ts(ls_id={id:1001}, delta=414038837, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:38.775039] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:38.783334] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] [sql nio session](*s={this:0x7f98f72d0030, session_id:3221589415, trace_id:YB42AC4D0007-00063C4965FA43DC-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:245}, err:0, last_decode_time:1755161751714366, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1662, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:38.783363] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=28] [sql nio session](*s={this:0x7f98f72d5030, session_id:3221590200, trace_id:YB42AC4D0007-00063C4965FA43E0-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:248}, err:0, last_decode_time:1755161776777167, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1593, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:38.791433] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=19] tx gc loop thread is running(MTL_ID()=1) [2025-08-14 09:02:38.791444] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=11] try gc retain ctx [2025-08-14 09:02:38.791444] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=940574] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:02:37.777148] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] [COMPACTION DAG STATUS] (start_time=1755162148722981, end_time=1755162157777146, dag_status={}) [2025-08-14 09:02:38.752824] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:38.748257] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=966] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:38.793468] WDIAG ev_invoke_pending (ev.c:2136) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=45140][errcode=0] epoll handler too much time :45218 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:38.748309] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:38.793508] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=45129][errcode=0] epoll handler too much time :45205 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:38.751848] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:38.794100] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=42186][errcode=0] epoll handler too much time :42259 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:38.766416] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=18] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:38.794190] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27726] temporary file module use no blocks [2025-08-14 09:02:38.794227] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=36] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:38.794254] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=26] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:38.794276] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=21] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623060204874, flush_io_finished_ret_:0, flush_io_finished_round_:26296, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:38.791449] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=5] try to retry start_working [2025-08-14 09:02:38.795574] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=4107] get_max_applied_scn(scn={val:1755161547738516001, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:50, palf_committed_end_lsn_:{lsn:5282737655}, palf_committed_end_scn_:{val:1755161547738516001, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161547738516001, v:0}}) [2025-08-14 09:02:38.795603] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=23] get_min_unreplayed_log_info(lsn={lsn:5282737655}, scn={val:1755100947825033001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161624848121, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:13063726, ret_code_:0}}, next_to_submit_lsn_:{lsn:5282737655}, next_to_submit_scn_:{val:1755100947825033001, v:0}, base_lsn_:{lsn:4831543296}, base_scn_:{val:1755063445733337001, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7605000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:5282737655}, end_lsn:{lsn:5282737655}, read_buf:{buf_len_:3747840, buf_:0x7f98f7605000}, block_size:67104768, log_storage_:0x7f993e529bf0, log_storage_:{log_tail:{lsn:5282737655}, readable_log_tail:{lsn:5282737655}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:5234171904}, min_scn:{val:1755152244802535004, v:0}, curr_block_id:78, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:206, min_block_id:42, max_block_id:79, curr_writable_block_id:78}, logical_block_size_:67104768, curr_block_writable_size_:18539017, block_header_serialize_buf_:0x7f993e52a8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:85, read_io_size:301367296, read_disk_cost_ts:2364604, start_lsn:{lsn:4831543296}}}}}}) [2025-08-14 09:02:38.778741] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=3466] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:38.797680] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=18907][errcode=0] epoll handler too much time :18948 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:38.799511] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=955229] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:02:38.802685] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=346][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:38.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:38.802723] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:38.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:38.802738] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196be10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:38.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:38.802752] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728d430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7290f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:38.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:38.802766] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:38.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:38.802780] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73922b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:38.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:38.802794] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:38.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:38.802807] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:38.799, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:38.802821] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:38.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:38.810180] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=2072][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:38.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.810221] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40b7f0, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40bad0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:10.00s, expected_run_time:09:02:38.802, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.812746] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:38.809, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:38.812789] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eaa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:38.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:38.820989] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=4506] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:38.821038] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] get_max_applied_scn(scn={val:1755161719775179002, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:45, palf_committed_end_lsn_:{lsn:8461098908}, palf_committed_end_scn_:{val:1755161719775179002, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161719775179002, v:0}}) [2025-08-14 09:02:38.821551] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] get wrs ts(ls_id={id:1}, delta=439045717, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:38.821574] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:38.828359] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:38.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.843326] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73945b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:38.837, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:38.844453] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [876][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E999C16-0-0] [lt=27] ObArchiveSender is running(thread_index=3) [2025-08-14 09:02:38.844531] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] EASY SLOW: thread: 7f99542fb300, lastrun: 1755162115.752503 cost: 43.092024 loop:26416, slowcnt: 363 [2025-08-14 09:02:38.851631] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:38.861364] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=2049][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:38.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.869240] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=21] PNIO [ratelimit] time: 1755162158869229, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007635, add_bytes: 0 [2025-08-14 09:02:38.874696] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8] get wrs ts(ls_id={id:1}, delta=409132583, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:38.874733] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=34] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:38.875253] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26] get wrs ts(ls_id={id:1001}, delta=414147022, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:38.875268] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:38.921054] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=20] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:38.923867] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:38.911, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:38.928423] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:38.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.962249] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:38.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.991385] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:38.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:38.953305] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=20] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:02:39.745689] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:39.748464] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:39.750291] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [660]OB_MALLOC COST TOO MUCH TIME, cost_time=727250, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=727249], seq:[0] [737]OB_MALLOC COST TOO MUCH TIME, cost_time=730509, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=730507], seq:[0] [729]OB_MALLOC COST TOO MUCH TIME, cost_time=838636, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1], seq:[0, 1] [857]OB_MALLOC COST TOO MUCH TIME, cost_time=785331, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:02:38.726076] EDIAG [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1033849, owner: ObLog, click_count: 3, time dist:[FORMAT_END=11, ALLOC_END=1033831, APPEND_END=7], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0xf994589 0x981942b 0xf98df79 0x96b2cac 0x96b1c88 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:38.726076] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=17] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 340643840 used= 301262512 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 490 washed_blocks= 2214 washed_size= 904953856 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 52641792 used= 50996736 count= 3317 avg_used= 15374 block_cnt= 1499 chunk_cnt= 42 mod=TZInfoArray [MEMORY] hold= 35672064 used= 35651584 count= 1 avg_used= 35651584 block_cnt= 1 chunk_cnt= 1 mod=qdtable [MEMORY] hold= 33292288 used= 33275904 count= 16 avg_used= 2079744 block_cnt= 16 chunk_cnt= 16 mod=ObTxDesc [MEMORY] hold= 25064000 used= 24981120 count= 159 avg_used= 157113 block_cnt= 105 chunk_cnt= 49 mod=TenantCtxAlloca [MEMORY] hold= 12873728 used= 12856400 count= 1 avg_used= 12856400 block_cnt= 1 chunk_cnt= 1 mod=MemDumpContext [MEMORY] hold= 7397376 used= 7340256 count= 7 avg_used= 1048608 block_cnt= 7 chunk_cnt= 7 mod=ConcurHashMap [MEMORY] hold= 7275040 used= 6334546 count= 1107 avg_used= 5722 block_cnt= 571 chunk_cnt= 6 mod=CharsetInit [MEMORY] hold= 5373952 used= 5236736 count= 32 avg_used= 163648 block_cnt= 32 chunk_cnt= 15 mod=Iteratorsql_session_id_=3221590970) [2025-08-14 09:02:47.726419] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=447950520, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:47.726449] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=3, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [567]OB_MALLOC COST TOO MUCH TIME, cost_time=294402, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [857]OB_MALLOC COST TOO MUCH TIME, cost_time=263386, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:02:47.727058] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=447951587, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:47.727086] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:47.727796] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=619988250, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:47.727814] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=4, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:47.728292] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=391][errcode=0] get_fetch_log_dst failed or invalid(ret=0, palf_id=1, self="172.77.0.7:2882", fetch_log_dst="172.77.0.7:2882") [2025-08-14 09:02:47.728669] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:47.728715] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:47.728728] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:47.728746] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:47.728759] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:47.728801] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:47.728821] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:01:49.731, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:01:42.751], idx=659, next_allow_print_ts="09:02:19.731") [2025-08-14 09:02:47.728834] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:47.728850] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:47.728871] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=763, timeout ts=09:02:10.603, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:40.603], idx=763, next_allow_print_ts="09:02:40.603") [2025-08-14 09:02:47.728883] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:47.728894] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:47.728927] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:47.728947] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=619989336, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:47.728962] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:47.729049] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=417986659, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:47.729063] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:47.729163] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:47.729205] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:47.729489] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:47.729669] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1001}, delta=423001557, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:47.729693] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:47.729743] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=30] get wrs ts(ls_id={id:1}, delta=417988188, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:47.729756] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:47.730451] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=30] get wrs ts(ls_id={id:1001}, delta=423001557, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:47.730467] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:47.730954] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 444993536 used= 389494752 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 70 washed_blocks= 287 washed_size= 70221824 [MEMORY] request_cached_chunk_cnt= 4 [MEMORY] hold= 389494752 used= 379248899 count= 32869 avg_used= 11538 mod=SUMMARY [2025-08-14 09:02:47.730997] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=16] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 230686720 used= 228884480 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 block_cnt= 110 chunk_cnt= 110 mod=Memstore [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:02:47.731011] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= EXECUTE_CTX_ID hold= 1163264 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 15 washed_size= 13516800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:47.731026] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:02:47.731041] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 53714944 used= 47487424 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 84 washed_blocks= 334 washed_size= 126640128 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 47487424 used= 43098388 count= 3994 avg_used= 10790 mod=SUMMARY [2025-08-14 09:02:47.731055] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= WORK_AREA hold= 1458176 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 13 washed_blocks= 66 washed_size= 25804800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:47.731069] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 14204928 used= 10970224 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 277 washed_size= 31932416 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10970224 used= 6078924 count= 18573 avg_used= 327 mod=SUMMARY [2025-08-14 09:02:47.731085] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 111149056 used= 108896256 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 block_cnt= 211 chunk_cnt= 53 mod=CoStack [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 mod=SUMMARY [2025-08-14 09:02:47.731104] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 56623104 used= 56623104 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 56623104 used= 0 count= 27 avg_used= 0 mod=SUMMARY [2025-08-14 09:02:47.731128] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=18] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 34365376 limit= 429496720 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 34365376 used= 31086608 count= 475 avg_used= 65445 mod=SUMMARY [2025-08-14 09:02:47.731143] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 8003584 used= 4947392 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 10 washed_blocks= 23 washed_size= 12967936 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4947392 used= 4793344 count= 604 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:47.731159] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:47.731173] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 65536 used= 31024 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 2031616 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 31024 used= 27058 count= 20 avg_used= 1352 mod=SUMMARY [2025-08-14 09:02:47.731191] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1 ctx_id= RPC_CTX_ID hold= 10633216 used= 3489792 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3489792 used= 2299648 count= 146 avg_used= 15751 mod=SUMMARY [2025-08-14 09:02:47.731209] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 3481600 used= 2391232 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 10 washed_size= 4907008 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2391232 used= 2367040 count= 36 avg_used= 65751 mod=SUMMARY [2025-08-14 09:02:47.731225] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 3964928 used= 3080192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 3 washed_size= 229376 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3080192 used= 2983936 count= 376 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:47.731254] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=19] [MEMORY] tenant_id= 1 ctx_id= SCHEMA_SERVICE hold= 212992 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1884160 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:47.731270] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:47.731284] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= MERGE_NORMAL_CTX_ID hold= 884736 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 6 washed_blocks= 9 washed_size= 11698176 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:47.731303] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant: 500, limit: 9223372036854775807 hold: 691585024 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 338358272 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 35405824 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 85983232 limit= 9223372036854775807 [MEMORY] ctx_id= LIBEASY hold_bytes= 3145728 limit= 9223372036854775807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 10485760 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 2940928 limit= 9223372036854775807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 37355520 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 78925824 limit= 9223372036854775807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 98983936 limit= 9223372036854775807 [2025-08-14 09:02:47.731333] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 338358272 used= 300915440 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 489 washed_blocks= 2213 washed_size= 903045120 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 300915440 used= 293521765 count= 12645 avg_used= 23212 mod=SUMMARY [2025-08-14 09:02:47.731358] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 500 ctx_id= GLIBC hold= 35405824 used= 21971824 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 188 washed_blocks= 821 washed_size= 371441664 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 21971824 used= 10369125 count= 131702 avg_used= 78 mod=SUMMARY [2025-08-14 09:02:47.731373] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= CO_STACK hold= 85983232 used= 83091456 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 83091456 used= 82936896 count= 161 avg_used= 515136 block_cnt= 161 chunk_cnt= 41 mod=CoStack [MEMORY] hold= 83091456 used= 82936896 count= 161 avg_used= 515136 mod=SUMMARY [2025-08-14 09:02:47.731387] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 500 ctx_id= LIBEASY hold= 3145728 used= 3070816 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1048576 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3070816 used= 2859784 count= 223 avg_used= 12824 mod=SUMMARY [2025-08-14 09:02:47.731400] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= LOGGER_CTX_ID hold= 10485760 used= 14565376 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 14565376 used= 14558208 count= 7 avg_used= 2079744 mod=SUMMARY [2025-08-14 09:02:47.731423] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= RPC_CTX_ID hold= 2940928 used= 1384448 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 54 washed_blocks= 67 washed_size= 110305280 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1384448 used= 914304 count= 57 avg_used= 16040 mod=SUMMARY [2025-08-14 09:02:47.731438] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= PKT_NIO hold= 37355520 used= 35866112 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 45 washed_blocks= 66 washed_size= 88760320 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 35866112 used= 34920256 count= 119 avg_used= 293447 mod=SUMMARY [2025-08-14 09:02:47.731462] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= SCHEMA_SERVICE hold= 78925824 used= 68912768 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 22 washed_size= 5767168 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 68912768 used= 67650911 count= 4857 avg_used= 13928 mod=SUMMARY [2025-08-14 09:02:47.731476] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 500 ctx_id= UNEXPECTED_IN_500 hold= 98983936 used= 97178960 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 97178960 used= 86786496 count= 3015 avg_used= 28784 mod=SUMMARY [2025-08-14 09:02:47.731491] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant: 508, limit: 1073741824 hold: 15613952 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 5128192 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10485760 limit= 9223372036854775807 [2025-08-14 09:02:47.731501] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 508 ctx_id= DEFAULT_CTX_ID hold= 5128192 used= 3765328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3765328 used= 3723912 count= 84 avg_used= 44332 mod=SUMMARY [2025-08-14 09:02:47.731528] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 508 ctx_id= CO_STACK hold= 10485760 used= 10321920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10321920 used= 10302720 count= 20 avg_used= 515136 mod=SUMMARY [2025-08-14 09:02:47.731576] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 509, limit: 1073741824 hold: 15613952 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 5128192 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10485760 limit= 9223372036854775807 [2025-08-14 09:02:47.731586] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 509 ctx_id= DEFAULT_CTX_ID hold= 5128192 used= 3765328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3765328 used= 3723912 count= 84 avg_used= 44332 mod=SUMMARY [2025-08-14 09:02:47.731609] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 509 ctx_id= CO_STACK hold= 10485760 used= 10321920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10321920 used= 10302720 count= 20 avg_used= 515136 mod=SUMMARY [2025-08-14 09:02:47.731656] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant: 1001, limit: 1073741824 hold: 606650368 cache_hold: 44040192 cache_used: 44040192 cache_item_count: 21 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 302309376 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 54525952 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 3080192 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 483328 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 45514752 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 5521408 limit= 157286400 [MEMORY] ctx_id= GLIBC hold_bytes= 15499264 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 73424896 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35651584 limit= 214748360 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 3252224 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 12582912 limit= 1073741800 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 5570560 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3866624 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 1179648 limit= 9223372036854775807 [2025-08-14 09:02:47.731683] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1001 ctx_id= DEFAULT_CTX_ID hold= 302309376 used= 268651920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 81 washed_blocks= 406 washed_size= 113410048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 268651920 used= 262543974 count= 3771 avg_used= 69621 mod=SUMMARY [2025-08-14 09:02:47.731707] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= MEMSTORE_CTX_ID hold= 54525952 used= 54099968 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 54099968 used= 54075216 count= 26 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:02:47.731717] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= EXECUTE_CTX_ID hold= 3080192 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 8 washed_size= 7405568 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:47.731728] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:02:47.731739] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= PLAN_CACHE_CTX_ID hold= 45514752 used= 36449984 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 77 washed_blocks= 201 washed_size= 132743168 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 36449984 used= 32916258 count= 3312 avg_used= 9938 mod=SUMMARY [2025-08-14 09:02:47.731746] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= WORK_AREA hold= 5521408 used= 0 limit= 157286400 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 9 washed_blocks= 40 washed_size= 17547264 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:47.731763] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=14] [MEMORY] tenant_id= 1001 ctx_id= GLIBC hold= 15499264 used= 10889728 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 25 washed_blocks= 164 washed_size= 47415296 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10889728 used= 6020141 count= 18313 avg_used= 328 mod=SUMMARY [2025-08-14 09:02:47.731771] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= CO_STACK hold= 73424896 used= 71221248 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 32 washed_blocks= 34 washed_size= 2072576 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 71221248 used= 71088768 count= 138 avg_used= 515136 block_cnt= 138 chunk_cnt= 36 mod=CoStack [MEMORY] hold= 71221248 used= 71088768 count= 138 avg_used= 515136 mod=SUMMARY [2025-08-14 09:02:47.731783] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= KVSTORE_CACHE_ID hold= 44040192 used= 44040192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 44040192 used= 0 count= 21 avg_used= 0 mod=SUMMARY [2025-08-14 09:02:47.731801] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 33906624 limit= 214748360 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33906624 used= 30629648 count= 468 avg_used= 65447 mod=SUMMARY [2025-08-14 09:02:47.731808] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= TX_CALLBACK_CTX_ID hold= 3252224 used= 2342592 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 3 washed_blocks= 8 washed_size= 3039232 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2342592 used= 2269696 count= 286 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:47.731817] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:47.731826] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= RPC_CTX_ID hold= 12582912 used= 1474560 limit= 1073741800 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1474560 used= 975360 count= 60 avg_used= 16256 mod=SUMMARY [2025-08-14 09:02:47.731845] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1001 ctx_id= TX_DATA_TABLE hold= 5570560 used= 2342784 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 8 washed_size= 2818048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2342784 used= 2319424 count= 30 avg_used= 77314 mod=SUMMARY [2025-08-14 09:02:47.731857] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= MDS_DATA_ID hold= 3866624 used= 2785280 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 5 washed_size= 4521984 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2785280 used= 2698240 count= 340 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:47.731874] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= MERGE_NORMAL_CTX_ID hold= 1179648 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 8 washed_blocks= 24 washed_size= 15597568 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:47.731893] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant: 1002, limit: 8589934592 hold: 1946038272 cache_hold: 48234496 cache_used: 48234496 cache_item_count: 23 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 1007513600 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 415236096 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 1114112 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 950272 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 211812352 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 1441792 limit= 429496725 [MEMORY] ctx_id= GLIBC hold_bytes= 12582912 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 149913600 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 37748736 limit= 1717986900 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 7766016 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 8077312 limit= 8589934500 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 5718016 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3047424 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 1032192 limit= 9223372036854775807 [2025-08-14 09:02:47.731923] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= DEFAULT_CTX_ID hold= 1007513600 used= 916842032 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 13 [MEMORY] hold= 185860096 used= 185534336 count= 114 avg_used= 1627494 block_cnt= 114 chunk_cnt= 104 mod=MysqlRequesReco [MEMORY] hold= 167792640 used= 167772192 count= 1 avg_used= 167772192 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 83927040 used= 83886080 count= 2 avg_used= 41943040 block_cnt= 2 chunk_cnt= 2 mod=LogGroupBuffer [MEMORY] hold= 67190784 used= 67109344 count= 4 avg_used= 16777336 block_cnt= 4 chunk_cnt= 4 mod=MrgMemPoolChk [MEMORY] hold= 916842032 used= 899631689 count= 9457 avg_used= 95128 mod=SUMMARY [2025-08-14 09:02:47.731946] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=17] [MEMORY] tenant_id= 1002 ctx_id= MEMSTORE_CTX_ID hold= 415236096 used= 411992064 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 411992064 used= 411803568 count= 198 avg_used= 2079816 block_cnt= 198 chunk_cnt= 198 mod=Memstore [MEMORY] hold= 411992064 used= 411803568 count= 198 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:02:47.731956] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= EXECUTE_CTX_ID hold= 1114112 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 7 washed_size= 7274496 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:47.731966] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= TRANS_CTX_MGR_ID hold= 950272 used= 933888 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1146880 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 933888 used= 921984 count= 2 avg_used= 460992 mod=SUMMARY [2025-08-14 09:02:47.731978] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= PLAN_CACHE_CTX_ID hold= 211812352 used= 201995008 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 176930880 used= 160228962 count= 13450 avg_used= 11912 block_cnt= 13450 chunk_cnt= 101 mod=SqlPhyPlan [MEMORY] hold= 201995008 used= 183048143 count= 17906 avg_used= 10222 mod=SUMMARY [2025-08-14 09:02:47.731987] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= WORK_AREA hold= 1441792 used= 0 limit= 429496725 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 14 washed_blocks= 54 washed_size= 27918336 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:47.731997] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= GLIBC hold= 12582912 used= 11425280 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 11425280 used= 6322556 count= 19216 avg_used= 329 mod=SUMMARY [2025-08-14 09:02:47.732008] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1002 ctx_id= CO_STACK hold= 149913600 used= 147087360 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 68 washed_blocks= 70 washed_size= 3178496 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 147087360 used= 146813760 count= 285 avg_used= 515136 block_cnt= 285 chunk_cnt= 73 mod=CoStack [MEMORY] hold= 147087360 used= 146813760 count= 285 avg_used= 515136 mod=SUMMARY [2025-08-14 09:02:47.732023] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= KVSTORE_CACHE_ID hold= 48234496 used= 48234496 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 48234496 used= 0 count= 23 avg_used= 0 mod=SUMMARY [2025-08-14 09:02:47.732033] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= META_OBJ_CTX_ID hold= 37748736 used= 35741632 limit= 1717986900 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 35741632 used= 32457488 count= 496 avg_used= 65438 mod=SUMMARY [2025-08-14 09:02:47.732043] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= TX_CALLBACK_CTX_ID hold= 7766016 used= 6934400 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 17 washed_size= 6914048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 6934400 used= 6729728 count= 848 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:47.732054] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= LOB_CTX_ID hold= 147456 used= 16384 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 16384 used= 15872 count= 2 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:47.732067] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= RPC_CTX_ID hold= 8077312 used= 3170304 limit= 8589934500 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 6 washed_blocks= 6 washed_size= 10797056 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3170304 used= 2083968 count= 135 avg_used= 15436 mod=SUMMARY [2025-08-14 09:02:47.732079] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= TX_DATA_TABLE hold= 5718016 used= 4521856 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 9 washed_blocks= 16 washed_size= 13156352 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4521856 used= 4480128 count= 40 avg_used= 112003 mod=SUMMARY [2025-08-14 09:02:47.732092] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= MDS_DATA_ID hold= 3047424 used= 1998848 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 4 washed_size= 1146880 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1998848 used= 1936384 count= 244 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:47.732102] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= MDS_CTX_ID hold= 147456 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:47.732115] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:47.732125] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= MERGE_NORMAL_CTX_ID hold= 1032192 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 12 washed_size= 13647872 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:47.733420] INFO [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:171) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=8] handle gts request statistics(total_rt=477686, total_cnt=12, avg_rt=3.674507692307692196e+04, avg_cnt=1.199999999999999956e+00) [2025-08-14 09:02:47.733519] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=97][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162167274937], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:47.733606] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162167274937], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:47.733633] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=23] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162167274937]}) [2025-08-14 09:02:47.734453] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=940778] blacklist_loop exec finished(cost_time=24, is_enabled=true, send_cnt=0) [2025-08-14 09:02:47.736452] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:47.726, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:47.736491] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739eb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:47.721, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:47.737385] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=958] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:47.742269] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=32676] gc stale ls task succ [2025-08-14 09:02:47.742321] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=918] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:47.743605] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:47.745032] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:02:47.745168] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:125) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=22] try to adjust log cb pool [2025-08-14 09:02:47.745193] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=16] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:47.747432] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [409][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=42719] kill and revert session(conn.sessid_=3221590940, proxy_sessid=0, ret=0) [2025-08-14 09:02:47.756017] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1125] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:47.756064] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:47.756088] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:47.756127] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7860] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:47.756269] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:47.757088] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:47.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:47.758139] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4878] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:47.758226] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:47.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:47.759229] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=7] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:47.760220] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:47.745064] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:8607757, max_lsn:1604090331, max_scn:{val:1755161756730847001, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:1556737504}, reuse_lsn:{lsn:1604090331}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:1562147291}}, last_submit_lsn:{lsn:1604090203}, last_submit_end_lsn:{lsn:1604090331}, last_submit_log_id:8607757, last_submit_log_pid:50, max_flushed_lsn:{lsn:1604090203}, max_flushed_end_lsn:{lsn:1604090331}, max_flushed_log_pid:50, committed_end_lsn:{lsn:1604090090}, last_slide_log_id:8607755, last_slide_scn:{val:1755161749741193000, v:0}, last_slide_lsn:{lsn:1604089968}, last_slide_end_lsn:{lsn:1604090090}, last_slide_log_pid:50, last_slide_log_accum_checksum:1446576759, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f990bfe4530}, [StateMgr]={this:0x7f990bfe89b0, self:"172.77.0.7:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"PENDING", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:51}, leader:"0.0.0.0:0", leader_epoch:-1, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:1604090331}, scan_disk_log_finished:true, last_check_start_id:8607756, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161901828039, palf_role_change_cb:0x7f990bfef470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:50, prev:{version:1, config:{config_version:{proposal_id:49, config_seq:45}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:50, config_seq:46}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_mode_pid:50}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_end_lsn:{lsn:1604088821}, prev_mode_pid:50}, reconfig_barrier:{prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_end_lsn:{lsn:1604088821}, prev_mode_pid:50}, persistent_config_version:{proposal_id:50, config_seq:46}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755161738854289, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755161738854289, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f990bfe5c70}, [ModeMgr]={palf_id:1, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:50, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:1543409664}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:51}, log_config_meta:{version:3, proposal_id:50, prev:{version:1, config:{config_version:{proposal_id:49, config_seq:45}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:50, config_seq:46}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_mode_pid:50}, log_snapshot_meta:{version:2, base_lsn:{lsn:1543409664}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:712704}, readable_log_tail:{lsn:712704}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:1610514432}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/meta", dir_fd:225, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66392064, block_header_serialize_buf_:0x7f990bfeabb8, flashback_version:0}, log_storage:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1755161901828095, this:0x7f990bfe90f0}, [Reconfirm]={palf_id:1, self:"172.77.0.7:2882", state:"RECONFIRM_MODE_META", new_proposal_id:51, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"172.77.0.7:2882", majority_max_accept_pid:50, majority_max_lsn:{lsn:1604090331}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:1755161901828373, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f990bfe8b30}) [2025-08-14 09:02:47.761753] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16656] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161974728707, last_flushed_end_lsn:{lsn:1604090331}}]) [2025-08-14 09:02:47.761773] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=18] current log_task status(palf_id=1, self="172.77.0.7:2882", log_id=8607756, log_task={header:{begin_lsn:{lsn:1604090090}, end_lsn:{lsn:1604090203}, log_id:8607756, min_scn:{val:1755161749741193001, v:0}, max_scn:{val:1755161749741193001, v:0}, data_len:57, proposal_id:50, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:1604090090}, data_checksum:2031283194, accum_checksum:759186326, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1755161750716165, freeze_ts:1755161750716173, submit_ts:1755161750716183, flushed_ts:1755161853765770, gen_to_freeze cost time:8, gen_to_submit cost time:18, submit_to_flush cost time:103049587}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161974728707, last_flushed_end_lsn:{lsn:1604090331}}]) [2025-08-14 09:02:47.761816] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] get_fetch_log_dst failed or invalid(ret=0, palf_id=1001, self="172.77.0.7:2882", fetch_log_dst="172.77.0.7:2882") [2025-08-14 09:02:47.757542] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:47.760308] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=78] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3958) [2025-08-14 09:02:47.761959] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=1646][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:47.761977] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:47.761987] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3959) [2025-08-14 09:02:47.761994] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:47.762175] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:47.762186] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3960) [2025-08-14 09:02:47.762194] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:47.762337] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:47.761875] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=58] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:02:47.766944] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:47.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:47.766995] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:47.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:47.767279] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:47.750, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:47.767300] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:47.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:47.767313] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d80b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:47.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:47.767326] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:47.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:47.767340] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891992330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196d710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:47.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:47.767353] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:47.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:47.770523] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=210] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:47.771805] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:47.772417] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:47.775493] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=5043] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:47.775539] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=5110] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:47.777346] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:47.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:47.777408] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=290] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:47.777427] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] temporary file module use no blocks [2025-08-14 09:02:47.777447] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=19] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:47.777461] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:47.777477] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623069217125, flush_io_finished_ret_:0, flush_io_finished_round_:26302, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:47.777523] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:47.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:47.777584] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=59][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:47.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:47.778875] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=17] gc stale ls task succ [2025-08-14 09:02:47.782706] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=1508] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:02:47.784412] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=571] current monitor number(seq_=-1) [2025-08-14 09:02:47.784461] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:47.784510] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=23] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:47.787567] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:47.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:47.787611] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72751b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:47.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:47.787626] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d31b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:47.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:47.787640] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73959b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:47.773, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:47.791981] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:47.797598] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:47.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:47.797631] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72856b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:47.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:47.797648] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:47.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:47.802531] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=22] gc stale ls task succ [2025-08-14 09:02:47.807579] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:47.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:47.807657] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919920b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196d590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:47.790, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:47.822133] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=23] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:47.824721] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:47.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:47.826694] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [758][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB90CF-0-0] [lt=40][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1001, svr="172.77.0.7:2882", version={val:1755161719775179002, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162167826547) [2025-08-14 09:02:47.826719] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [758][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB90CF-0-0] [lt=24][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1001, req={req_server:"172.77.0.7:2882", version:{val:1755161719775179002, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162167826547}, twrs={inited:true, tenant_id:1001, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162167726478}, server_version_for_stat:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162167726478}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:47.826890] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [316][pnio1][T1001][YB42AC4D0007-00063C4960CB90CF-0-0] [lt=20][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1001, delta=570421, last_succ_cluster_heartbeat_tstamp_=1755162167256469) [2025-08-14 09:02:47.827230] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=448051779, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:47.827291] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=57] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:47.828566] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=620088442, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:47.828591] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:47.829808] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=409608008, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=251715, succ_cluster_heartbeat_count=1499, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161006924036004, v:0}, local_cluster_delta=1160905770, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:02:47.829893] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=39] get wrs ts(ls_id={id:1}, delta=418088563, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:47.829907] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:47.829940] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27] get_max_applied_scn(scn={val:1755161744727824001, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:49, palf_committed_end_lsn_:{lsn:7323686009}, palf_committed_end_scn_:{val:1755161747742105001, v:0}, last_check_scn_:{val:1755161749741193000, v:0}, max_applied_cb_scn_:{val:1755161744727824001, v:0}}) [2025-08-14 09:02:47.829959] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get_min_unreplayed_log_info(lsn={lsn:7323685240}, scn={val:1755100946796590002, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873485, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:8946764, ret_code_:0}}, next_to_submit_lsn_:{lsn:7323685240}, next_to_submit_scn_:{val:1755100946796590002, v:0}, base_lsn_:{lsn:7180210176}, base_scn_:{val:1753840803166611000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:7323685240}, end_lsn:{lsn:7323685240}, read_buf:{buf_len_:3747840, buf_:0x7f98f7a05000}, block_size:67104768, log_storage_:0x7f9901131bf0, log_storage_:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:28, read_io_size:98508800, read_disk_cost_ts:645339, start_lsn:{lsn:7180210176}}}}}}) [2025-08-14 09:02:47.830643] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=80] get wrs ts(ls_id={id:1001}, delta=423101932, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:47.830659] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:47.845247] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=12] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:47.857968] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:47.840, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:47.858002] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d54b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ecd90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:47.846, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:47.858017] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ecc10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:47.848, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:47.858244] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=667267][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb3fb8, timer:0x7f99333cb180, task:0x7f99333cb220, task_type:"N9oceanbase8observer18ObTableLoadService21ObClientTaskPurgeTaskE", scheduled_time:1755161984781007, pushed_time:0, delay:1000000}, running_task_token={this:0x7f9931bb3ed8, timer:0x7f99333cb180, task:0x7f99333cb208, task_type:"N9oceanbase8observer18ObTableLoadService25ObClientTaskAutoAbortTaskE", scheduled_time:1755161959782824, pushed_time:1755162045729074, delay:1000000}, total_delay=183077236, threashold=1000000, tenant_id=1001) [2025-08-14 09:02:47.858309] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=65][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb3b58, timer:0x7f99333cb180, task:0x7f99333cb1a8, task_type:"N9oceanbase8observer18ObTableLoadService17ObCheckTenantTaskE", scheduled_time:1755162046728894, pushed_time:0, delay:1000000}, running_task_token={this:0x7f9931bb3ed8, timer:0x7f99333cb180, task:0x7f99333cb208, task_type:"N9oceanbase8observer18ObTableLoadService25ObClientTaskAutoAbortTaskE", scheduled_time:1755161959782824, pushed_time:1755162045729074, delay:1000000}, total_delay=121129415, threashold=1000000, tenant_id=1001) [2025-08-14 09:02:47.871830] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:47.922201] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=29] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:47.924784] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:47.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:47.929569] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=21] PNIO [ratelimit] time: 1755162167929566, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007593, add_bytes: 0 [2025-08-14 09:02:47.929892] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get_max_applied_scn(scn={val:1755161749741193000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:50, palf_committed_end_lsn_:{lsn:1604090090}, palf_committed_end_scn_:{val:1755161749741193000, v:0}, last_check_scn_:{val:1755161749741193001, v:0}, max_applied_cb_scn_:{val:1755161749741193000, v:0}}) [2025-08-14 09:02:47.929919] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] get_min_unreplayed_log_info(lsn={lsn:1604088821}, scn={val:1755100948296325001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873479, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:12492179, ret_code_:0}}, next_to_submit_lsn_:{lsn:1604088821}, next_to_submit_scn_:{val:1755100948296325001, v:0}, base_lsn_:{lsn:1409200128}, base_scn_:{val:1753840802540914000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:1604088821}, end_lsn:{lsn:1604088821}, read_buf:{buf_len_:3747840, buf_:0x7f98f6a05000}, block_size:67104768, log_storage_:0x7f990bfebbf0, log_storage_:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:42, read_io_size:147709952, read_disk_cost_ts:1061450, start_lsn:{lsn:1409200128}}}}}}) [2025-08-14 09:02:47.929995] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=73] get wrs ts(ls_id={id:1}, delta=418188010, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:47.930007] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:47.930535] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1001}, delta=423201379, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:47.930546] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:47.931822] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=620192137, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:47.931840] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:47.945306] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=20] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:47.958246] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] EASY SLOW: thread: 7f99542fb300, lastrun: 1755162115.752503 cost: 52.205727 loop:26416, slowcnt: 436 [2025-08-14 09:02:47.959248] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb4958, timer:0x7f9966ee3f40, task:0x7f99333d35a8, task_type:"N9oceanbase3omt11ObTenantSrs27TenantSrsUpdatePeriodicTaskE", scheduled_time:1755161742719922, pushed_time:0, delay:0}, running_task_token={this:0x7f9931bb4178, timer:0x7f9966ee3f40, task:0x7f99333d2fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161723481487, pushed_time:1755161737766048, delay:900000000}, total_delay=425239325, threashold=1000000, tenant_id=1001) [2025-08-14 09:02:47.959277] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb4338, timer:0x7f9966ee3f40, task:0x7f99333da1b0, task_type:"N9oceanbase5table20ObTableObjectPoolMgr26ObTableSessEliminationTaskE", scheduled_time:1755161742732119, pushed_time:0, delay:5000000}, running_task_token={this:0x7f9931bb4178, timer:0x7f9966ee3f40, task:0x7f99333d2fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161723481487, pushed_time:1755161737766048, delay:900000000}, total_delay=425227157, threashold=1000000, tenant_id=1001) [2025-08-14 09:02:47.959286] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb4418, timer:0x7f9966ee3f40, task:0x7f99333da1c8, task_type:"N9oceanbase5table20ObTableObjectPoolMgr27ObTableSessSysVarUpdateTaskE", scheduled_time:1755161742735029, pushed_time:0, delay:5000000}, running_task_token={this:0x7f9931bb4178, timer:0x7f9966ee3f40, task:0x7f99333d2fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161723481487, pushed_time:1755161737766048, delay:900000000}, total_delay=425224257, threashold=1000000, tenant_id=1001) [2025-08-14 09:02:47.959299] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb44f8, timer:0x7f9966ee3f40, task:0x7f99333da1e0, task_type:"N9oceanbase5table20ObTableObjectPoolMgr28ObTableReqResEliminationTaskE", scheduled_time:1755161742735074, pushed_time:0, delay:5000000}, running_task_token={this:0x7f9931bb4178, timer:0x7f9966ee3f40, task:0x7f99333d2fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161723481487, pushed_time:1755161737766048, delay:900000000}, total_delay=425224225, threashold=1000000, tenant_id=1001) [2025-08-14 09:02:47.959328] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:47.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:47.971889] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:02:47.971918] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:47.987947] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:02:47.987998] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:02:47.988036] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:02:47.988081] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:02:47.988099] INFO [COMMON] print_io_status (ob_io_manager.cpp:2639) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:2, norm_iops:0, bw:8183, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:471353, total:471358) [2025-08-14 09:02:47.988166] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=53] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, param_config_:{memory_limit:536870912, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=865568, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:02:47.988256] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2826) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=85] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:2, avg_bw:8182, [delay/us]: prepare:0, schedule:0, submit:600000000, device:471353, total:471358) [2025-08-14 09:02:47.988275] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:02:47.988289] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:47.988310] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:47.988321] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SERVER] net_in=1kB/s, net_out=1kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=1280000kB/s [2025-08-14 09:02:48.012541] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=65][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:48.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.012594] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=51][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:48.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.022255] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:48.024856] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:47.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.029839] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:48.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.030007] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=418288495, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:48.030027] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:48.030584] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1001}, delta=423301864, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:48.030597] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:48.031854] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=620292239, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:48.031873] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:48.045339] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=18] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:48.071962] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:48.082356] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=20] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:02:48.093089] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:48.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.122327] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=20] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:48.124918] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:48.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.130043] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=418388108, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:48.130062] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:48.130634] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] get wrs ts(ls_id={id:1001}, delta=423401477, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:48.130646] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:48.131922] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=620392382, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:48.131950] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:48.145376] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=18] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:48.160323] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:48.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.171985] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:48.213867] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:48.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.222387] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=19] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:48.226470] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:48.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.233962] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:48.233982] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162168233956], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:48.233999] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162168233956], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:48.234014] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162168233956]}) [2025-08-14 09:02:48.234029] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=11] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:48.234043] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162168234028], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:48.234053] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162168234028]}) [2025-08-14 09:02:48.234256] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:48.234269] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:48.234281] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162168234230], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:48.234292] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162168234230]}) [2025-08-14 09:02:48.245437] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=18] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:48.262531] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:48.262569] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:227) [1031][PxTargetMgr0][T1][Y0-0000000000000000-0-0] [lt=34] refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3961) [2025-08-14 09:02:48.262577] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:48.262593] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=11] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:48.262607] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=14] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3962) [2025-08-14 09:02:48.262613] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:48.262780] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:48.262796] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=16] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3963) [2025-08-14 09:02:48.262801] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:48.265449] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:48.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.265481] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:48.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.272045] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:48.275366] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:48.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.305546] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:48.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.326531] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:48.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.345498] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=24] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:48.360450] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:48.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.372096] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:48.426592] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:48.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.445567] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=21] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:48.472156] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:48.472601] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:48.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.518848] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:48.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.518892] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:48.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.526656] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:48.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.527039] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=52][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:48.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.545597] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=20] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:48.560543] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:48.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.572204] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:48.599252] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:48.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.603524] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:48.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.626717] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:48.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.645657] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=31] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:48.672286] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:48.672719] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:48.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.710055] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:48.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.229425] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:48.714878] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=484707][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:48.714929] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [857]OB_MALLOC COST TOO MUCH TIME, cost_time=484581, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=482886, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:02:48.714955] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:48.714970] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:48.714989] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:48.715001] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:01:49.731, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:01:42.751], idx=659, next_allow_print_ts="09:02:19.731") [2025-08-14 09:02:48.715013] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:48.715029] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:48.715042] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=763, timeout ts=09:02:10.603, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:40.603], idx=763, next_allow_print_ts="09:02:40.603") [2025-08-14 09:02:48.715054] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:48.715066] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:48.715079] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:48.715271] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:48.715287] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:48.716494] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=620977654, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:48.716516] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=2, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:48.717099] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=620977654, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:48.717110] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:48.715435] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119812272, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-08-14 09:02:48.717309] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=1857] get wrs ts(ls_id={id:1}, delta=418976025, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:48.717324] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:48.717899] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1001}, delta=423989394, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:48.717911] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:48.717932] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=418976025, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:48.717943] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:48.718510] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1001}, delta=423990462, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:48.718524] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:48.720396] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:48.322447] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=19] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:48.721544] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=398482] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:48.722548] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:48.715488] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=9] new mysql sessid created(conn.sessid_=3221590972, support_ssl=false) [2025-08-14 09:02:48.723224] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=7679] sm conn init succ(conn.sessid_=3221590972, sess.client_addr_="172.77.0.1:48092") [2025-08-14 09:02:48.723255] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1090) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=25] accept one succ(*s={this:0x7f98f72a6030, session_id:3221590972, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:264}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:48.025395] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=29][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:02:48.028596] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=22][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:02:48.715700] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [964][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F87-0-0] [lt=17][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1002, svr="172.77.0.7:2882", version={val:1755161758221798000, v:0}, valid_part_count=0, total_part_count=2, generate_timestamp=1755162168230055) [2025-08-14 09:02:48.727044] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [964][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F87-0-0] [lt=11319][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1002, req={req_server:"172.77.0.7:2882", version:{val:1755161758221798000, v:0}, valid_part_count:0, total_part_count:2, generate_timestamp:1755162168230055}, twrs={inited:true, tenant_id:1002, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162168717927}, server_version_for_stat:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162168717927}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:48.021776] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [571][T1_ArcSender][T1][YB42AC4D0007-00063C495BA99C16-0-0] [lt=26][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:02:48.727178] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [316][pnio1][T1002][YB42AC4D0007-00063C49612B8F87-0-0] [lt=14][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1002, delta=1762245, last_succ_cluster_heartbeat_tstamp_=1755162166964931) [2025-08-14 09:02:48.727611] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=418985079, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:48.727630] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:48.728217] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1001}, delta=423998448, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:48.728241] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:48.728823] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=193] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:48.730158] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:48.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.730284] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=27] [WRS] update tenant weak read server version(tenant_id=1001, server_version={version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162167826601}, version_delta=448955104) [2025-08-14 09:02:48.730937] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=27] get wrs ts(ls_id={id:1}, delta=448951093, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:48.730961] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=9, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:48.732391] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.722, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:48.732891] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc190, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc470, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:10.00s, expected_run_time:09:02:48.720, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.734127] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=32] get wrs ts(ls_id={id:1}, delta=448951093, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:48.734148] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:48.734181] WDIAG [SQL.SESSION] set_session_state_ (ob_basic_session_info.cpp:6475) [998][T1002_L0_G0][T1002][YB42AC4D0007-00063C4962CA306F-0-0] [lt=1219][errcode=-5066] session is killed(ret=-5066, sessid_=3221590940, proxy_sessid_=0, state=1) [2025-08-14 09:02:48.734216] WDIAG [RPC.OBMYSQL] async_write_data (ob_sql_sock_session.cpp:162) [998][T1002_L0_G0][T1002][YB42AC4D0007-00063C4962CA306F-0-0] [lt=29][errcode=-4009] sock has error(ret=-4009) [2025-08-14 09:02:48.734225] WDIAG [SERVER] flush_buffer (obmp_packet_sender.cpp:1070) [998][T1002_L0_G0][T1002][YB42AC4D0007-00063C4962CA306F-0-0] [lt=7][errcode=-4009] write response fail(ret=-4009) [2025-08-14 09:02:48.734250] WDIAG [SERVER] process (obmp_query.cpp:385) [998][T1002_L0_G0][T1002][YB42AC4D0007-00063C4962CA306F-0-0] [lt=23][errcode=0] failed to flush_buffer(tmp_ret=-4009) [2025-08-14 09:02:48.734377] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [14881][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task too much delay in priority queue(token={this:0x7f9970fc6cd8, timer:0x7f9966a67db0, task:0x564d6d07b128, task_type:"N9oceanbase8observer28ObServerLSMetaTableCheckTaskE", scheduled_time:1755162057757198, pushed_time:1755162060725640, delay:0}, delay_in_priority_queue=2968442, threashold=1000000) [2025-08-14 09:02:48.734554] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [14881][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=177][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f9970fc6cd8, timer:0x7f9966a67db0, task:0x564d6d07b128, task_type:"N9oceanbase8observer28ObServerLSMetaTableCheckTaskE", scheduled_time:1755162057757198, pushed_time:1755162060725640, delay:0}, delay_in_thread_pool=106064139, threashold=500000) [2025-08-14 09:02:48.737353] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=8] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:48.737387] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=33][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162168737349], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:48.737406] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162168737349], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:48.737421] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162168737349]}) [2025-08-14 09:02:48.737434] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162168737349]}) [2025-08-14 09:02:48.737445] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:48.737457] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:48.737469] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:48.737478] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162168737468], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:48.737490] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162168737468]}) [2025-08-14 09:02:48.737499] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162168737468]}) [2025-08-14 09:02:48.737678] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:48.737690] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:48.737702] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162168737674], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:48.737714] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162168737674]}) [2025-08-14 09:02:48.737723] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162168737674]}) [2025-08-14 09:02:48.738814] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:48.741187] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [408][sql_nio1][T1002][Y0-0000000000000000-0-0] [lt=949570] succeed to dispatch to tenant mysql queue(tenant_id=1002) [2025-08-14 09:02:48.741206] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:281) [408][sql_nio1][T1002][Y0-0000000000000000-0-0] [lt=18] mysql login queue(mysql_queue->queue_.size()=1) [2025-08-14 09:02:48.741228] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [408][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] kill and revert session(conn.sessid_=3221590970, proxy_sessid=0, ret=0) [2025-08-14 09:02:48.743394] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:48.743435] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=25] gc stale ls task succ [2025-08-14 09:02:48.743549] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:48.743590] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891826cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891918190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:48.745026] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [413][sql_nio6][T1002][Y0-0000000000000000-0-0] [lt=18] succeed to dispatch to tenant mysql queue(tenant_id=1002) [2025-08-14 09:02:48.745122] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [14881][TimerWK4][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f9970fc4d58, timer:0x7f9970f438b0, task:0x564d6d07c120, task_type:"N9oceanbase8observer8ObServer17ObCTASCleanUpTaskE", scheduled_time:1755162071765838, pushed_time:1755162071771622, delay:60000000}, delay_in_thread_pool=96962954, threashold=500000) [2025-08-14 09:02:48.745323] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:48.745533] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:48.748683] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=20] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:48.749326] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:985) [409][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=26] can close safely, do destroy(*s={this:0x7f98f733d030, session_id:3221590940, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:247}, err:5, last_decode_time:1755162160746190, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1422, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-08-14 09:02:48.750062] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:48.750101] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:48.750186] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:48.753479] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4384] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:48.754759] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:48.754797] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919845b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:48.755437] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:48.759013] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:48.759786] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=29] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:48.761673] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:48.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.762205] WDIAG [SERVER.OMT] process_one (ob_worker_processor.cpp:82) [998][T1002_L0_G0][T1002][YB42AC4D0007-00063C4962CA306F-0-0] [lt=13][errcode=-4009] process request fail(ret=-4009) [2025-08-14 09:02:48.764369] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1163] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:02:48.764415] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=38] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:02:48.766539] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:48.766584] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:48.766599] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:48.766613] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:48.767273] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:48.766627] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eacb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:48.766966] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=57][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:48.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.767390] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=420][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:48.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.768190] WDIAG [SERVER.OMT] process (ob_worker_processor.cpp:160) [998][T1002_L0_G0][T1002][YB42AC4D0007-00063C4962CA306F-0-0] [lt=20][errcode=-4009] process request fail(ret=-4009) [2025-08-14 09:02:48.767434] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=12049] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [374]OB_MALLOC COST TOO MUCH TIME, cost_time=951008, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=41, WASH_BLOCK_END=950967], seq:[0, 1, 2] [729]OB_MALLOC COST TOO MUCH TIME, cost_time=897397, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=2], seq:[0, 1] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=912488, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [2025-08-14 09:02:48.770688] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:48.772351] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:48.779507] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:48.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.782819] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=55] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:02:48.803265] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:48.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.815375] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40b7f0, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40bad0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:10.00s, expected_run_time:09:02:48.802, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.872411] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=16] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:02:48.872428] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:48.972265] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb3d18, timer:0x7f99333cb180, task:0x7f99333cb1d8, task_type:"N9oceanbase8observer18ObTableLoadService8ObGCTaskE", scheduled_time:1755161987758103, pushed_time:0, delay:30000000}, running_task_token={this:0x7f9931bb3ed8, timer:0x7f99333cb180, task:0x7f99333cb208, task_type:"N9oceanbase8observer18ObTableLoadService25ObClientTaskAutoAbortTaskE", scheduled_time:1755161959782824, pushed_time:1755162045729074, delay:1000000}, total_delay=181214161, threashold=1000000, tenant_id=1001) [2025-08-14 09:02:48.972603] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:48.991906] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:48.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.959785] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=893307][errcode=0] EASY SLOW: thread: 7f9953efb300, lastrun: 1755162168.772372 cost: 0.187411 loop:26529, slowcnt: 1 [2025-08-14 09:02:49.021269] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:49.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:49.727639] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=705298][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:49.212|207=706.37ms|210=1us|222=1us|total=706.40ms) [2025-08-14 09:02:49.727667] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=-4389] timer task use too much time(end=1755162169727666, start=1755162169021268, delta=706398) [2025-08-14 09:02:49.727684] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=435333, expected_run_ts_us_=1755162169292349, ObClockGenerator::getRealClock()=1755162169727683, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:49.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:49.073007] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:48.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:49.073292] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:49.727743] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=653458][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:49.244928] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:49.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:49.727761] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=482101][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:49.244|207=482.83ms|210=0us|222=1us|total=482.85ms) [2025-08-14 09:02:49.727772] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task use too much time(end=1755162169727772, start=1755162169244926, delta=482846) [2025-08-14 09:02:49.619740] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:49.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:49.700748] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [556][T1_OBJLockGC][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fafd0, caller:ob_table_lock_service.cpp:start:276, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE6$_1273, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fb2b0, time_wheel:0x7f994b19db50, thread_pool:0x7f994b19d330, is_running:1, time_interval:10.00s, expected_run_time:09:02:49.696, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:49.727704] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:49.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:49.727723] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=653724][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:49.730|207=654.72ms|210=0us|222=1us|total=654.73ms) [2025-08-14 09:02:49.729399] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1672][errcode=-4389] timer task use too much time(end=1755162169729398, start=1755162169073005, delta=656393) [2025-08-14 09:02:49.727779] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] task scheduled too delay(scheduled_delay=483145, expected_run_ts_us_=1755162169244633, ObClockGenerator::getRealClock()=1755162169727778, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:49.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:49.729423] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1640][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:49.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:49.727796] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=107890][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:49.619|207=108.06ms|210=0us|222=1us|total=108.07ms) [2025-08-14 09:02:49.729463] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=1664][errcode=-4389] timer task use too much time(end=1755162169729462, start=1755162169619737, delta=109725) [2025-08-14 09:02:48.768204] WDIAG [SERVER.OMT] process_request (ob_th_worker.cpp:242) [998][T1002_L0_G0][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4009] process request fail(ret=-4009) [2025-08-14 09:02:48.777259] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=741][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.767, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:49.730961] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=952257][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:48.777|207=953.70ms|210=1us|222=2us|total=953.72ms) [2025-08-14 09:02:49.730981] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4389] timer task use too much time(end=1755162169730981, start=1755162168777257, delta=953724) [2025-08-14 09:02:49.730992] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=951591, expected_run_ts_us_=1755162168779400, ObClockGenerator::getRealClock()=1755162169730992, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:49.731008] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:49.731021] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=952688, expected_run_ts_us_=1755162168778332, ObClockGenerator::getRealClock()=1755162169731021, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:49.731033] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:49.731045] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=952463, expected_run_ts_us_=1755162168778581, ObClockGenerator::getRealClock()=1755162169731045, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:49.731056] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:49.731068] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=960913, expected_run_ts_us_=1755162168770154, ObClockGenerator::getRealClock()=1755162169731068, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.770, task_priority:1, with_handle_protected:0}) [403]OB_MALLOC COST TOO MUCH TIME, cost_time=952565, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=952525], seq:[0, 1, 2] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=952052, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=952012], seq:[0, 1, 2] [409]OB_MALLOC COST TOO MUCH TIME, cost_time=958550, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=958543], seq:[0] [2025-08-14 09:01:35.725872] [959][T1002_L0_G0][T1002][YB42AC4D0007-00063C4962CA306D-0-0] LOGGER COST TOO MUCH TIME, cost: 1029348, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1029343, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x261b569c 0x261b52ae 0x93badc4 0x927f06e 0x926c154 0x12c5a0c9 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:49.731118] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=590] get_min_unreplayed_log_info(lsn={lsn:5282737655}, scn={val:1755100947825033001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161624848121, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:13063726, ret_code_:0}}, next_to_submit_lsn_:{lsn:5282737655}, next_to_submit_scn_:{val:1755100947825033001, v:0}, base_lsn_:{lsn:4831543296}, base_scn_:{val:1755063445733337001, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7605000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:5282737655}, end_lsn:{lsn:5282737655}, read_buf:{buf_len_:3747840, buf_:0x7f98f7605000}, block_size:67104768, log_storage_:0x7f993e529bf0, log_storage_:{log_tail:{lsn:5282737655}, readable_log_tail:{lsn:5282737655}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:5234171904}, min_scn:{val:1755152244802535004, v:0}, curr_block_id:78, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:206, min_block_id:42, max_block_id:79, curr_writable_block_id:78}, logical_block_size_:67104768, curr_block_writable_size_:18539017, block_header_serialize_buf_:0x7f993e52a8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:85, read_io_size:301367296, read_disk_cost_ts:2364604, start_lsn:{lsn:4831543296}}}}}}) [2025-08-14 09:02:49.731660] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=69] get wrs ts(ls_id={id:1}, delta=621992763, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:49.731675] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=2, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:49.731705] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=621993186, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=253139, succ_cluster_heartbeat_count=2206, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161000606550000, v:0}, local_cluster_delta=1169125152, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:02:49.731735] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161101770923, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755161572801635) [2025-08-14 09:02:49.732202] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=621992763, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:49.732212] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:49.728317] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=45] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:48.845248] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=25459][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc5378, timer:0x7f9970ecbf90, task:0x564d6cd25100, task_type:"N9oceanbase10rootserver23ObIngressBWAllocServiceE", scheduled_time:1755162167753034, pushed_time:0, delay:3000000}, running_task_token={this:0x7f9970fc5458, timer:0x7f9970ecbf90, task:0x564d6cd25880, task_type:"N9oceanbase10rootserver18ObSSNTAllocServiceE", scheduled_time:1755162050765815, pushed_time:1755162164753047, delay:3000000}, total_delay=1092213, threashold=1000000, tenant_id=500) [2025-08-14 09:02:48.778938] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=228] gc stale ls task succ [857]OB_MALLOC COST TOO MUCH TIME, cost_time=903552, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=42, WASH_BLOCK_END=903510], seq:[0, 1, 2] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=901194, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=50, WASH_BLOCK_END=901143], seq:[0, 1, 2] [737]OB_MALLOC COST TOO MUCH TIME, cost_time=881245, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=39, WASH_BLOCK_END=881206], seq:[0, 1, 2] [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=946982, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=50, WASH_BLOCK_END=946932], seq:[0, 1, 2] [876]OB_MALLOC COST TOO MUCH TIME, cost_time=865312, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=44, WASH_BLOCK_END=865268], seq:[0, 1, 2] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=956527, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=36, WASH_BLOCK_END=956457, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=32], seq:[0, 1, 2, 3, 4] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=794602, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=53, WASH_BLOCK_END=794548], seq:[0, 1, 2] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=944757, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=39, WASH_BLOCK_END=944718], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=770088, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=36, WASH_BLOCK_END=770052], seq:[0, 1, 2] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=956020, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=52, WASH_BLOCK_END=955968], seq:[0, 1, 2] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=956007, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=39, WASH_BLOCK_END=955967], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=713571, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=51, WASH_BLOCK_END=713520], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=646773, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=36, WASH_BLOCK_END=646736], seq:[0, 1, 2] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=512772, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=44, WASH_BLOCK_END=512728], seq:[0, 1, 2] [381]OB_MALLOC COST TOO MUCH TIME, cost_time=494742, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=43, WASH_BLOCK_END=494699], seq:[0, 1, 2] [513]OB_MALLOC COST TOO MUCH TIME, cost_time=403398, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=403358], seq:[0, 1, 2] [717]OB_MALLOC COST TOO MUCH TIME, cost_time=940092, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=34, WASH_BLOCK_END=939890, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=168], seq:[0, 1, 2, 3, 4] [409]OB_MALLOC COST TOO MUCH TIME, cost_time=965541, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=960239], seq:[0, 1] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=961119, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=33, WASH_BLOCK_END=961083, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:49.731079] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:49.732761] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1674][errcode=0] task scheduled too delay(scheduled_delay=949109, expected_run_ts_us_=1755162168783651, ObClockGenerator::getRealClock()=1755162169732761, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72716b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a4910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:49.732781] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72716b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a4910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:49.732792] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=944764, expected_run_ts_us_=1755162168788027, ObClockGenerator::getRealClock()=1755162169732792, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:49.732800] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:48.784513] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=16] current monitor number(seq_=-1) [2025-08-14 09:02:48.937182] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=20] PNIO [ratelimit] time: 1755162168937180, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007614, add_bytes: 0 [2025-08-14 09:02:49.329469] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=36][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a4e38, timer:0x7f9967161f10, task:0x7f994b3f4c40, task_type:"N9oceanbase7obmysql20ObConstructQueueTaskE", scheduled_time:1755162048238296, pushed_time:0, delay:500000}, running_task_token={this:0x7f994e1a4d58, timer:0x7f9967161f10, task:0x7f994b3f4bf0, task_type:"N9oceanbase7obmysql15ObEliminateTaskE", scheduled_time:1755162042245325, pushed_time:1755162047738501, delay:500000}, total_delay=121091171, threashold=1000000, tenant_id=1) [2025-08-14 09:02:49.712774] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fb3b0, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fb690, time_wheel:0x7f99589fd990, thread_pool:0x7f9959fff170, is_running:1, time_interval:2.00s, expected_run_time:09:02:49.698, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.792082] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=21] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:49.733653] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=940142] temporary file module use no blocks [2025-08-14 09:02:49.733664] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:49.733676] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:49.733687] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623070231774, flush_io_finished_ret_:0, flush_io_finished_round_:26325, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:48.830261] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=90][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:48.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:49.733783] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=902153][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:48.830|207=903.52ms|210=0us|222=2us|total=903.54ms) [2025-08-14 09:02:49.732816] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] task scheduled too delay(scheduled_delay=944549, expected_run_ts_us_=1755162168788267, ObClockGenerator::getRealClock()=1755162169732816, ret=0, ret="OB_SUCCESS", *this={this:0x7f98918100b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:49.018491] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:49.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:48.866314] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [876][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E999C16-0-0] [lt=20] ObArchiveSender is running(thread_index=3) [2025-08-14 09:02:49.685922] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=45546][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:1, time_interval:3.00s, expected_run_time:09:02:49.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:49.733802] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4389] timer task use too much time(end=1755162169733802, start=1755162168830260, delta=903542) [2025-08-14 09:02:49.720269] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:49.737645] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17348][errcode=0] epoll handler too much time :17389 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:48.762194] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:155) [409][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=34] unlock session of tenant(conn.sessid_=3221590940, proxy_sessid=0, conn.tenant_id_=1002) [2025-08-14 09:02:49.735727] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=2903][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918100b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:49.737954] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=2219][errcode=0] task scheduled too delay(scheduled_delay=956741, expected_run_ts_us_=1755162168781212, ObClockGenerator::getRealClock()=1755162169737954, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:49.737979] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:49.721799] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:49.726590] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=30] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:49.737318] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=717737][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:49.184|207=718.83ms|210=1us|222=1us|total=718.84ms) [2025-08-14 09:02:49.738074] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16250][errcode=0] epoll handler too much time :16287 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:49.737992] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=944197, expected_run_ts_us_=1755162168793794, ObClockGenerator::getRealClock()=1755162169737991, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.793, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:49.738979] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=985][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.793, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:49.738999] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] task scheduled too delay(scheduled_delay=946598, expected_run_ts_us_=1755162168792400, ObClockGenerator::getRealClock()=1755162169738998, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180fe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:49.737521] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=194][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1002}) [2025-08-14 09:02:49.739850] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=2325] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1002}) [2025-08-14 09:02:49.738525] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11912] temporary file module use no blocks [2025-08-14 09:02:49.738631] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1311][errcode=-4389] timer task use too much time(end=1755162169738631, start=1755162169018489, delta=720142) [2025-08-14 09:02:49.739010] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180fe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:49.746434] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7408][errcode=0] task scheduled too delay(scheduled_delay=940686, expected_run_ts_us_=1755162168805747, ObClockGenerator::getRealClock()=1755162169746434, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.805, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:49.746477] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.805, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:49.746490] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=933934, expected_run_ts_us_=1755162168812555, ObClockGenerator::getRealClock()=1755162169746490, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73927b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.812, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:49.746556] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=232] get wrs ts(ls_id={id:1}, delta=420000829, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:49.746578] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:49.747160] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1001}, delta=425014198, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:49.747168] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:49.747191] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=12] gc stale ls task succ [575]OB_MALLOC COST TOO MUCH TIME, cost_time=942214, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=43, WASH_BLOCK_END=909365, ALLOC_BLOCK#start=12, ALLOC_BLOCK#end=30131], seq:[0, 1, 2, 3, 4] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=981861, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=981853], seq:[0] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=983365, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1368, WASH_BLOCK_END=107, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=981884], seq:[0, 1, 2, 3, 4] [2025-08-14 09:01:40.625248] EDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EE9F-0-0] [lt=12][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1107207, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1107194, ObMalloc#start=1, ObMalloc#end=4, ALLOC_END=0, APPEND_END=8], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ed35c 0x9715e63 0x971553a 0x9714b16 0x971487f 0x1dc1f9ca 0x1dd7a832 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [567]OB_MALLOC COST TOO MUCH TIME, cost_time=892167, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:02:49.753841] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:49.755806] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:49.757336] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:49.760919] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=20] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:49.741932] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=3400] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:49.742087] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=3447][errcode=0] task scheduled too delay(scheduled_delay=736599, expected_run_ts_us_=1755162169005487, ObClockGenerator::getRealClock()=1755162169742086, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:49.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:49.763765] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:49.746501] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73927b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.812, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:49.763852] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17322][errcode=0] task scheduled too delay(scheduled_delay=948385, expected_run_ts_us_=1755162168815466, ObClockGenerator::getRealClock()=1755162169763852, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7399090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.815, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:49.763879] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7399090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.815, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:49.762076] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=20109] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:49.762553] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=20434][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:49.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:49.764837] INFO [COMMON] loop_running_dag_net_list (ob_tenant_dag_scheduler.cpp:3845) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] loop running dag net list(ret=0, running_dag_net_list_size=0, slow_dag_net_cnt=0, finished_dag_net_list_size=0) [2025-08-14 09:02:49.764847] INFO [COMMON] loop_finished_dag_net_list (ob_tenant_dag_scheduler.cpp:3868) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] loop finsihed dag net list(ret=0, finished_dag_net_list_size=0) [2025-08-14 09:02:48.821612] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=11] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:49.765290] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=942244] tx gc loop thread is running(MTL_ID()=1) [2025-08-14 09:02:49.765297] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=6] try gc retain ctx [2025-08-14 09:02:49.765301] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=5] try to retry start_working [2025-08-14 09:02:49.764411] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=2331] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623071163965, flush_io_finished_ret_:0, flush_io_finished_round_:26337, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:49.763889] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=947591, expected_run_ts_us_=1755162168816298, ObClockGenerator::getRealClock()=1755162169763889, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:49.765337] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1444][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:49.765352] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=938023, expected_run_ts_us_=1755162168827329, ObClockGenerator::getRealClock()=1755162169765352, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73954b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.827, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:49.764442] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1882][errcode=0] task scheduled too delay(scheduled_delay=675125, expected_run_ts_us_=1755162169089316, ObClockGenerator::getRealClock()=1755162169764442, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:49.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:49.766871] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:02:49.766907] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:02:49.770445] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:49.774376] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1672][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:49.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:49.777442] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=1084] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:49.778978] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=952118] gc stale ls task succ [2025-08-14 09:02:49.804578] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:49.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:49.833163] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=167] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:49.833618] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=403461][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a5ed8, timer:0x7f994b3feee0, task:0x7f994b3fef68, task_type:"N9oceanbase8observer18ObTableLoadService25ObClientTaskAutoAbortTaskE", scheduled_time:1755162048762959, pushed_time:0, delay:1000000}, running_task_token={this:0x7f994e1a5df8, timer:0x7f994b3feee0, task:0x7f994b3fef50, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162023758648, pushed_time:1755162122757495, delay:1000000}, total_delay=121070658, threashold=1000000, tenant_id=1) [2025-08-14 09:02:49.837269] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=420095706, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:49.930325] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:49.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:49.933216] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=15] tx gc loop thread is running(MTL_ID()=1001) [2025-08-14 09:02:49.933230] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=13] try gc retain ctx [2025-08-14 09:02:49.933245] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=15] try to retry start_working [2025-08-14 09:02:49.933256] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=4] get_max_applied_scn(scn={val:1755161719775179002, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:45, palf_committed_end_lsn_:{lsn:8461098908}, palf_committed_end_scn_:{val:1755161719775179002, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161719775179002, v:0}}) [2025-08-14 09:02:49.933277] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=16] get_min_unreplayed_log_info(lsn={lsn:8461098908}, scn={val:1755100947600130001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161920778323, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:23084275, ret_code_:0}}, next_to_submit_lsn_:{lsn:8461098908}, next_to_submit_scn_:{val:1755100947600130001, v:0}, base_lsn_:{lsn:7515734016}, base_scn_:{val:1753840867171901045, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6e05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:8461098908}, end_lsn:{lsn:8461098908}, read_buf:{buf_len_:3747840, buf_:0x7f98f6e05000}, block_size:67104768, log_storage_:0x7f99297a3bf0, log_storage_:{log_tail:{lsn:8461098908}, readable_log_tail:{lsn:8461098908}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8455200768}, min_scn:{val:1755160384851871002, v:0}, curr_block_id:126, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:215, min_block_id:93, max_block_id:127, curr_writable_block_id:126}, logical_block_size_:67104768, curr_block_writable_size_:61206628, block_header_serialize_buf_:0x7f99297a48b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:198, read_io_size:699822080, read_disk_cost_ts:4178035, start_lsn:{lsn:7515734016}}}}}}) [2025-08-14 09:02:49.944834] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=794647] PNIO [ratelimit] time: 1755162169944833, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007653, add_bytes: 0 [2025-08-14 09:02:49.737482] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=774524][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:48.961|207=775.70ms|210=1us|222=1us|total=775.71ms) [2025-08-14 09:02:50.722364] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=983391][errcode=-4389] timer task use too much time(end=1755162170722362, start=1755162168961781, delta=1760581) [2025-08-14 09:02:50.722381] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=967110, expected_run_ts_us_=1755162169755269, ObClockGenerator::getRealClock()=1755162170722379, ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:49.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:50.722419] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:49.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:50.722757] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EE9F-0-0] [lt=1107207][errcode=0] cost too much time:mds_tenant_service.cpp:static int oceanbase::storage::mds::ObTenantMdsService::for_each_mds_table_in_ls(oceanbase::storage::ObLS &, const ObFunction &), (this=|threshold=10.00s|start at 09:01:40.603|280=0us|291=0us|total=70.12s) [2025-08-14 09:02:50.722833] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EE9F-0-0] [lt=18] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:02:50.722844] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EE9F-0-0] [lt=7][errcode=0] cost too much time:mds_tenant_service.cpp:static int oceanbase::storage::mds::ObTenantMdsService::for_each_ls_in_tenant(const ObFunction &), (this=|threshold=3.00s|start at 09:01:40.603|207=0us|211=5us|218=3us|211=70.12s|218=3us|211=55us|total=70.12s) [2025-08-14 09:02:50.722864] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EE9F-0-0] [lt=18][errcode=-4389] run this task use too much time(*p_task_={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:0, time_interval:3.00s, expected_run_time:09:02:52.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:50.722877] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EE9F-0-0] [lt=11][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:02:50.722|total=70.12s) [2025-08-14 09:02:50.722890] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EE9F-0-0] [lt=8][errcode=-4016] task executed too delay(delay_time=58038466, *p_task_={this:0x7f99571630c0, caller:mds_tenant_service.cpp:mtl_start:117, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_145, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571633a0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:0, time_interval:15.00s, expected_run_time:09:02:52.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:50.723319] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEA1-0-0] [lt=11] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:02:49.738498] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=965953] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3964) [2025-08-14 09:02:50.723364] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=983376][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:50.723386] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=6] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:50.723396] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3965) [2025-08-14 09:02:50.723401] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:50.723569] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:50.723595] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=25] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3966) [2025-08-14 09:02:50.723600] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:49.765362] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73954b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.827, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724400] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=957585][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:49.765|207=959.04ms|210=0us|222=2us|total=959.07ms) [2025-08-14 09:02:50.724421] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4389] timer task use too much time(end=1755162170724421, start=1755162169765351, delta=959070) [2025-08-14 09:02:50.724433] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1850748, expected_run_ts_us_=1755162168873684, ObClockGenerator::getRealClock()=1755162170724432, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.873, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724452] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:48.873, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724471] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1476298, expected_run_ts_us_=1755162169248172, ObClockGenerator::getRealClock()=1755162170724470, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.248, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724483] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.248, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724503] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=988973, expected_run_ts_us_=1755162169735529, ObClockGenerator::getRealClock()=1755162170724503, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7270030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a3b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.735, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724515] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7270030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a3b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.735, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.025880] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:50.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:50.724529] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=984622, expected_run_ts_us_=1755162169739906, ObClockGenerator::getRealClock()=1755162170724529, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ecb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.739, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724541] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ecb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.739, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724536] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=697593][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:50.258|207=698.65ms|210=0us|222=2us|total=698.68ms) [2025-08-14 09:02:50.724553] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=991793, expected_run_ts_us_=1755162169732759, ObClockGenerator::getRealClock()=1755162170724553, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7353730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7396990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724565] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7353730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7396990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724565] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=-4389] timer task use too much time(end=1755162170724564, start=1755162170025878, delta=698686) [2025-08-14 09:02:50.724586] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] task scheduled too delay(scheduled_delay=987724, expected_run_ts_us_=1755162169736861, ObClockGenerator::getRealClock()=1755162170724585, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724600] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724597] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] task scheduled too delay(scheduled_delay=432247, expected_run_ts_us_=1755162170292349, ObClockGenerator::getRealClock()=1755162170724597, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:50.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:50.724611] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=984455, expected_run_ts_us_=1755162169740156, ObClockGenerator::getRealClock()=1755162170724611, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ea30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724615] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:50.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:50.724623] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ea30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724636] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=970336, expected_run_ts_us_=1755162169754299, ObClockGenerator::getRealClock()=1755162170724636, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919afbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724647] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919afbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724661] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=966128, expected_run_ts_us_=1755162169758532, ObClockGenerator::getRealClock()=1755162170724661, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919854b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724673] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919854b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724683] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=965950, expected_run_ts_us_=1755162169758733, ObClockGenerator::getRealClock()=1755162170724683, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.720639] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:50.724693] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724704] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=965733, expected_run_ts_us_=1755162169758970, ObClockGenerator::getRealClock()=1755162170724704, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724715] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724725] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=971816, expected_run_ts_us_=1755162169752908, ObClockGenerator::getRealClock()=1755162170724725, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d45b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724735] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d45b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724745] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=958153, expected_run_ts_us_=1755162169766592, ObClockGenerator::getRealClock()=1755162170724745, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72747b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724755] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72747b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724767] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=958358, expected_run_ts_us_=1755162169766408, ObClockGenerator::getRealClock()=1755162170724767, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724784] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724795] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=949174, expected_run_ts_us_=1755162169775620, ObClockGenerator::getRealClock()=1755162170724794, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724805] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724815] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=949159, expected_run_ts_us_=1755162169775655, ObClockGenerator::getRealClock()=1755162170724814, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724826] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724836] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=952934, expected_run_ts_us_=1755162169771901, ObClockGenerator::getRealClock()=1755162170724836, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724846] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724856] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=951884, expected_run_ts_us_=1755162169772971, ObClockGenerator::getRealClock()=1755162170724855, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724873] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724885] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=939025, expected_run_ts_us_=1755162169785859, ObClockGenerator::getRealClock()=1755162170724885, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724895] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724905] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=939744, expected_run_ts_us_=1755162169785160, ObClockGenerator::getRealClock()=1755162170724904, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724914] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724925] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=920845, expected_run_ts_us_=1755162169804079, ObClockGenerator::getRealClock()=1755162170724925, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919836b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724942] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919836b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724953] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=894552, expected_run_ts_us_=1755162169830400, ObClockGenerator::getRealClock()=1755162170724953, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724963] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724973] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=893947, expected_run_ts_us_=1755162169831025, ObClockGenerator::getRealClock()=1755162170724972, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.831, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724984] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.831, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.724994] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=862861, expected_run_ts_us_=1755162169862133, ObClockGenerator::getRealClock()=1755162170724994, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.862, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.725004] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:49.862, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:49.765372] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=927][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:49.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:50.725045] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=958220][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:49.764|207=960.60ms|210=1us|222=1us|total=960.61ms) [2025-08-14 09:02:50.725055] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task use too much time(end=1755162170725055, start=1755162169764441, delta=960614) [2025-08-14 09:02:50.725064] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=0] task scheduled too delay(scheduled_delay=1520766, expected_run_ts_us_=1755162169204297, ObClockGenerator::getRealClock()=1755162170725064, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:49.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:50.725077] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:49.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:50.725096] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=965961, expected_run_ts_us_=1755162169759134, ObClockGenerator::getRealClock()=1755162170725095, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:49.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:50.725106] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:49.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:50.725120] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=719633, expected_run_ts_us_=1755162170005487, ObClockGenerator::getRealClock()=1755162170725120, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:50.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:50.725130] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:50.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:50.725967] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=622986373, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:50.725987] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:50.726632] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] get wrs ts(ls_id={id:1}, delta=622986373, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:50.726647] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:49.837290] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:49.840375] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=3822][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:49.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:50.726997] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=885278][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:49.840|207=886.62ms|210=0us|222=2us|total=886.64ms) [2025-08-14 09:02:50.727017] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4389] timer task use too much time(end=1755162170727016, start=1755162169840373, delta=886643) [2025-08-14 09:02:50.030419] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:49.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:50.727723] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=696246][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:50.304|207=697.30ms|210=0us|222=2us|total=697.32ms) [2025-08-14 09:02:50.727740] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] timer task use too much time(end=1755162170727739, start=1755162170030418, delta=697321) [2025-08-14 09:02:49.741953] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=22] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:02:49.738122] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [863][T1002_ArcSrv][T1002][YB42AC4D0007-00063C495E699C16-0-0] [lt=13602][errcode=0] long time to hold one guard(schema_mgr=0x7f988fae9820, tenant_id=1001, version=1753074886385104, cur_timestamp=1755162100757811, ref_timestamp=1755161936840739, lbt()="0x26a1806d 0x974f3e5 0x974e159 0x974e084 0x948fab3 0x97257c0 0x97256f4 0x9725691 0x1f5a4693 0x113c7c62 0x113e7767 0x113e5f96 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3") [2025-08-14 09:02:50.040389] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2907] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:02:50.730339] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=688863] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:02:50.730385] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:02:50.730435] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:02:50.730452] INFO [COMMON] print_io_status (ob_io_manager.cpp:2639) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4113, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:1861, total:1866) [2025-08-14 09:02:50.730472] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, param_config_:{memory_limit:536870912, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=865568, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:02:50.721947] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=859] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:50.730598] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3991] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:50.730609] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EEC1-0-0] [lt=3071][errcode=-4016] task executed too delay(delay_time=852817, *p_task_={this:0x7f99571fc8b0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcb90, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:0, time_interval:3.00s, expected_run_time:09:02:52.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:50.730736] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B929E-0-0] [lt=27][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162169831770) [2025-08-14 09:02:50.730769] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B929E-0-0] [lt=27][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162169831770}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162170726030}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162170726030}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:50.730840] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=302][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:50.730906] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [316][pnio1][T1][YB42AC4D0007-00063C49607B929E-0-0] [lt=12][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1, delta=4990498, last_succ_cluster_heartbeat_tstamp_=1755162165740408) [2025-08-14 09:02:50.731511] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=622992470, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:50.731539] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:50.733507] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=947029] current monitor number(seq_=-1) [2025-08-14 09:02:50.733560] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=945118] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:50.735755] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EEC3-0-0] [lt=26] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-08-14 09:02:50.736830] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=956133] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:50.736877] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=956327] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:50.737487] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=194] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:50.730518] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2826) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=35] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4114, [delay/us]: prepare:0, schedule:0, submit:600000000, device:1861, total:1866) [2025-08-14 09:02:50.735269] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=58] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:50.740355] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=5071] temporary file module use no blocks [2025-08-14 09:02:50.740369] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:50.740383] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:50.740405] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=21] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623072173514, flush_io_finished_ret_:0, flush_io_finished_round_:26326, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:49.740317] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6327][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:50.741627] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=999789][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:50.741641] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:50.741659] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:50.741679] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:01:49.731, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:01:42.751], idx=659, next_allow_print_ts="09:02:19.731") [2025-08-14 09:02:50.741691] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:50.741706] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:50.741718] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:50.741728] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:50.741741] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:50.741908] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:50.741921] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:49.735752] INFO [SQL.PC] need_check_schema_version (ob_plan_cache_value.cpp:2334) [1003][BlackListServic][T1][YB42AC4D0007-00063C495F09BD6A-0-0] [lt=35140] need check schema(new_schema_version=1753073979411448, cached_tenant_schema_version=1753073979411448, is_contain_synonym()=false, contain_sys_name_table_=true, is_contain_tmp_tbl()=false, is_contain_sys_pl_object()=false, need_check=true, constructed_sql_=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, ?) as weak_read_scn, nvl(b.migrate_status, ?) as migrate_status, nvl(b.tx_blocked, ?) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2025-08-14 09:02:50.743184] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1464) [14881][TimerWK4_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=31] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=4817133568, global_cache_size=147734528, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 509, 1, 1001, 1002]) [2025-08-14 09:02:50.743428] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6361] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:49.780252] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=631] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:50.743508] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=961801][errcode=0] epoll handler too much time :963265 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:49.802656] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=928814] gc stale ls task succ [2025-08-14 09:02:50.743869] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=181] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:50.743883] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162170743865], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:50.743899] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162170743865], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:50.743915] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162170743865]}) [2025-08-14 09:02:50.743927] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=68, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:02:50.743944] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:50.743955] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:50.743966] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=8] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:50.743982] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162170743965], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:50.743994] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162170743965]}) [2025-08-14 09:02:50.744194] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:50.744207] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:50.744220] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162170744190], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:50.744442] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=218] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162170744190]}) [2025-08-14 09:02:50.746089] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:50.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.746129] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918114b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:50.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.746143] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728aa90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:50.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.746156] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:50.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.746170] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:50.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.746183] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:50.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:49.741962] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1002930, , BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:49.741962] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=25] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:02:50.749393] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=23] gc stale ls task succ [2025-08-14 09:02:50.749760] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=934712] [sql nio session](*s={this:0x7f98f733f030, session_id:3221589417, trace_id:YB42AC4D0007-00063C496359B268-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:252}, err:0, last_decode_time:1755161813752329, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1778, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:50.749792] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=32] [sql nio session](*s={this:0x7f98f733e030, session_id:3221589980, trace_id:YB42AC4D0007-00063C496359B261-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:251}, err:0, last_decode_time:1755161748778070, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1681, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:50.749806] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=13] [sql nio session](*s={this:0x7f98f7332030, session_id:3221590132, trace_id:YB42AC4D0007-00063C496359B265-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:266}, err:0, last_decode_time:1755161768783802, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:2125, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:50.749828] INFO [COMMON] wash (ob_kvcache_store.cpp:529) [14881][TimerWK4_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=41] Wash time detail, (compute_wash_size_time=1972605, refresh_score_time=6433, wash_time=18, reclaim_time=0, reclaimed_size=0) [2025-08-14 09:02:50.738129] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7587] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:02:50.750273] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12120] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:50.750294] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:50.750305] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SERVER] net_in=0kB/s, net_out=0kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=1280000kB/s [2025-08-14 09:02:49.969308] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] EASY SLOW: thread: 7f99542fb300, lastrun: 1755162115.752503 cost: 54.216803 loop:26416, slowcnt: 456 [2025-08-14 09:02:49.993469] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=8156][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:49.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:50.750609] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=755993][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:49.993|207=757.14ms|210=0us|222=1us|total=757.15ms) [2025-08-14 09:02:50.750623] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] timer task use too much time(end=1755162170750623, start=1755162169993467, delta=757156) [2025-08-14 09:02:50.750638] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=506004, expected_run_ts_us_=1755162170244633, ObClockGenerator::getRealClock()=1755162170750638, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:50.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:50.750653] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:50.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:49.736297] EDIAG [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=2862][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1012932, , BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1c03a517 0x97a8d73 0x97a88d2 0x97a7481 0x97a6a43 0x97a42c5 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:49.736297] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=2862] get wrs ts(ls_id={id:1}, delta=449959899, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:50.750789] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1012932] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=2, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:50.725164] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [723][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=21] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=16, total_destructed_detector=16, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f99333562b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-08-14 09:02:50.750840] WDIAG [DETECT] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [723][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=25635][errcode=0] cost too much time:ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t), (this=|threshold=100.00ms|start at 09:02:50.750|total=959.85ms) [2025-08-14 09:02:49.756865] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15582][errcode=0] epoll handler too much time :15617 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:50.750942] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=992564] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:50.738166] INFO [STORAGE] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:123) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=888046] wead read timestamp is too old(timestamp={val:1755161749741193000, v:0}, gts_scn={val:1755161762721798002, v:0}, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1755161749741193000, v:0}}) [2025-08-14 09:02:50.749542] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=4] finish update min start scn(max_decided_scn={val:1755161547738516001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1755161493768447002, v:0}, keep_alive_scn_:{val:1755161493768447002, v:0}, update_ts_:1755162170730986}) [2025-08-14 09:02:50.741311] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=69] finish update min start scn(max_decided_scn={val:1755161719775179002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1755161287778691002, v:0}, keep_alive_scn_:{val:1755161287778691002, v:0}, update_ts_:1755162170741302}) [2025-08-14 09:02:50.751922] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119797572, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755162138745740) [2025-08-14 09:02:50.752539] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=450968630, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:50.752556] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:50.753171] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13231] get wrs ts(ls_id={id:1001}, delta=426015985, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:50.753190] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:50.753258] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=421002616, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:50.753269] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:50.753809] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1001}, delta=426015985, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:50.753826] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:50.755252] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1228] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:50.756462] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1344] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:50.756499] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:50.756525] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1160] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:50.756565] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=829] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:50.757835] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:50.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.757885] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:50.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.757902] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:50.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.757994] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:50.760312] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=14] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:49.743013] EDIAG [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1016244, , BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1af3d850 0x1af3c294 0x20071394 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:50.753933] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:50.760758] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [564][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=2020719] XA scheduler heartbeat task statistics(avg_time=3244) [2025-08-14 09:02:49.743013] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] [COMPACTION DAG STATUS] (start_time=1755162157777146, end_time=1755162169738930, dag_status={}) [2025-08-14 09:02:49.746513] EDIAG [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1014289, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1014279, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1af3d850 0x1af3c294 0x20071394 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:49.746513] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=10] [COMPACTION DAG STATUS] (start_time=1755162158742029, end_time=1755162169739794, dag_status={}) [2025-08-14 09:02:50.762405] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=41] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:50.764929] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] [COMPACTION DAG STATUS] (start_time=1755162159780765, end_time=1755162170764928, dag_status={}) [2025-08-14 09:02:50.761837] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:706) [14881][TimerWK4_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=53] Cache wash clean map node details(ret=0, clean_node_count=9, clean_time=0, clean_start_pos=440398, clean_num=31457) [2025-08-14 09:02:50.765818] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=916] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:50.767401] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=1574] temporary file module use no blocks [2025-08-14 09:02:50.767413] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:50.767427] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:50.767442] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623072205097, flush_io_finished_ret_:0, flush_io_finished_round_:26338, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:50.767474] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:50.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:50.767503] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:50.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:50.767896] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fc830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:50.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.767919] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:02:50.770281] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:50.771605] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:50.773894] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=7] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:02:50.773925] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=24] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:02:50.774014] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:02:50.778506] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:50.778651] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=160] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:50.778670] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16] temporary file module use no blocks [2025-08-14 09:02:50.778679] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:50.778690] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:50.778707] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=18] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623072218349, flush_io_finished_ret_:0, flush_io_finished_round_:26305, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:50.779052] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=9] gc stale ls task succ [2025-08-14 09:02:50.783187] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=200] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:02:50.802718] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=939462] gc stale ls task succ [2025-08-14 09:02:50.818219] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:50.802, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.818271] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=50][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:50.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.818287] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728f9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:50.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.822445] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:50.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:50.827093] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:50.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:50.828293] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eae10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:50.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.830950] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:50.831565] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=623091989, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:50.831592] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:50.838322] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:50.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.851999] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=451076816, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=252286, succ_cluster_heartbeat_count=1901, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161003593738002, v:0}, local_cluster_delta=1167258257, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:02:50.852711] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=51] get wrs ts(ls_id={id:1}, delta=451076613, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:50.852727] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:50.853344] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=421111864, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:50.853362] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:50.853968] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1001}, delta=426125233, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:50.853987] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:50.878557] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:50.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.878610] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=51][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:50.866, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:50.922506] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:50.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:50.927154] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:50.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:50.930973] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:50.931657] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=623191697, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:50.931679] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:50.939879] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a6a38, timer:0x7f9966fddee0, task:0x7f994f1b35a8, task_type:"N9oceanbase3omt11ObTenantSrs27TenantSrsUpdatePeriodicTaskE", scheduled_time:1755161929777851, pushed_time:0, delay:0}, running_task_token={this:0x7f994e1a65d8, timer:0x7f9966fddee0, task:0x7f994f1bbcc0, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161694824965, pushed_time:1755161925772608, delay:5000000}, total_delay=241162027, threashold=1000000, tenant_id=1) [2025-08-14 09:02:50.952673] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=451176731, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:50.952697] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:50.953347] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=421211779, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:50.953366] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:50.953997] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25] get wrs ts(ls_id={id:1001}, delta=426225148, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:50.954017] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:50.994852] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:50.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.019059] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:51.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.019093] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:51.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.020736] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:51.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.022561] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:50.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.027230] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:50.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.031032] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:02:51.031063] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:51.050657] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:02:51.050715] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:02:51.050758] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:02:51.050811] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:02:51.050831] INFO [COMMON] print_io_status (ob_io_manager.cpp:2639) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:3, norm_iops:0, bw:12784, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:4027, total:4033) [2025-08-14 09:02:51.050863] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, param_config_:{memory_limit:536870912, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=865568, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:02:51.051419] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=780058][errcode=0] EASY SLOW: thread: 7f99542fb300, lastrun: 1755162115.752503 cost: 55.298914 loop:26416, slowcnt: 459 [2025-08-14 09:02:51.100299] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:51.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.122622] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:51.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.127250] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:51.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.131098] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:51.139966] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=152] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:02:51.222672] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:51.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.227332] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:51.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.231139] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:51.242123] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:51.242170] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:51.242179] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:51.242203] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:51.242218] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:51.242269] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:51.242280] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:01:49.731, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:01:42.751], idx=659, next_allow_print_ts="09:02:19.731") [2025-08-14 09:02:51.242292] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:51.242306] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:51.242318] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:51.242335] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:51.242347] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:51.242507] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:51.242521] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:51.244687] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=16] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:51.244713] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162171244683], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:51.244738] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162171244683], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:51.244759] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162171244683]}) [2025-08-14 09:02:51.244778] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=16] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:51.244791] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162171244777], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:51.244815] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=23] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162171244777]}) [2025-08-14 09:02:51.244998] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:51.245015] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.245027] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162171244994], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:51.245052] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=23] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162171244994]}) [2025-08-14 09:02:51.246322] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:51.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.256405] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:51.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.272895] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=469][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:51.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.303113] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:51.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.322697] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:51.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.327425] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:51.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.331200] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:51.331256] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.422722] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:51.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.427465] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:51.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.431243] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:51.497993] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:51.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.522762] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:51.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.524448] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:51.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.527552] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:51.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.531302] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:51.622799] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:51.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.627616] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:51.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.631360] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:51.631432] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=306][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:51.555, task_priority:1, with_handle_protected:1}) [729]OB_MALLOC COST TOO MUCH TIME, cost_time=666152, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16448, [2025-08-14 09:02:51.211955] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:51.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.719467] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=20833][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fb3b0, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fb690, time_wheel:0x7f99589fd990, thread_pool:0x7f9959fff170, is_running:1, time_interval:2.00s, expected_run_time:09:02:51.698, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.719693] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4049] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:51.721964] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8592] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:51.722876] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:51.655, task_priority:1, with_handle_protected:1}) [416]OB_MALLOC COST TOO MUCH TIME, cost_time=671463, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=671461], seq:[0] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=691985, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:02:51.724902] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=623984613, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:51.724929] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:51.725983] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=623987140, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:51.726006] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [857]OB_MALLOC COST TOO MUCH TIME, cost_time=673448, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=775336, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=775334], seq:[0] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=775356, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=775352], seq:[0, 1] [729]OB_MALLOC COST TOO MUCH TIME, cost_time=678417, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=666170], seq:[0, 1] [2025-08-14 09:02:51.727914] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=421985623, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:51.727937] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:51.728488] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1001}, delta=426998992, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:51.728506] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:51.728544] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=421985623, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:51.728555] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:51.729088] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1001}, delta=427001092, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:51.729108] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:50.952759] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=14] PNIO [ratelimit] time: 1755162170952757, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007924, add_bytes: 0 [2025-08-14 09:02:49.738947] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:244) [409][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=974203] connection close(sessid=3221590940, proxy_sessid=0, tenant_id=1002, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42AC4D0007-00063C4962CA306F-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:18, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{receive:obp_mysql_header_{len_:981, rec_:981, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[2]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{send:obp_mysql_header_{pkt_num_:21, seq_:23}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{len_:9, seq_:24}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{pkt_num_:1, seq_:25}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{len_:9, seq_:26}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[7]:{receive:obp_mysql_header_{len_:18, rec_:18, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[8]:{send:obp_mysql_header_{len_:11, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[9]:{send:obp_mysql_header_{len_:11, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[10]:{send:obp_mysql_header_{len_:11, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[11]:{send:obp_mysql_header_{len_:11, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[12]:{receive:obp_mysql_header_{len_:39, rec_:39, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[13]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[14]:{send:obp_mysql_header_{pkt_num_:1, seq_:3}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[15]:{send:obp_mysql_header_{len_:9, seq_:4}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[16]:{send:obp_mysql_header_{pkt_num_:1, seq_:5}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[17]:{send:obp_mysql_header_{len_:9, seq_:6}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-08-14 09:02:51.733564] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:274) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=22733][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=533) [2025-08-14 09:02:51.733614] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:953) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=48][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2025-08-14 09:02:51.733635] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=14] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:51.733646] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.733660] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=13][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:51.733676] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162171244777]}) [2025-08-14 09:02:51.733717] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=27] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:51.733752] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [280][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=965612][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:11, tid:497}]) [2025-08-14 09:02:51.050915] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2826) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=47] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:4, avg_bw:12783, [delay/us]: prepare:0, schedule:0, submit:600000000, device:4027, total:4033) [2025-08-14 09:02:51.733864] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=681902] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:02:51.733887] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:51.733900] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:51.733910] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SERVER] net_in=0kB/s, net_out=0kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=1280000kB/s [2025-08-14 09:02:51.734399] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=11] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:51.734416] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.734428] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=12][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:51.736284] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=10] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:51.736547] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=255][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.728537] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:51.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.734788] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72756b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.736750] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:51.736748] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1953][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.724, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.736771] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a36b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.729947] INFO [COMMON] wash (ob_kvcache_hazard_domain.h:326) [14881][TimerWK4_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5362] allocator wash time (wash_time_us=26, freed_blocks_num=0, memory_efficiency=8.976912630149388450e-01) [2025-08-14 09:02:51.736627] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=79][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:51.719403] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=506677][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:51.211|207=507.44ms|210=0us|222=1us|total=522.23ms) [2025-08-14 09:02:51.736998] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=17567][errcode=-4389] timer task use too much time(end=1755162171736997, start=1755162171211953, delta=525044) [2025-08-14 09:02:51.737014] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=477878, expected_run_ts_us_=1755162171259134, ObClockGenerator::getRealClock()=1755162171737013, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:51.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.737031] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:51.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.737034] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=12] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:51.737044] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=481557, expected_run_ts_us_=1755162171255487, ObClockGenerator::getRealClock()=1755162171737044, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:51.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.737056] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:51.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.737073] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=147756, expected_run_ts_us_=1755162171589316, ObClockGenerator::getRealClock()=1755162171737073, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:51.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.737080] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=11] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:51.737087] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:51.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.737672] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=359] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:51.737688] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.737700] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=11][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:50.765872] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=10142] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136829134139001, v:0}, clog_base_lsn:{lsn:8186781696}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1753840867372367034, v:0}, all_id_meta:{id_meta:[{limited_id:1755161729775179000, latest_log_ts:{val:1755161719775179001, v:0}}, {limited_id:18000001, latest_log_ts:{val:1755157440846556001, v:0}}, {limited_id:60000001, latest_log_ts:{val:1755135657266496001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:46, palf_env_:0x7f99321fa030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-08-14 09:02:51.738639] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=971289] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:51.738668] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=15] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:50.765792] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1906] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136826952869001, v:0}, clog_base_lsn:{lsn:5099962368}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1755071646732636000, v:0}, all_id_meta:{id_meta:[{limited_id:1755161555795565000, latest_log_ts:{val:1755161545796642001, v:0}}, {limited_id:22000001, latest_log_ts:{val:1755135645787806003, v:0}}, {limited_id:60000001, latest_log_ts:{val:1755135646400653001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:50, palf_env_:0x7f9950bf4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-08-14 09:02:51.738702] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=971438] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:51.738717] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:51.738731] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=16] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:51.738740] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.738753] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=13][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:51.732203] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=451954794, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:51.739148] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6930] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=2, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:51.739338] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [412][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=1908787] [sql nio session](*s={this:0x7f98f733a030, session_id:3221590255, trace_id:YB42AC4D0007-00063C49651A3232-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:257}, err:0, last_decode_time:1755161952736711, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1716, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:51.739382] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:51.739396] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.739407] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=9][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:51.223680] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:51.739425] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=514964] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3967) [2025-08-14 09:02:51.739436] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:51.739463] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=16] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:51.739474] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=11] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3968) [2025-08-14 09:02:51.739482] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:51.739670] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:51.739681] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=11] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3969) [2025-08-14 09:02:51.739689] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:51.739718] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=451963464, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:51.739736] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:51.740032] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [759][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB90D4-0-0] [lt=20][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1001, svr="172.77.0.7:2882", version={val:1755161719775179002, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162171052106) [2025-08-14 09:02:51.740061] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [759][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB90D4-0-0] [lt=22][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1001, req={req_server:"172.77.0.7:2882", version:{val:1755161719775179002, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162171052106}, twrs={inited:true, tenant_id:1001, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162171739177}, server_version_for_stat:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162171739177}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:51.740291] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [316][pnio1][T1001][YB42AC4D0007-00063C4960CB90D4-0-0] [lt=13][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1001, delta=3913380, last_succ_cluster_heartbeat_tstamp_=1755162167826890) [2025-08-14 09:02:51.740327] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:51.740336] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.740346] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=9][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:51.741150] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:51.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.741938] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:51.742260] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:51.742714] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=47] get wrs ts(ls_id={id:1}, delta=451966376, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:51.742730] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:51.743736] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:51.744392] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:51.744553] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:51.744608] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:51.744627] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:51.744657] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:51.744676] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:51.744699] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:51.744717] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:01:49.731, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:01:42.751], idx=659, next_allow_print_ts="09:02:19.731") [2025-08-14 09:02:51.744734] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:51.744755] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:51.744782] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:51.744799] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:51.744818] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:51.744994] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:51.745017] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:51.745274] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=14] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:51.745292] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162171745270], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:51.745318] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162171745270], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:51.745332] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162171745270]}) [2025-08-14 09:02:51.745343] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162171745270]}) [2025-08-14 09:02:51.745356] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=11] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:51.745365] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162171745355], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:51.745381] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162171745355]}) [2025-08-14 09:02:51.745543] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:51.745555] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=11] get number(ret=-4038, service_type_=0, range=1, base_id=1755162171743371000, start_id=0, end_id=140295968341544) [2025-08-14 09:02:51.745567] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.745590] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162171745539], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:51.745603] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162171745539]}) [2025-08-14 09:02:51.745621] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162171745539]}) [2025-08-14 09:02:51.747939] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919bfbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aa690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.747976] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919911b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196cc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.747992] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d40b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.748005] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.748076] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:51.748092] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.748103] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=11][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:51.748711] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:51.748727] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.748737] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=9][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:51.744418] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=21] temporary file module use no blocks [2025-08-14 09:02:51.748810] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=4383] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:51.748824] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:51.748837] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623073180200, flush_io_finished_ret_:0, flush_io_finished_round_:26327, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:51.749368] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:51.749457] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:43.763830] EDIAG [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [854][T1002_ObXAHbWor][T1002][Y0-0000000000000000-0-0] [lt=6921][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2989382, owner: ObLog, click_count: 5, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=2989367, ALLOC_END=2, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b64d8ba 0x1b64d154 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:51.750988] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:51.748018] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.751192] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=3165][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.751215] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.719419] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=28] log compressor cycles once. (ret=0, cost_time=949035, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25898303488, fast_delete_log_mode=false) [2025-08-14 09:02:51.758363] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.744, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.758407] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aaf90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.760294] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=19] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:51.758419] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aba10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.761192] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=2764][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.761223] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196cb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.761259] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.749436] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:51.763713] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14253][errcode=0] epoll handler too much time :14284 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:51.752401] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=8] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:51.765186] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=12764][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.765205] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=18][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:51.753286] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:51.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.765339] INFO print_tenant_cache_info (ob_kvcache_inst_map.cpp:528) [14881][TimerWK4_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6851] [CACHE] tenant_id= 1 cache memory info: [CACHE] tenant_id= 1 | cache_name= schema_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 190 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= tablet_table_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 809 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= index_block_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 252 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= user_block_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 164 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= user_row_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 1645 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= bf_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 11 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= fuse_row_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 118 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= storage_meta_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 344 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= tx_data_kv_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 3 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= log_kv_cache | cache_size= 4160512 | cache_store_size= 2080768 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 28 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= opt_table_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 158 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= opt_column_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 264 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= opt_ds_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 5 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= opt_system_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 1 | hold_size= 0 [2025-08-14 09:02:51.765433] INFO print_tenant_cache_info (ob_kvcache_inst_map.cpp:528) [14881][TimerWK4_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=17] [CACHE] tenant_id= 1001 cache memory info: [CACHE] tenant_id= 1001 | cache_name= index_block_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 156 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= user_block_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 89 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= user_row_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 1745 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= bf_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 8 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= fuse_row_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 128 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= storage_meta_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 948 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= tx_data_kv_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 2 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= log_kv_cache | cache_size= 4160512 | cache_store_size= 2080768 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 8 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= opt_table_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 117 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= opt_column_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 399 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= opt_system_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 1 | hold_size= 0 [2025-08-14 09:02:51.765499] INFO print_tenant_cache_info (ob_kvcache_inst_map.cpp:528) [14881][TimerWK4_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] [CACHE] tenant_id= 1002 cache memory info: [CACHE] tenant_id= 1002 | cache_name= index_block_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 929 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= user_block_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 144 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= user_row_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 279 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= bf_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 7 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= fuse_row_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 242 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= storage_meta_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 950 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= tx_data_kv_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 4 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= log_kv_cache | cache_size= 4160512 | cache_store_size= 2080768 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 13 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= opt_table_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 197 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= opt_column_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 272 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= opt_ds_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 32 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= opt_system_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_retired_size= 0 | cache_map_size= 2079744 | kv_cnt= 1 | hold_size= 0 [2025-08-14 09:02:51.766054] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=11] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:51.766075] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=21][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.766089] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=14][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:51.754386] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6522] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:51.766373] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11968][errcode=0] epoll handler too much time :11995 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:51.755352] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:51.766525] WDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11155][errcode=0] epoll handler too much time :11181 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:51.768002] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=21] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:51.769027] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:51.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.769067] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:51.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.757305] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:51.769293] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11969][errcode=0] epoll handler too much time :11997 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:51.763330] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:51.769322] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.751, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.769379] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=15] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:51.769393] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.769425] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=32][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:51.769994] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:51.770011] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.770021] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=10][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:51.770260] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:51.770592] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:51.770606] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.770616] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=9][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:51.771189] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:51.771203] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.771212] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=9][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:51.771781] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=8] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:51.771795] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.771804] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=9][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:51.772375] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.772442] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:51.772954] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.773531] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.774135] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=41][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.774710] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.774762] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10] LogLoopThread round_cost_time(us)(round_cost_time=12) [2025-08-14 09:02:51.774787] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:02:51.774817] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:02:51.775285] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.775859] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.776551] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.777137] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.777716] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.778298] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.778491] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:51.778837] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.778858] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=69] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:51.778880] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=17] temporary file module use no blocks [2025-08-14 09:02:51.778889] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:51.778903] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:51.779063] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:51.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.779089] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=15] gc stale ls task succ [2025-08-14 09:02:51.779371] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.779399] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.779974] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.780523] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=17][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.781075] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=17][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.781655] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.782211] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:51.783304] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=16] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:02:51.789400] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919acf10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.789434] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.799461] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919abe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.799492] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.799513] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199fe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919acd90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.799527] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e80b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.799544] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a31b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ede10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.809246] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:51.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.819576] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.806, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.819617] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.819628] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.819653] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.800, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.819667] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728e330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7290690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.805, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.819680] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.803, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.822941] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:51.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.828602] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=8099][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:51.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.838735] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=8] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:51.838787] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=6] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:51.839695] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.824, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.842828] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=452066815, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:51.842855] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:51.843795] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:51.849717] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.832, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.879884] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ade10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.864, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.879921] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.879939] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919adc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.879960] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919adb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.889947] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72865b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.878, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.909352] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=320][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a5998, timer:0x7f9935f8df20, task:0x7f99200f5a78, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr15MediumCheckTaskE", scheduled_time:1755161721762022, pushed_time:0, delay:20000000}, running_task_token={this:0x7f991b7a58b8, timer:0x7f9935f8df20, task:0x7f99200f5a38, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr14MediumLoopTaskE", scheduled_time:1755161718796975, pushed_time:1755161718805054, delay:120000000}, total_delay=450147329, threashold=1000000, tenant_id=1002) [2025-08-14 09:02:51.909396] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6418, timer:0x7f994b1dfb70, task:0x7f9920110068, task_type:"N9oceanbase5share22ObIndexUsageReportTaskE", scheduled_time:1755161727099859, pushed_time:0, delay:900000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=444809536, threashold=1000000, tenant_id=1002) [2025-08-14 09:02:51.909412] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6bf8, timer:0x7f994b1dfb70, task:0x7f99201e29f8, task_type:"N9oceanbase5table21ObTableGroupCommitMgr19ObTableGroupOpsTaskE", scheduled_time:1755161732717636, pushed_time:0, delay:1000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=439191776, threashold=1000000, tenant_id=1002) [2025-08-14 09:02:51.909426] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6878, timer:0x7f994b1dfb70, task:0x7f9920110178, task_type:"N9oceanbase5share27ObIndexUsageRefreshConfTaskE", scheduled_time:1755161733727027, pushed_time:0, delay:2000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=438182399, threashold=1000000, tenant_id=1002) [2025-08-14 09:02:51.909439] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6958, timer:0x7f994b1dfb70, task:0x7f9920110810, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161734741162, pushed_time:0, delay:5000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=437168277, threashold=1000000, tenant_id=1002) [2025-08-14 09:02:51.909450] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6a38, timer:0x7f994b1dfb70, task:0x7f9920110af8, task_type:"N9oceanbase5share23ObStorageIOUsageRepoter23TenantIOUsageReportTaskE", scheduled_time:1755161734803774, pushed_time:0, delay:5000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=437105676, threashold=1000000, tenant_id=1002) [2025-08-14 09:02:51.909462] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6e98, timer:0x7f994b1dfb70, task:0x7f99201075a8, task_type:"N9oceanbase3omt11ObTenantSrs27TenantSrsUpdatePeriodicTaskE", scheduled_time:1755161734810627, pushed_time:0, delay:0}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=437098835, threashold=1000000, tenant_id=1002) [2025-08-14 09:02:51.909473] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a64f8, timer:0x7f994b1dfb70, task:0x7f992010e1b0, task_type:"N9oceanbase5table20ObTableObjectPoolMgr26ObTableSessEliminationTaskE", scheduled_time:1755161735735271, pushed_time:0, delay:5000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=436174202, threashold=1000000, tenant_id=1002) [2025-08-14 09:02:51.909483] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a65d8, timer:0x7f994b1dfb70, task:0x7f992010e1c8, task_type:"N9oceanbase5table20ObTableObjectPoolMgr27ObTableSessSysVarUpdateTaskE", scheduled_time:1755161735797729, pushed_time:0, delay:5000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=436111754, threashold=1000000, tenant_id=1002) [2025-08-14 09:02:51.909494] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a66b8, timer:0x7f994b1dfb70, task:0x7f992010e1e0, task_type:"N9oceanbase5table20ObTableObjectPoolMgr28ObTableReqResEliminationTaskE", scheduled_time:1755161735811808, pushed_time:0, delay:5000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=436097686, threashold=1000000, tenant_id=1002) [2025-08-14 09:02:51.909505] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a4ff8, timer:0x7f994b1dfb70, task:0x7f9922ff3e50, task_type:"N9oceanbase3sql3dtl23ObDTLIntermResultGCTaskE", scheduled_time:1755161742809616, pushed_time:0, delay:10000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=429099889, threashold=1000000, tenant_id=1002) [2025-08-14 09:02:51.909517] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a5b58, timer:0x7f99535c3f30, task:0x7f9922ffee90, task_type:"N9oceanbase7storage21ObTenantFreezeInfoMgr24UpdateLSResvSnapshotTaskE", scheduled_time:1755161744768148, pushed_time:0, delay:10000000}, running_task_token={this:0x7f991b7a5a78, timer:0x7f99535c3f30, task:0x7f9922ffee78, task_type:"N9oceanbase7storage21ObTenantFreezeInfoMgr10ReloadTaskE", scheduled_time:1755161737748006, pushed_time:1755161737749908, delay:3000000}, total_delay=427141369, threashold=1000000, tenant_id=1002) [2025-08-14 09:02:51.909527] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a7838, timer:0x7f994b1dfb70, task:0x7f9920110820, task_type:"N9oceanbase10rootserver16ObMViewMdsOpTaskE", scheduled_time:1755161749755625, pushed_time:0, delay:5000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=422153902, threashold=1000000, tenant_id=1002) [2025-08-14 09:02:51.909540] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a74b8, timer:0x7f994b1dfb70, task:0x7f9920110730, task_type:"N9oceanbase10rootserver23ObMViewPushSnapshotTaskE", scheduled_time:1755161804754766, pushed_time:0, delay:60000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=367154774, threashold=1000000, tenant_id=1002) [2025-08-14 09:02:51.909551] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a7218, timer:0x7f994b1dfb70, task:0x7f99201107f8, task_type:"N9oceanbase10rootserver24ObMViewCleanSnapshotTaskE", scheduled_time:1755161804755624, pushed_time:0, delay:60000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=367153927, threashold=1000000, tenant_id=1002) [2025-08-14 09:02:51.909564] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6098, timer:0x7f99200ff180, task:0x7f99200ff208, task_type:"N9oceanbase8observer18ObTableLoadService25ObClientTaskAutoAbortTaskE", scheduled_time:1755162050765334, pushed_time:0, delay:1000000}, running_task_token={this:0x7f991b7a5fb8, timer:0x7f99200ff180, task:0x7f99200ff1f0, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162050755138, pushed_time:1755162084760013, delay:1000000}, total_delay=121144229, threashold=1000000, tenant_id=1002) [2025-08-14 09:02:51.910035] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728de30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7290b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.899, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.910068] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:51.902, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.923005] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:51.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.928663] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:51.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:51.938772] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=22] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:51.938848] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:51.943854] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:02:51.943883] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:51.960032] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=775381] PNIO [ratelimit] time: 1755162171960030, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007273, add_bytes: 0 [2025-08-14 09:02:51.997437] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=11991][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:51.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:52.010544] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:52.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:52.010596] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:52.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:52.023039] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:51.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:52.023163] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:52.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:52.028725] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:51.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:52.038831] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=27] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:52.038889] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:52.043916] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:52.090930] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:52.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:52.121206] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.111, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:52.123087] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:52.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:52.128781] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:52.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:52.134349] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:02:52.134415] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:02:52.134462] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:02:52.134507] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:02:52.134535] INFO [COMMON] print_io_status (ob_io_manager.cpp:2639) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:3779, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:14345, total:14351) [2025-08-14 09:02:52.134562] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, param_config_:{memory_limit:536870912, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=865568, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:02:52.134621] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2826) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=54] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:3779, [delay/us]: prepare:0, schedule:0, submit:600000000, device:14345, total:14351) [2025-08-14 09:02:52.134638] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:02:52.134655] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:52.134670] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:52.134692] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS SERVER] net_in=0kB/s, net_out=0kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=1280000kB/s [421]OB_MALLOC COST TOO MUCH TIME, cost_time=598829, tenant_id=500, label=IoControl, ctx_id=0, prio=0, size=65280, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=3], seq:[0, 1] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=911360, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [857]OB_MALLOC COST TOO MUCH TIME, cost_time=907683, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [729]OB_MALLOC COST TOO MUCH TIME, cost_time=798114, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1], seq:[0, 1] [808]OB_MALLOC COST TOO MUCH TIME, cost_time=236559, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=69, WASH_BLOCK_END=236490], seq:[0, 1, 2] [381]OB_MALLOC COST TOO MUCH TIME, cost_time=500337, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=50, WASH_BLOCK_END=500286], seq:[0, 1, 2] [575]OB_MALLOC COST TOO MUCH TIME, cost_time=607129, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=37, WASH_BLOCK_END=607091], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=523098, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=49, WASH_BLOCK_END=523049], seq:[0, 1, 2] [768]OB_MALLOC COST TOO MUCH TIME, cost_time=102559, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=68, WASH_BLOCK_END=102490], seq:[0, 1, 2] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=517396, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=65, WASH_BLOCK_END=517331], seq:[0, 1, 2] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=471693, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=59, WASH_BLOCK_END=471634], seq:[0, 1, 2] [2025-08-14 09:02:52.737791] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=11] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:51.778916] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623073218545, flush_io_finished_ret_:0, flush_io_finished_round_:26306, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:52.737658] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:52.751414] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13733][errcode=0] epoll handler too much time :13769 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [372]OB_MALLOC COST TOO MUCH TIME, cost_time=508541, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=51, WASH_BLOCK_END=500977, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=4259], seq:[0, 1, 2, 3, 4] [471]OB_MALLOC COST TOO MUCH TIME, cost_time=885267, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=32896, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=885265], seq:[0] [864]OB_MALLOC COST TOO MUCH TIME, cost_time=934093, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=934091], seq:[0] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=983401, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=983392], seq:[0] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=983422, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=983417], seq:[0, 1] [822]OB_MALLOC COST TOO MUCH TIME, cost_time=982697, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=982692], seq:[0] [822]OB_MALLOC COST TOO MUCH TIME, cost_time=982704, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=982703], seq:[0, 1] [2025-08-14 09:02:51.774836] EDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1952592, owner: ObLog, click_count: 5, time dist:[FORMAT_END=969872, ObMalloc#start=1, ObMalloc#end=982710, ALLOC_END=8, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0xfc0e73d 0x993f560 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [381]OB_MALLOC COST TOO MUCH TIME, cost_time=982300, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=982296], seq:[0] [381]OB_MALLOC COST TOO MUCH TIME, cost_time=982309, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=982308], seq:[0, 1] [2025-08-14 09:02:52.245853] EDIAG [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1482730, owner: ObLog, click_count: 7, time dist:[FORMAT_END=20, ObMalloc#start=0, ObMalloc#end=500363, ObMalloc#start=31, ObMalloc#end=982314, ALLOC_END=2, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b537806 0x1b4f816d 0x1b4fbf53 0x1b5014e0 0x1b5009d1 0x1b50387c 0x1b8905de 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [575]OB_MALLOC COST TOO MUCH TIME, cost_time=982787, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=982781], seq:[0] [575]OB_MALLOC COST TOO MUCH TIME, cost_time=982800, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=982799], seq:[0, 1] [2025-08-14 09:02:52.138962] EDIAG [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1589972, owner: ObLog, click_count: 7, time dist:[FORMAT_END=8, ObMalloc#start=0, ObMalloc#end=607138, ObMalloc#start=17, ObMalloc#end=982806, ALLOC_END=3, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1bf2ec62 0x986ebd7 0x986bee6 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [772]OB_MALLOC COST TOO MUCH TIME, cost_time=982857, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=982852], seq:[0] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=1517542, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=64, WASH_BLOCK_END=534614, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=982863], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:52.211535] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1517575, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=6, ALLOC_BLOCK#end=1, FORMAT_END=17, ObMalloc#start=1, ObMalloc#end=1517549, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [770]OB_MALLOC COST TOO MUCH TIME, cost_time=982982, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=982978], seq:[0] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=982989, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=982988], seq:[0, 1] [2025-08-14 09:02:52.223138] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1506144, owner: ObLog, click_count: 7, time dist:[FORMAT_END=26, ObMalloc#start=1, ObMalloc#end=523106, ObMalloc#start=17, ObMalloc#end=982993, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [768]OB_MALLOC COST TOO MUCH TIME, cost_time=983016, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=983011], seq:[0] [768]OB_MALLOC COST TOO MUCH TIME, cost_time=983025, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=983024], seq:[0, 1] [2025-08-14 09:02:52.644334] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1085659, owner: ObLog, click_count: 7, time dist:[FORMAT_END=40, ObMalloc#start=1, ObMalloc#end=102568, ObMalloc#start=18, ObMalloc#end=983030, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [737]OB_MALLOC COST TOO MUCH TIME, cost_time=983101, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=983097], seq:[0] [737]OB_MALLOC COST TOO MUCH TIME, cost_time=1590410, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=64, WASH_BLOCK_END=607239, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=983107], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:52.138892] EDIAG [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1590430, owner: ObLog, click_count: 5, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=1590417, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1bf2ec62 0x986ebd7 0x986bee6 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [856]OB_MALLOC COST TOO MUCH TIME, cost_time=983165, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=983160], seq:[0] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=1585409, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=65, WASH_BLOCK_END=602171, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=983172], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:52.143976] EDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] LOGGER COST TOO MUCH TIME, cost: 1585428, owner: ObLog, click_count: 5, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=1585415, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b52e837 0x996d04c 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [658]OB_MALLOC COST TOO MUCH TIME, cost_time=983203, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=983199], seq:[0] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=983211, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=983210], seq:[0, 1] [2025-08-14 09:02:52.228878] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1500700, owner: ObLog, click_count: 7, time dist:[FORMAT_END=33, ObMalloc#start=1, ObMalloc#end=517428, ObMalloc#start=21, ObMalloc#end=983216, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [385]OB_MALLOC COST TOO MUCH TIME, cost_time=983263, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=983259], seq:[0] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=1002121, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=44, WASH_BLOCK_END=18806, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=983269], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:52.728256] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9926][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1002146, owner: ObLog, click_count: 5, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=1002133, ALLOC_END=2, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [660]OB_MALLOC COST TOO MUCH TIME, cost_time=983282, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=983278], seq:[0] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=983288, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=983288], seq:[0, 1] [2025-08-14 09:02:52.274727] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1455049, owner: ObLog, click_count: 7, time dist:[FORMAT_END=23, ObMalloc#start=1, ObMalloc#end=471700, ObMalloc#start=30, ObMalloc#end=983294, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [419]OB_MALLOC COST TOO MUCH TIME, cost_time=983333, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=983329], seq:[0] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=1584950, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=54, WASH_BLOCK_END=601555, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=983339], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:52.144678] EDIAG try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1584966, owner: ObLog, click_count: 5, time dist:[FORMAT_END=8, ObMalloc#start=1, ObMalloc#end=1584956, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x26499505 0x97074a3 0x26498fa9 0x7f9973e081cf 0x7f9973a39dd3 [382]OB_MALLOC COST TOO MUCH TIME, cost_time=983365, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=983360], seq:[0] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=1008993, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=56, WASH_BLOCK_END=25565, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=983371], seq:[0, 1, 2, 3, 4] [1031]OB_MALLOC COST TOO MUCH TIME, cost_time=983376, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=983372], seq:[0] [1031]OB_MALLOC COST TOO MUCH TIME, cost_time=1490046, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=42, WASH_BLOCK_END=506621, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=983382], seq:[0, 1, 2, 3, 4] [763]OB_MALLOC COST TOO MUCH TIME, cost_time=983408, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=983402], seq:[0] [763]OB_MALLOC COST TOO MUCH TIME, cost_time=983417, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=983416], seq:[0, 1] [2025-08-14 09:02:52.688634] EDIAG [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEA3-0-0] [lt=6][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1041999, owner: ObLog, click_count: 9, time dist:[FORMAT_END=16, ObMalloc#start=1, ObMalloc#end=58535, ObMalloc#start=22, ObMalloc#end=983423, ALLOC_END=2, APPEND_END=0, ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=1], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1dc20366 0x1dc1fa40 0x1dd7bdd2 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:52.721545] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1009829, owner: ObLog, click_count: 7, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=1009000, ALLOC_END=816, APPEND_END=0, ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=2], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [399]OB_MALLOC COST TOO MUCH TIME, cost_time=1005452, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1005445], seq:[0] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=1005479, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1005473], seq:[0, 1] [2025-08-14 09:02:52.747304] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1005500, owner: ObLog, click_count: 5, time dist:[FORMAT_END=9, ObMalloc#start=0, ObMalloc#end=1005486, ALLOC_END=4, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:00:15.769921] EDIAG [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:594) [677][T1001_Occam][T1001][YB42AC4D0007-00063C4967999C34-0-0] [lt=1004242][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1000941, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1000939, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x1c437743 0x1c43b8d4 0x1c47767a 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [730]OB_MALLOC COST TOO MUCH TIME, cost_time=1008786, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1008779], seq:[0] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=1008799, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1008798], seq:[0, 1] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=1007683, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1007678], seq:[0] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=1007691, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1007690], seq:[0, 1] [2025-08-14 09:02:52.748645] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=54][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1007766, owner: ObLog, click_count: 5, time dist:[FORMAT_END=67, ObMalloc#start=1, ObMalloc#end=1007697, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [777]OB_MALLOC COST TOO MUCH TIME, cost_time=1006506, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1006502], seq:[0] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=1006515, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1006514], seq:[0, 1] [383]OB_MALLOC COST TOO MUCH TIME, cost_time=1006202, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1006198], seq:[0] [383]OB_MALLOC COST TOO MUCH TIME, cost_time=1006209, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1006208], seq:[0, 1] [2025-08-14 09:02:52.750317] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1006224, owner: ObLog, click_count: 5, time dist:[FORMAT_END=8, ObMalloc#start=1, ObMalloc#end=1006214, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [568]OB_MALLOC COST TOO MUCH TIME, cost_time=1006238, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1006233], seq:[0] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=1006245, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1006244], seq:[0, 1] [2025-08-14 09:02:52.750340] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=15162][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1006259, owner: ObLog, click_count: 5, time dist:[FORMAT_END=7, ObMalloc#start=1, ObMalloc#end=1006250, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:52.737840] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=10] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:52.510014] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a4e38, timer:0x7f99535f3df0, task:0x7f9922ff6bf0, task_type:"N9oceanbase7obmysql15ObEliminateTaskE", scheduled_time:1755161931284195, pushed_time:0, delay:500000}, running_task_token={this:0x7f991b7a4f18, timer:0x7f99535f3df0, task:0x7f9922ff6c40, task_type:"N9oceanbase7obmysql20ObConstructQueueTaskE", scheduled_time:1755161931282757, pushed_time:1755161931285404, delay:500000}, total_delay=241225816, threashold=1000000, tenant_id=1002) [2025-08-14 09:02:52.053741] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] EASY SLOW: thread: 7f99542fb300, lastrun: 1755162115.752503 cost: 56.301234 loop:26416, slowcnt: 469 [2025-08-14 09:02:52.249143] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:52.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:52.752667] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=502759][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:52.249|207=503.52ms|210=0us|222=2us|total=503.54ms) [2025-08-14 09:02:52.752690] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4389] timer task use too much time(end=1755162172752689, start=1755162172249141, delta=503548) [2025-08-14 09:02:52.752702] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=508067, expected_run_ts_us_=1755162172244633, ObClockGenerator::getRealClock()=1755162172752701, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:52.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:52.752724] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:52.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:52.753293] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:52.719736] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:51.782788] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=17][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:52.753943] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34155][errcode=0] epoll handler too much time :34220 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:52.753952] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=969696][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:52.753969] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162171745355]}) [2025-08-14 09:02:52.748940] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:52.753988] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=5032] temporary file module use no blocks [2025-08-14 09:02:52.753996] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:52.743295] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:52.754007] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:52.754017] WDIAG ev_invoke_pending (ev.c:2136) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10705][errcode=0] epoll handler too much time :10727 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:52.754020] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623074188633, flush_io_finished_ret_:0, flush_io_finished_round_:26328, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:52.245148] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:52.754507] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=508581][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:02:52.754528] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:52.746624] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:52.754545] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:52.756411] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=1851][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:52.756425] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:52.756440] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:52.756450] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:01:49.731, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:01:42.751], idx=659, next_allow_print_ts="09:02:19.731") [2025-08-14 09:02:52.758365] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:52.761034] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=15] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:52.762301] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:52.762433] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6002] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:52.754626] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1832) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=10][errcode=-4012] get gts fail(ret=-4012, timeout_us=1000000, request_time=[mts=1755162171733556]) [2025-08-14 09:02:52.737877] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=24] blacklist_loop exec finished(cost_time=15, is_enabled=true, send_cnt=0) [2025-08-14 09:02:52.739316] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=23] current monitor number(seq_=-1) [2025-08-14 09:02:52.739346] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=12] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:52.741367] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=422993315, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:52.764956] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23546] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:52.763291] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1844) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=8650][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1755162198682141, request_time=[mts=1755162171733556]) [2025-08-14 09:02:52.756460] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:52.765437] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8956][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:52.765448] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:52.765456] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:52.765465] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:52.765626] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:52.765636] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:52.766972] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28] get wrs ts(ls_id={id:1001}, delta=428038450, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:52.766989] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:52.742242] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=28] get wrs ts(ls_id={id:1}, delta=452959329, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:52.768469] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26185] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=3, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:52.742716] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=624995992, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:52.768496] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25739] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=6, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:52.743269] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:52.768520] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25212][errcode=0] epoll handler too much time :25261 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:52.771312] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=20] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:52.771335] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=17] temporary file module use no blocks [2025-08-14 09:02:52.771344] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:52.771353] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:52.771363] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623074207794, flush_io_finished_ret_:0, flush_io_finished_round_:26340, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:52.771678] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161101770923, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755161572801635) [2025-08-14 09:02:52.772183] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=625032402, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:52.772195] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:52.772730] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=452997103, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:52.775318] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:52.778785] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:52.877472] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=372][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c0b90, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571c0e70, time_wheel:0x7f9959fef990, thread_pool:0x7f9959fef170, is_running:1, time_interval:3.00s, expected_run_time:09:02:52.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:52.880532] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=398][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb38b8, timer:0x7f9966a59cb0, task:0x7f99333bfa78, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr15MediumCheckTaskE", scheduled_time:1755161750786742, pushed_time:0, delay:20000000}, running_task_token={this:0x7f9931bb37d8, timer:0x7f9966a59cb0, task:0x7f99333bfa38, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr14MediumLoopTaskE", scheduled_time:1755161743804000, pushed_time:1755161743806666, delay:120000000}, total_delay=422093788, threashold=1000000, tenant_id=1001) [2025-08-14 09:02:52.931751] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:52.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:52.952676] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=242150][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6db8, timer:0x7f994b1dfb70, task:0x7f99201e33a0, task_type:"N9oceanbase5table20ObTableClientInfoMgr27ObTableClientInfoRetireTaskE", scheduled_time:1755161750768689, pushed_time:0, delay:60000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=422183986, threashold=1000000, tenant_id=1002) [2025-08-14 09:02:53.031836] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:52.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.131958] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:53.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.135464] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:02:53.146453] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=766410][errcode=0] EASY SLOW: thread: 7f99543f7300, lastrun: 1755162172.745397 cost: 0.401053 loop:26525, slowcnt: 4 [2025-08-14 09:02:52.772739] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:52.779155] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=13] gc stale ls task succ [2025-08-14 09:02:52.967646] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=24] PNIO [ratelimit] time: 1755162172967644, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007614, add_bytes: 0 [2025-08-14 09:02:52.994799] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:52.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.046554] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=60][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a5a78, timer:0x7f996714bf20, task:0x7f994b3fce90, task_type:"N9oceanbase7storage21ObTenantFreezeInfoMgr24UpdateLSResvSnapshotTaskE", scheduled_time:1755161931783996, pushed_time:0, delay:10000000}, running_task_token={this:0x7f994e1a5998, timer:0x7f996714bf20, task:0x7f994b3fce78, task_type:"N9oceanbase7storage21ObTenantFreezeInfoMgr10ReloadTaskE", scheduled_time:1755161924772450, pushed_time:1755161924772520, delay:3000000}, total_delay=241262557, threashold=1000000, tenant_id=1) [2025-08-14 09:02:53.717420] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=721523][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:52.994|207=722.62ms|210=1us|222=1us|total=722.66ms) [2025-08-14 09:02:53.717456] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=-4389] timer task use too much time(end=1755162173717455, start=1755162172994796, delta=722659) [2025-08-14 09:02:53.717473] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=472838, expected_run_ts_us_=1755162173244633, ObClockGenerator::getRealClock()=1755162173717472, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:53.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.717494] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:53.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.232036] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=49][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:53.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.717783] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=485010][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:53.232|207=485.75ms|210=0us|222=1us|total=485.77ms) [2025-08-14 09:02:53.717806] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4389] timer task use too much time(end=1755162173717806, start=1755162173232033, delta=485773) [2025-08-14 09:02:53.265800] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:53.717841] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=451345][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:53.717855] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:53.717879] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:53.717888] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:53.717905] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:52.767014] INFO [STORAGE.TRANS] amend (ob_tenant_weak_read_server_version_mgr.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [WRS] tenant has no valid partition on current server, generate weak read snapshot version based on current timestamp(tenant_id=1002, epoch_tstamp_=1755162172741347, total_part_count_=2, valid_inner_part_count_=0, valid_user_part_count_=0, version_={val:1755161758221798000, v:0}, base_version_when_no_valid_partition={val:1755161758221798000, v:0}) [2025-08-14 09:02:53.719355] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=950896] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=415497553, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=251725, succ_cluster_heartbeat_count=1500, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161006924036004, v:0}, local_cluster_delta=1166795315, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:02:53.135517] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:02:52.772060] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:53.717914] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:01:49.731, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:01:42.751], idx=659, next_allow_print_ts="09:02:19.731") [2025-08-14 09:02:53.720092] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=2165][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:00:12.405, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:42.405], idx=677, next_allow_print_ts="09:00:42.405") [2025-08-14 09:02:53.720115] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:53.720129] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:53.720140] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:53.720149] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:53.720324] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:53.720606] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fb3b0, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fb690, time_wheel:0x7f99589fd990, thread_pool:0x7f9959fff170, is_running:1, time_interval:2.00s, expected_run_time:09:02:53.698, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.721331] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:53.720335] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:52.783409] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:02:53.728997] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=625990391, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:53.729014] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:53.729037] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162173728509}, version_delta=625990520) [2025-08-14 09:02:53.729526] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=625990391, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:53.729534] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:53.730045] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=943159] get wrs ts(ls_id={id:1}, delta=453954786, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:53.730054] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:52.745406] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=26] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:53.730334] WDIAG ev_invoke_pending (ev.c:2136) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=983438][errcode=0] epoll handler too much time :984936 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:51.774836] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1001, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:8571658, max_lsn:7323687369, max_scn:{val:1755161749741193002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:7278610047}, reuse_lsn:{lsn:7323686131}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:7281744329}}, last_submit_lsn:{lsn:7323686131}, last_submit_end_lsn:{lsn:7323687369}, last_submit_log_id:8571658, last_submit_log_pid:49, max_flushed_lsn:{lsn:7323686009}, max_flushed_end_lsn:{lsn:7323686131}, max_flushed_log_pid:49, committed_end_lsn:{lsn:7323686009}, last_slide_log_id:8571656, last_slide_scn:{val:1755161747742105001, v:0}, last_slide_lsn:{lsn:7323685887}, last_slide_end_lsn:{lsn:7323686009}, last_slide_log_pid:49, last_slide_log_accum_checksum:2505458373, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f990112a530}, [StateMgr]={this:0x7f990112e9b0, self:"172.77.0.7:2882", palf_id:1001, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"PENDING", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:49}, leader:"0.0.0.0:0", leader_epoch:-1, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:7323687369}, scan_disk_log_finished:true, last_check_start_id:8571657, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161767744317, palf_role_change_cb:0x7f9901135470, allow_vote:true}, [ConfigMgr]={palf_id:1001, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:49, prev:{version:1, config:{config_version:{proposal_id:48, config_seq:47}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:49, config_seq:48}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_mode_pid:49}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_end_lsn:{lsn:7323685240}, prev_mode_pid:49}, reconfig_barrier:{prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_end_lsn:{lsn:7323685240}, prev_mode_pid:49}, persistent_config_version:{proposal_id:49, config_seq:48}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755161739755312, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755161739755312, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f990112bc70}, [ModeMgr]={palf_id:1001, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, accepted_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1001, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1753069671378640001, v:0}, base_lsn_for_block_gc:{lsn:7247314944}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:49}, log_config_meta:{version:3, proposal_id:49, prev:{version:1, config:{config_version:{proposal_id:48, config_seq:47}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:49, config_seq:48}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_mode_pid:49}, log_snapshot_meta:{version:2, base_lsn:{lsn:7247314944}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1753069671378640000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}}, log_meta_storage:{log_tail:{lsn:765952}, readable_log_tail:{lsn:765952}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7381524480}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/meta", dir_fd:232, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66338816, block_header_serialize_buf_:0x7f9901130bb8, flashback_version:0}, log_storage:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, palf_epoch:2, last_purge_throttling_ts:1755161769777679, this:0x7f990112f0f0}, [Reconfirm]={palf_id:1001, self:"172.77.0.7:2882", state:"WAITING_LOG_FLUSHED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:1755161769777679, this:0x7f990112eb30}) [2025-08-14 09:02:52.245853] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:53.730851] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=1482730][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162172245847], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:53.730867] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162172245847], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:53.730880] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162172245847]}) [2025-08-14 09:02:53.730890] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:53.730897] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:53.730912] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=5] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:53.730918] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162173730909], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:53.730925] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162173730909]}) [2025-08-14 09:02:53.730934] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7] gts statistics(tenant_id=1001, gts_rpc_cnt=0, get_gts_cache_cnt=76, get_gts_with_stc_cnt=35, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:02:53.731107] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:53.731115] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:53.731127] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162173731102], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:53.731135] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162173731102]}) [2025-08-14 09:02:53.731142] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6] gts statistics(tenant_id=1002, gts_rpc_cnt=0, get_gts_cache_cnt=78, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:02:52.138962] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:53.731370] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1589972] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:52.211535] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:52.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.731425] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1517575][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:52.211|207=1.52s|210=0us|222=2us|total=1.52s) [2025-08-14 09:02:53.731442] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] timer task use too much time(end=1755162173731441, start=1755162172211533, delta=1519908) [2025-08-14 09:02:53.731451] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=0] task scheduled too delay(scheduled_delay=1472315, expected_run_ts_us_=1755162172259134, ObClockGenerator::getRealClock()=1755162173731450, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:52.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.731464] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:52.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.731473] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1475986, expected_run_ts_us_=1755162172255487, ObClockGenerator::getRealClock()=1755162173731473, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:52.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.731485] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:52.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.731498] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1142181, expected_run_ts_us_=1755162172589316, ObClockGenerator::getRealClock()=1755162173731498, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:52.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.731507] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:52.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:52.223138] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:52.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.731578] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1506144][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:52.223|207=1.51s|210=1us|222=1us|total=1.51s) [2025-08-14 09:02:53.731586] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] timer task use too much time(end=1755162173731586, start=1755162172223136, delta=1508450) [2025-08-14 09:02:52.644334] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571634a0, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957163780, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:5.00s, expected_run_time:09:02:52.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.731653] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=1085659][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:52.644|207=1.09s|210=0us|222=2us|total=1.09s) [2025-08-14 09:02:53.731663] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task use too much time(end=1755162173731663, start=1755162172644331, delta=1087332) [2025-08-14 09:02:53.731669] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=4][errcode=0] task scheduled too delay(scheduled_delay=1176393, expected_run_ts_us_=1755162172555275, ObClockGenerator::getRealClock()=1755162173731669, ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:52.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.731688] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:52.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:52.138892] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=22] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:53.731748] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=1590430] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:52.143976] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:53.731819] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1585428][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:52.228878] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:52.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.731866] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1500700][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:52.228|207=1.50s|210=0us|222=2us|total=1.50s) [2025-08-14 09:02:53.731874] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] timer task use too much time(end=1755162173731874, start=1755162172228876, delta=1502998) [2025-08-14 09:02:52.728256] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9926] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:53.731935] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1002146][errcode=0] epoll handler too much time :1003689 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:52.274727] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:52.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.731995] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1455049][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:52.274|207=1.46s|210=0us|222=1us|total=1.46s) [2025-08-14 09:02:53.732003] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4389] timer task use too much time(end=1755162173732002, start=1755162172274725, delta=1457277) [2025-08-14 09:02:52.144678] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=19] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:02:52.239783] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=16] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:52.688634] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEA3-0-0] [lt=6] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:02:53.731950] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:52.721545] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:53.732926] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1009829][errcode=0] epoll handler too much time :1011391 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:53.732938] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:53.732009] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=5][errcode=0] task scheduled too delay(scheduled_delay=1439659, expected_run_ts_us_=1755162172292349, ObClockGenerator::getRealClock()=1755162173732008, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:52.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.732951] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=940][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:52.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.733004] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEA3-0-0] [lt=1041999][errcode=0] cost too much time:mds_tenant_service.cpp:void oceanbase::storage::mds::ObTenantMdsTimer::dump_special_mds_table_status_task(), (this=|threshold=1.00s|start at 09:02:52.688|total=1.04s) [2025-08-14 09:02:53.733012] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEA3-0-0] [lt=8][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571630c0, caller:mds_tenant_service.cpp:mtl_start:117, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_145, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571633a0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:0, time_interval:15.00s, expected_run_time:09:03:07.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.733026] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEA3-0-0] [lt=12][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:02:53.733|total=1.04s) [2025-08-14 09:02:53.733036] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEA3-0-0] [lt=7][errcode=-4016] task executed too delay(delay_time=1044947, *p_task_={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:0, time_interval:3.00s, expected_run_time:09:02:55.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.735540] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=33] get_max_applied_scn(scn={val:1755161749741193000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:50, palf_committed_end_lsn_:{lsn:1604090090}, palf_committed_end_scn_:{val:1755161749741193000, v:0}, last_check_scn_:{val:1755161749741193001, v:0}, max_applied_cb_scn_:{val:1755161749741193000, v:0}}) [2025-08-14 09:02:53.735747] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [998][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F8E-0-0] [lt=961011][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1002, svr="172.77.0.7:2882", version={val:1755161758221798000, v:0}, valid_part_count=0, total_part_count=2, generate_timestamp=1755162173719389) [2025-08-14 09:02:53.735762] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [998][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F8E-0-0] [lt=14][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1002, req={req_server:"172.77.0.7:2882", version:{val:1755161758221798000, v:0}, valid_part_count:0, total_part_count:2, generate_timestamp:1755162173719389}, twrs={inited:true, tenant_id:1002, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162172741347}, server_version_for_stat:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162172741347}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:53.735838] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [316][pnio1][T1002][YB42AC4D0007-00063C49612B8F8E-0-0] [lt=21][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1002, delta=5008662, last_succ_cluster_heartbeat_tstamp_=1755162168727176) [2025-08-14 09:02:53.719445] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=945951][errcode=0] epoll handler too much time :947397 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:53.737137] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:53.735565] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] get_min_unreplayed_log_info(lsn={lsn:1604088821}, scn={val:1755100948296325001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873479, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:12492179, ret_code_:0}}, next_to_submit_lsn_:{lsn:1604088821}, next_to_submit_scn_:{val:1755100948296325001, v:0}, base_lsn_:{lsn:1409200128}, base_scn_:{val:1753840802540914000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:1604088821}, end_lsn:{lsn:1604088821}, read_buf:{buf_len_:3747840, buf_:0x7f98f6a05000}, block_size:67104768, log_storage_:0x7f990bfebbf0, log_storage_:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:42, read_io_size:147709952, read_disk_cost_ts:1061450, start_lsn:{lsn:1409200128}}}}}}) [2025-08-14 09:02:53.737301] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=1730] get wrs ts(ls_id={id:1}, delta=423994626, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:53.737317] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:53.737340] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get_max_applied_scn(scn={val:1755161744727824001, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:49, palf_committed_end_lsn_:{lsn:7323686009}, palf_committed_end_scn_:{val:1755161747742105001, v:0}, last_check_scn_:{val:1755161749741193000, v:0}, max_applied_cb_scn_:{val:1755161744727824001, v:0}}) [2025-08-14 09:02:53.737489] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEA5-0-0] [lt=9] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:02:53.737355] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get_min_unreplayed_log_info(lsn={lsn:7323685240}, scn={val:1755100946796590002, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873485, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:8946764, ret_code_:0}}, next_to_submit_lsn_:{lsn:7323685240}, next_to_submit_scn_:{val:1755100946796590002, v:0}, base_lsn_:{lsn:7180210176}, base_scn_:{val:1753840803166611000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:7323685240}, end_lsn:{lsn:7323685240}, read_buf:{buf_len_:3747840, buf_:0x7f98f7a05000}, block_size:67104768, log_storage_:0x7f9901131bf0, log_storage_:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:28, read_io_size:98508800, read_disk_cost_ts:645339, start_lsn:{lsn:7180210176}}}}}}) [2025-08-14 09:02:53.738760] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=862] get wrs ts(ls_id={id:1001}, delta=429009942, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:53.738777] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:53.738811] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=423996573, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:53.738819] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:53.739326] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8] get wrs ts(ls_id={id:1001}, delta=429009942, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:53.739335] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:52.881250] WDIAG [OCCAM] dump_statistics (ob_vtable_event_recycle_buffer.h:144) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EEC4-0-0] [lt=15][errcode=-4006] not init(ret=-4006) [2025-08-14 09:02:53.741577] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:53.744420] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:53.745422] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:53.747355] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9719] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:53.748662] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:53.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.749579] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=11670] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:53.753255] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:52.747304] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:52.749987] EDIAG [RS] skip_check_member_list_change_ (ob_ls_recovery_stat_handler.cpp:109) [777][T1002_TeRec][T1002][YB42AC4D0007-00063C495DDA003E-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1010332, owner: ObLog, click_count: 5, time dist:[FORMAT_END=9, ObMalloc#start=0, ObMalloc#end=1006523, ALLOC_END=3799, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x1067fb2c 0x1067ecc2 0x1067e03d 0x1002f2a3 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:52.239783] EDIAG [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1490071, owner: ObLog, click_count: 5, time dist:[FORMAT_END=16, ObMalloc#start=1, ObMalloc#end=1490052, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x18257d0c 0x1823b885 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:53.754273] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=19] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:53.754345] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1005500][errcode=0] epoll handler too much time :1007049 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:53.754361] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:51.802783] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=22] gc stale ls task succ [2025-08-14 09:02:53.757857] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=16] [sql nio session](*s={this:0x7f98f72b3030, session_id:3221562808, trace_id:YB42AC4D0007-00063C4965DA1F2E-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:267}, err:0, last_decode_time:1755161748766816, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1890, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:52.748645] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=54][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.757954] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1007766][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:52.748|207=1.01s|210=1us|222=1us|total=1.01s) [2025-08-14 09:02:53.757976] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4389] timer task use too much time(end=1755162173757976, start=1755162172748642, delta=1009334) [2025-08-14 09:02:53.757984] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] task scheduled too delay(scheduled_delay=1026773, expected_run_ts_us_=1755162172731210, ObClockGenerator::getRealClock()=1755162173757984, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.757997] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:52.750317] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:53.758081] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1006224][errcode=0] epoll handler too much time :1007772 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:52.750340] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=15162] gc stale ls task succ [2025-08-14 09:02:53.758138] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=1006259] gc stale ls task succ [2025-08-14 09:02:53.751026] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=970557] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:53.760456] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9411] temporary file module use no blocks [2025-08-14 09:02:53.760464] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:53.760476] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:53.760486] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623075190712, flush_io_finished_ret_:0, flush_io_finished_round_:26307, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:53.752006] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=14014] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:53.761083] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:53.761116] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=11] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:53.757878] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=22] [sql nio session](*s={this:0x7f98f72ab030, session_id:3221589435, trace_id:YB42AC4D0007-00063C4965DA1F35-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:246}, err:0, last_decode_time:1755161762721798, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:2456, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:53.761784] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=3899] [sql nio session](*s={this:0x7f98f72a9030, session_id:3221590528, trace_id:YB42AC4D0007-00063C4965DA1F34-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:250}, err:0, last_decode_time:1755161762715415, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:205, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:53.761820] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=35] [sql nio session](*s={this:0x7f98f72b5030, session_id:3221590961, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:10, sql_initiative_shutdown:false, reader:{fd:278}, err:5, last_decode_time:1755162075745307, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:53.757905] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:53.761922] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:53.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.719435] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=582999] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:02:53.762095] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=42549] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:02:53.762113] INFO [COMMON] print_io_status (ob_io_manager.cpp:2639) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:2516, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:4766, total:4773) [2025-08-14 09:02:53.762132] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, param_config_:{memory_limit:536870912, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=865568, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:02:53.762170] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2826) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=33] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:2516, [delay/us]: prepare:0, schedule:0, submit:600000000, device:4766, total:4773) [2025-08-14 09:02:53.762190] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:02:53.762200] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:53.762212] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:53.762220] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SERVER] net_in=0kB/s, net_out=0kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=1280000kB/s [2025-08-14 09:02:53.762305] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=1490071] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3970) [2025-08-14 09:02:53.762322] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:53.762339] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=6] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:53.762349] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:227) [1031][PxTargetMgr0][T1001][Y0-0000000000000000-0-0] [lt=8] refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3971) [2025-08-14 09:02:53.762357] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:53.762535] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=20] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:53.762578] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=43] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3972) [2025-08-14 09:02:53.762582] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:53.762613] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:53.758095] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:53.758007] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1025761, expected_run_ts_us_=1755162172732246, ObClockGenerator::getRealClock()=1755162173758007, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891991430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ce10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.762755] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=4738][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891991430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ce10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.762778] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] task scheduled too delay(scheduled_delay=1020371, expected_run_ts_us_=1755162172742406, ObClockGenerator::getRealClock()=1755162173762777, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aac90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.762815] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aac90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.762825] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1015119, expected_run_ts_us_=1755162172747705, ObClockGenerator::getRealClock()=1755162173762825, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.762835] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.762845] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1020310, expected_run_ts_us_=1755162172742535, ObClockGenerator::getRealClock()=1755162173762845, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.762854] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.762864] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1015557, expected_run_ts_us_=1755162172747306, ObClockGenerator::getRealClock()=1755162173762864, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e62b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.762873] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e62b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.762885] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1017184, expected_run_ts_us_=1755162172745700, ObClockGenerator::getRealClock()=1755162173762885, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73940b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.762895] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73940b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.762905] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1005405, expected_run_ts_us_=1755162172757499, ObClockGenerator::getRealClock()=1755162173762905, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eeb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.762913] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eeb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.762923] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1005277, expected_run_ts_us_=1755162172757645, ObClockGenerator::getRealClock()=1755162173762922, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73936b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eec90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.762935] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73936b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eec90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.762944] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1003128, expected_run_ts_us_=1755162172759815, ObClockGenerator::getRealClock()=1755162173762944, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.762952] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.762961] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=996411, expected_run_ts_us_=1755162172766550, ObClockGenerator::getRealClock()=1755162173762961, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.762970] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.762981] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1002442, expected_run_ts_us_=1755162172760539, ObClockGenerator::getRealClock()=1755162173762981, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72765b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.762990] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72765b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.762999] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=994301, expected_run_ts_us_=1755162172768698, ObClockGenerator::getRealClock()=1755162173762999, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763009] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:51.802783] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=22][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1951924, owner: ObLog, click_count: 7, time dist:[FORMAT_END=930404, ObMalloc#start=3, ObMalloc#end=12690, ObMalloc#start=19, ObMalloc#end=1008805, ALLOC_END=3, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:53.763275] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=1951924] gc stale ls task succ [2025-08-14 09:00:18.718152] EDIAG [SERVER] query (ob_inner_sql_connection.cpp:951) [712][T1001_GCCollect][T1][YB42AC4D0007-00063C495FA9A613-0-0] [lt=6][errcode=-4283] LOGGER COST TOO MUCH TIME, cost: 154811264, owner: ObLog, click_count: 5, time dist:[FORMAT_END=154811256, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x124dff1f 0x96a5a77 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x1ee93940 0x1ee92221 0x1ee928a5 0xfd80a58 0xfd91a9a 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 08:58:20.253501] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [579][T1_MultiVersion][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 273107201, owner: ObLog, click_count: 5, time dist:[FORMAT_END=478325, ObMalloc#start=0, ObMalloc#end=1011385, ALLOC_END=271617488, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:53.763042] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:53.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.763348] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=302][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:53.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.763018] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=996649, expected_run_ts_us_=1755162172766368, ObClockGenerator::getRealClock()=1755162173763018, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763386] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=366][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763407] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] task scheduled too delay(scheduled_delay=1001714, expected_run_ts_us_=1755162172761692, ObClockGenerator::getRealClock()=1755162173763407, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763422] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763436] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=988883, expected_run_ts_us_=1755162172774552, ObClockGenerator::getRealClock()=1755162173763436, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aab10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.774, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763448] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aab10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.774, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763460] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=990746, expected_run_ts_us_=1755162172772714, ObClockGenerator::getRealClock()=1755162173763460, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739ef90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763472] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739ef90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763485] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=983485, expected_run_ts_us_=1755162172779999, ObClockGenerator::getRealClock()=1755162173763485, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a45b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763498] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a45b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763510] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=987980, expected_run_ts_us_=1755162172775530, ObClockGenerator::getRealClock()=1755162173763510, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eee10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763521] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eee10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763532] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=987824, expected_run_ts_us_=1755162172775708, ObClockGenerator::getRealClock()=1755162173763532, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73931b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763544] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73931b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763558] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=976451, expected_run_ts_us_=1755162172787107, ObClockGenerator::getRealClock()=1755162173763558, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763570] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:00:18.718152] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:951) [712][T1001_GCCollect][T1][YB42AC4D0007-00063C495FA9A613-0-0] [lt=6][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2025-08-14 09:02:53.763583] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=977168, expected_run_ts_us_=1755162172786414, ObClockGenerator::getRealClock()=1755162173763583, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763604] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763617] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=976869, expected_run_ts_us_=1755162172786748, ObClockGenerator::getRealClock()=1755162173763617, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7273c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763631] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7273c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763644] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=979721, expected_run_ts_us_=1755162172783922, ObClockGenerator::getRealClock()=1755162173763644, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d27b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763656] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d27b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763668] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=974283, expected_run_ts_us_=1755162172789385, ObClockGenerator::getRealClock()=1755162173763668, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d94b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763680] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d94b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763694] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=983375, expected_run_ts_us_=1755162172780319, ObClockGenerator::getRealClock()=1755162173763694, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.780, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763707] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.780, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763719] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=983230, expected_run_ts_us_=1755162172780489, ObClockGenerator::getRealClock()=1755162173763719, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739bbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.780, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763732] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739bbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.780, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763744] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=981975, expected_run_ts_us_=1755162172781769, ObClockGenerator::getRealClock()=1755162173763744, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fa10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763758] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fa10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763770] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=980549, expected_run_ts_us_=1755162172783221, ObClockGenerator::getRealClock()=1755162173763770, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f727dc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f729ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763783] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f727dc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f729ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763796] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=980100, expected_run_ts_us_=1755162172783695, ObClockGenerator::getRealClock()=1755162173763796, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f727d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f729af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763808] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f727d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f729af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763822] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=977978, expected_run_ts_us_=1755162172785844, ObClockGenerator::getRealClock()=1755162173763822, ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763835] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763848] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=970105, expected_run_ts_us_=1755162172793742, ObClockGenerator::getRealClock()=1755162173763848, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891814fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.793, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763860] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891814fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.793, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763873] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=967425, expected_run_ts_us_=1755162172796447, ObClockGenerator::getRealClock()=1755162173763872, ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763886] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763899] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=956625, expected_run_ts_us_=1755162172807273, ObClockGenerator::getRealClock()=1755162173763899, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7facb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763912] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7facb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763928] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=768271, expected_run_ts_us_=1755162172995656, ObClockGenerator::getRealClock()=1755162173763928, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199fbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919acc10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.995, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763940] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199fbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919acc10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:52.995, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763969] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728bb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:53.718, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763982] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180fbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:53.719, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.763996] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:53.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.764010] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:53.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.764023] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:53.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.764037] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918105b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:53.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.764341] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=1016244] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:53.764369] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=28] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:53.764378] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:53.764397] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=19] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:53.764405] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:53.764415] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:53.764423] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:53.764432] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:53.764469] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=37] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:53.762998] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1952592] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [2025-08-14 09:02:53.764830] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1821] current log_task status(palf_id=1001, self="172.77.0.7:2882", log_id=8571657, log_task={header:{begin_lsn:{lsn:7323686009}, end_lsn:{lsn:7323686131}, log_id:8571657, min_scn:{val:1755161749741193000, v:0}, max_scn:{val:1755161749741193000, v:0}, data_len:66, proposal_id:49, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:7323686009}, data_checksum:2222604892, accum_checksum:1531236791, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1755161749745909, freeze_ts:1755161749745914, submit_ts:1755161749745940, flushed_ts:1755161785758706, gen_to_freeze cost time:5, gen_to_submit cost time:31, submit_to_flush cost time:36012766}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [2025-08-14 09:02:53.764958] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=25550] current monitor number(seq_=-1) [2025-08-14 09:02:53.765007] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=25549] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:53.765036] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] get_fetch_log_dst failed or invalid(ret=0, palf_id=1, self="172.77.0.7:2882", fetch_log_dst="172.77.0.7:2882") [2025-08-14 09:02:53.766044] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1004] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:02:53.766081] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=30] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:02:53.760254] INFO [MDS] dump_map_holding_item (mds_tenant_service.cpp:462) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EEC4-0-0] [lt=856814] finish scan map holding items(scan_cnt=0) [2025-08-14 09:02:53.763068] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [720][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571f9d70, caller:ob_table_lock_service.cpp:start:276, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE6$_1273, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa090, time_wheel:0x7f99375f7d10, thread_pool:0x7f99375f74f0, is_running:0, time_interval:10.00s, expected_run_time:09:03:02.332, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.767440] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [720][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=4363][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:02:53.767|total=1.35s) [2025-08-14 09:02:53.768781] WDIAG [STORAGE] check_and_do_freeze (ob_tenant_freezer.cpp:892) [677][T1001_Occam][T1001][YB42AC4D0007-00063C4967999C34-0-0] [lt=1000941][errcode=-4389] check and do freeze spend too much time(spend_time=191014853) [2025-08-14 09:02:53.768797] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [677][T1001_Occam][T1001][YB42AC4D0007-00063C4967999C34-0-0] [lt=14][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571fa190, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa470, time_wheel:0x7f99341cdbf0, thread_pool:0x7f99341cd3d0, is_running:0, time_interval:2.00s, expected_run_time:09:02:24.334, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.768816] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [677][T1001_Occam][T1001][YB42AC4D0007-00063C4967999C34-0-0] [lt=18][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:02:53.768|total=191.36s) [2025-08-14 09:02:53.749256] WDIAG [ARCHIVE] load_archive_round_attr (ob_archive_persist_mgr.cpp:294) [863][T1002_ArcSrv][T1002][YB42AC4D0007-00063C495E699C16-0-0] [lt=989724][errcode=-4012] get valid dest pair failed(ret=-4012, tenant_id_=1002) [2025-08-14 09:02:53.769411] WDIAG [ARCHIVE] persist_archive_progress_ (ob_archive_persist_mgr.cpp:341) [863][T1002_ArcSrv][T1002][YB42AC4D0007-00063C495E699C16-0-0] [lt=20121][errcode=-4012] load archive round attr failed(ret=-4012, attr={key:{tenant_id:0, dest_no:-1}, incarnation:1, dest_id:0, round_id:0, state:{status:"INVALID"}, start_scn:{val:0, v:0}, checkpoint_scn:{val:0, v:0}, max_scn:{val:0, v:0}, compatible:{version:1}, base_piece_id:0, used_piece_id:0, piece_switch_interval:0, frozen_input_bytes:0, frozen_output_bytes:0, active_input_bytes:0, active_output_bytes:0, deleted_input_bytes:0, deleted_output_bytes:0, path:"", comment:""}) [2025-08-14 09:02:53.770961] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16392] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:53.771020] WDIAG [STORAGE.TRANS] get_write_store_ctx (ob_trans_service_v4.cpp:1140) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=10][errcode=-4038] acquire ls snapshot for mvcc write fail(ret=-4038, ls_id={id:1}) [2025-08-14 09:02:53.773496] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:53.760, task_priority:1, with_handle_protected:1}) [2025-08-14 08:58:20.253501] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [579][T1_MultiVersion][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fb900, caller:ob_multi_version_garbage_collector.cpp:start:115, function_type:ZN9oceanbase19concurrency_control30ObMultiVersionGarbageCollector5startEvE6$_1159, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fbbe0, time_wheel:0x7f994b1d7b50, thread_pool:0x7f994b1d7330, is_running:1, time_interval:60.00s, expected_run_time:08:58:19.702, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.773861] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [579][T1_MultiVersion][T1][Y0-0000000000000000-0-0] [lt=273107201][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 08:58:20.253|207=273.52s|210=0us|222=1us|total=273.52s) [2025-08-14 09:02:53.773877] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [579][T1_MultiVersion][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] timer task use too much time(end=1755162173773877, start=1755161900253498, delta=273520379) [2025-08-14 09:01:50.791080] EDIAG [ELECT] prepare (election_proposer.cpp:380) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3935521, owner: ObLog, click_count: 7, time dist:[FORMAT_END=2936918, ObMalloc#start=1, ObMalloc#end=989358, ALLOC_END=9244, APPEND_END=0, ALLOC_BLOCK#start=8, ALLOC_BLOCK#end=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0xf9d67b1 0xf9d4184 0xfa09ca2 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:53.774306] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:53.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.774334] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:53.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.774348] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891815730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:53.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.774399] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:53.775694] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=13129][errcode=-4015] clock out of order(cur_ts=1755162173775690, cur_ts_=1755162173754375, delta=21315) [2025-08-14 09:02:53.776127] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EEC5-0-0] [lt=5849] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-08-14 09:02:53.777753] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:02:53.777780] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=24] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:02:53.777757] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:53.782288] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=936764] gc stale ls task succ [2025-08-14 09:02:53.784377] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eca90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:53.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.784407] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e94b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:53.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.784421] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:53.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.784435] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180ea30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:53.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.794413] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:53.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.794443] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:53.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.794456] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196be10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:53.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.794469] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728d430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7290f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:53.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.803760] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:53.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.804473] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:53.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.804494] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73922b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:53.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.804507] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:53.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.804519] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:53.799, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.804531] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:53.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.814531] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:53.809, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.814555] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eaa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:53.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.815219] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] [sql nio session](*s={this:0x7f98f72d0030, session_id:3221589415, trace_id:YB42AC4D0007-00063C4965FA43DC-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:245}, err:0, last_decode_time:1755161751714366, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1662, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:53.815266] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=46] [sql nio session](*s={this:0x7f98f72d5030, session_id:3221590200, trace_id:YB42AC4D0007-00063C4965FA43E0-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:248}, err:0, last_decode_time:1755161776777167, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1593, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:53.819904] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7] get wrs ts(ls_id={id:1}, delta=454044036, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:53.819927] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:53.827156] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=697378][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:02:53.827191] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=700568][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:02:53.827252] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [571][T1_ArcSender][T1][YB42AC4D0007-00063C495BA99C16-0-0] [lt=704280][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:02:53.829750] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1}, delta=626091224, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:53.829772] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:53.831412] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:53.831655] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:53.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.831810] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=4] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:53.832254] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=1584966] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:02:53.832276] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=4][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:53.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.838938] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1}, delta=424097065, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:53.838956] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:53.839548] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1001}, delta=429111494, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:53.839563] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:53.844705] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73945b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:53.837, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.854321] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a4d58, timer:0x7f994b1dfb70, task:0x7f991f004030, task_type:"N9oceanbase8observer20ObTableQueryASyncMgrE", scheduled_time:1755161750817104, pushed_time:0, delay:0}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=423037215, threashold=1000000, tenant_id=1002) [2025-08-14 09:02:53.865077] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] get_fetch_log_dst failed or invalid(ret=0, palf_id=1001, self="172.77.0.7:2882", fetch_log_dst="172.77.0.7:2882") [2025-08-14 09:02:53.920004] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=454144086, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:53.920025] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:53.925169] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:53.911, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:53.929692] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=626190261, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:53.929711] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:53.931473] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=14] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:53.931717] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:53.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.931821] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=13] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:53.932337] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:53.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.938986] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=424197093, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:53.939005] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:53.939513] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=429211531, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:53.939524] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:53.975260] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=748616] PNIO [ratelimit] time: 1755162173975259, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007615, add_bytes: 0 [2025-08-14 09:02:53.993442] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:53.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.017652] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:54.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.017685] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:54.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.025040] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:54.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.031529] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=14] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:54.031769] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:53.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.031832] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=11] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:54.098128] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:54.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.126244] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=669852][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a4578, timer:0x7f9956bffee0, task:0x7f9969bb3028, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr17RefreshConfigTaskE", scheduled_time:1755161932756260, pushed_time:0, delay:10000000}, running_task_token={this:0x7f994e1a4498, timer:0x7f9956bffee0, task:0x7f9969bb3010, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr11TableGCTaskE", scheduled_time:1755161923356444, pushed_time:1755161923356659, delay:20000}, total_delay=241369983, threashold=1000000, tenant_id=1) [2025-08-14 09:02:54.131578] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=23] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:54.131827] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:54.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.131889] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=10] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:54.163399] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:02:54.163450] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:02:54.163488] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:02:54.163534] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:02:54.163552] INFO [COMMON] print_io_status (ob_io_manager.cpp:2639) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:2, norm_iops:0, bw:10203, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:12237, total:12242) [2025-08-14 09:02:54.163572] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, param_config_:{memory_limit:536870912, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=865568, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:02:54.163616] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2826) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=40] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:3, avg_bw:10203, [delay/us]: prepare:0, schedule:0, submit:600000000, device:12237, total:12242) [2025-08-14 09:02:54.163640] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:02:54.163651] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:54.218105] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:54.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.218929] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:54.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.223435] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=2967][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:54.223464] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:54.223476] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:54.223493] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:54.223504] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:54.223521] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:54.223538] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:01:49.731, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:01:42.751], idx=659, next_allow_print_ts="09:02:19.731") [2025-08-14 09:02:54.223554] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:54.223565] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:54.223576] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:54.231393] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=6] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:54.231411] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162174231389], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:54.231433] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162174231389], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:54.231446] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162174231389]}) [2025-08-14 09:02:54.231459] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:54.231467] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162174231458], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:54.231478] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162174231458]}) [2025-08-14 09:02:54.231671] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:54.231688] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:54.231702] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162174231668], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:54.231712] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162174231668]}) [2025-08-14 09:02:54.231903] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=10] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:54.231926] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=26] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:54.231940] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:54.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.269272] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:54.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.269307] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:54.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.276775] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:54.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.306969] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:54.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.318167] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:54.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.331965] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=11] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:54.331990] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:54.332011] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:54.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.418230] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:54.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.431987] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=18] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:54.432065] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=14] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:54.432085] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:54.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.511579] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:54.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.511621] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:54.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.518270] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:54.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.528260] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:54.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.532058] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=20] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:54.532124] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:54.532144] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:54.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.592118] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:54.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.618331] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=50][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:54.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.632117] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=21] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:54.632211] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:54.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.632259] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:54.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.712921] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:54.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.718404] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:54.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.020053] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=454244833, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:54.720194] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=699079] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=8, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:54.032414] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:53.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.720534] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1936] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:54.720434] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=686978][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:54.324|207=688.02ms|210=0us|222=2us|total=688.16ms) [2025-08-14 09:02:54.720581] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=145][errcode=-4389] timer task use too much time(end=1755162174720580, start=1755162174032413, delta=688167) [2025-08-14 09:02:54.720654] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:54.718, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:54.632190] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [567]OB_MALLOC COST TOO MUCH TIME, cost_time=697169, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [857]OB_MALLOC COST TOO MUCH TIME, cost_time=688524, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:02:54.722318] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:54.223588] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:54.723636] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=499133][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:54.723657] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:54.724693] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=-4389] run this task use too much time(*p_task_={this:0x7f996f40b7f0, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40bad0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:0, time_interval:10.00s, expected_run_time:09:02:58.802, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:53.748808] EDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1076) [1029][ObStorageHADiag][T1][YB42AC4D0007-00063C4966799D69-0-0] [lt=29][errcode=-4012] LOGGER COST TOO MUCH TIME, cost: 1000601, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1000596, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x12ead58e 0x981b5d1 0x124ddd52 0x96a5a41 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x2044675f 0x1d70b480 0x1d7075ca 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:56.725534] EDIAG ev_invoke_pending (ev.c:2136) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=971553][errcode=0] LOGGER COST TOO MUCH TIME, cost: 57931281, owner: ObLog, click_count: 3, time dist:[FORMAT_END=57931279, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 08:59:06.825466] EDIAG [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [14963][T1_TimerWK9_TabletShell][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 227577256, owner: ObLog, click_count: 5, time dist:[FORMAT_END=10, ObMalloc#start=0, ObMalloc#end=1941174, ALLOC_END=225636071, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1c45bdeb 0x1c45a909 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:54.724710] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:02:54.724|total=74.99s) [2025-08-14 09:02:54.724725] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] task executed too delay(delay_time=74946371, *p_task_={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:0, time_interval:250.00ms, expected_run_time:09:02:54.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.724792] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=47][errcode=-4016] task executed too delay(delay_time=74916227, *p_task_={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:0, time_interval:500.00ms, expected_run_time:09:02:54.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.154621] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=697787][errcode=0] EASY SLOW: thread: 7f99542fb300, lastrun: 1755162115.752503 cost: 58.402117 loop:26416, slowcnt: 483 [2025-08-14 09:02:54.246051] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:54.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:53.826336] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=984663] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:02:54.726693] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=479912][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:54.246|207=480.64ms|210=0us|222=2us|total=480.65ms) [2025-08-14 09:02:53.806065] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=1002930] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:02:54.726708] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] timer task use too much time(end=1755162174726708, start=1755162174246049, delta=480659) [2025-08-14 09:02:54.726720] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=482086, expected_run_ts_us_=1755162174244633, ObClockGenerator::getRealClock()=1755162174726719, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:54.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.726734] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:54.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.727466] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=942546] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:02:54.728028] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=626982372, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:54.728053] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=8, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:54.728083] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161101770923, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755161572801635) [2025-08-14 09:02:53.803715] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=987016] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:02:54.728371] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=895136] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:02:54.728403] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:54.728415] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:54.728633] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=424979695, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:54.728655] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:54.729261] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=429993064, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:54.729276] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:54.729306] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=424979695, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:54.729316] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:54.729845] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=429993064, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:54.729857] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:54.729998] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [941][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F91-0-0] [lt=24][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1002, svr="172.77.0.7:2882", version={val:1755161758221798000, v:0}, valid_part_count=0, total_part_count=2, generate_timestamp=1755162174039001) [2025-08-14 09:02:54.730025] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [941][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F91-0-0] [lt=21][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1002, req={req_server:"172.77.0.7:2882", version:{val:1755161758221798000, v:0}, valid_part_count:0, total_part_count:2, generate_timestamp:1755162174039001}, twrs={inited:true, tenant_id:1002, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162174729298}, server_version_for_stat:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162174729298}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:54.730431] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=626982372, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:54.730447] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:54.731560] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:54.726, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:54.731589] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a67b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7397290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:54.728, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:54.732198] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=18] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:54.732248] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:54.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.732402] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=32] get wrs ts(ls_id={id:1}, delta=454945709, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:54.732420] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:54.732501] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=424979695, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:54.732515] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:54.733057] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=429993064, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:54.733073] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:54.733171] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [316][pnio1][T1002][YB42AC4D0007-00063C49612B8F91-0-0] [lt=8][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1002, delta=997333, last_succ_cluster_heartbeat_tstamp_=1755162173735838) [2025-08-14 09:02:54.728666] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=785] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:54.262667] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:54.736317] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=472933] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3973) [2025-08-14 09:02:54.736328] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:54.736346] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=6] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:54.736353] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3974) [2025-08-14 09:02:54.736358] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:54.736513] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:54.736521] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3975) [2025-08-14 09:02:54.736525] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=3][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:54.741484] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=45] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:54.741587] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:54.741645] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919d9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:54.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:54.741668] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7277c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eac90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:54.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:54.742428] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:54.163662] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:54.744195] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=579651] [IO STATUS SERVER] net_in=0kB/s, net_out=0kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=1280000kB/s [2025-08-14 09:02:54.736859] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd110, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd3f0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:10.00s, expected_run_time:09:02:54.726, task_priority:1, with_handle_protected:1}) [2025-08-14 09:01:56.725534] WDIAG ev_invoke_pending (ev.c:2136) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=971553][errcode=0] epoll handler too much time :973031 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:54.744465] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=57931281] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:54.744480] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:54.745416] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:53.771041] WDIAG [STORAGE] get_write_store_ctx_guard_ (ob_access_service.cpp:528) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=16][errcode=-4038] can not get write store ctx(ret=-4038, ls_id={id:1}, snapshot={this:0x7f9956cd2798, valid:true, source:5, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:21195872}, scn:{branch:0, seq:2}}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, tx_desc={this:0x7f98c907e020, tx_id:{txid:21195872}, state:1, addr:"172.77.0.7:2882", tenant_id:1, session_id:1, assoc_session_id:1, client_sid:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1755162170771423, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999987, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1753069619, cluster_version:17180067074, seq_base:1755162170724889, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:2}) [2025-08-14 09:02:54.745472] WDIAG [STORAGE] get_write_store_ctx_guard (ob_access_service.cpp:497) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=972956][errcode=-4038] fail to get write store ctx gurad(ret=-4038, ls_id={id:1}, tx_desc={this:0x7f98c907e020, tx_id:{txid:21195872}, state:1, addr:"172.77.0.7:2882", tenant_id:1, session_id:1, assoc_session_id:1, client_sid:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1755162170771423, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999987, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1753069619, cluster_version:17180067074, seq_base:1755162170724889, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:2}) [2025-08-14 09:02:53.717945] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:662) [826][T1002_CdcSrv][T1001][YB42AC4D0007-00063C495D29A61C-0-0] [lt=1700][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f98b0db3980, tx_id:{txid:0}, state:1, addr:"172.77.0.7:2882", tenant_id:1001, session_id:3221590971, assoc_session_id:3221590971, client_sid:3221590971, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1755162170772580, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180067074, seq_base:1755162170724890, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}) [2025-08-14 09:02:54.731947] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=8] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:54.746092] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=14121][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162174731943], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:54.746105] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162174731943], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:54.746120] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162174731943]}) [2025-08-14 09:02:54.746129] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162174731943]}) [2025-08-14 09:02:54.746138] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:54.746144] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:54.746156] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=6] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:54.746163] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162174746153], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:54.746170] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162174746153]}) [2025-08-14 09:02:54.746176] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162174746153]}) [2025-08-14 09:02:54.746344] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:54.746353] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:54.746362] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162174746340], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:54.746374] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162174746340]}) [2025-08-14 09:02:54.746380] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162174746340]}) [2025-08-14 09:02:54.746861] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:54.733, task_priority:1, with_handle_protected:1}) [2025-08-14 08:59:06.825466] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [14963][T1_TimerWK9_TabletShell][T1][Y0-0000000000000000-0-0] [lt=11] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-08-14 09:02:54.747328] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:54.747351] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:54.750164] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4640] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:54.751709] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:54.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:54.751730] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:54.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:54.735880] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=88358] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:54.753960] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [1029][ObStorageHADiag][T1][YB42AC4D0007-00063C4966799D69-0-0] [lt=1000601][errcode=-4283] result set close failed(ret=-4283) [2025-08-14 09:02:54.754565] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:54.755252] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:54.756165] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=25] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:54.756220] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=77] temporary file module use no blocks [2025-08-14 09:02:54.756230] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:54.756251] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=19] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:54.756264] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623076194125, flush_io_finished_ret_:0, flush_io_finished_round_:26330, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:54.756497] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6178, timer:0x7f99200ff180, task:0x7f99200ff220, task_type:"N9oceanbase8observer18ObTableLoadService21ObClientTaskPurgeTaskE", scheduled_time:1755162052780445, pushed_time:0, delay:1000000}, running_task_token={this:0x7f991b7a5fb8, timer:0x7f99200ff180, task:0x7f99200ff1f0, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162050755138, pushed_time:1755162084760013, delay:1000000}, total_delay=121976051, threashold=1000000, tenant_id=1002) [2025-08-14 09:02:54.756881] WDIAG [SQL.DAS] write_tablet (ob_dml_service.h:406) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=33][errcode=-4038] fail to get_write_store_ctx_guard(ret=-4038, ls_id_={id:1}) [2025-08-14 09:02:54.757437] WDIAG [SQL.DAS] insert_rows (ob_das_insert_op.cpp:136) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=548][errcode=-4038] insert rows to access service failed(ret=-4038) [2025-08-14 09:02:54.757450] WDIAG [SQL.DAS] open_op (ob_das_insert_op.cpp:111) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=11][errcode=-4038] fail to do insert(ret=-4038) [2025-08-14 09:02:54.757461] WDIAG [SQL.DAS] start_das_task (ob_das_task.cpp:353) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=8][errcode=-4038] open das task op failed(ret=-4038) [2025-08-14 09:02:54.757497] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4015] clock out of order(cur_ts=1755162174757496, cur_ts_=1755162174736331, delta=21165) [2025-08-14 09:02:54.756920] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [14963][T1_TimerWK9_TabletShell][T1][Y0-0000000000000000-0-0] [lt=227577256] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=7) [2025-08-14 09:02:54.758172] WDIAG alarm_if_necessary (ob_timer_service.cpp:143) [14963][T1_TimerWK9][T1][Y0-0000000000000000-0-0] [lt=1246][errcode=-4389] timer task's elapsed_time is greater than period(token={this:0x7f994e1a5378, timer:0x7f994b3efe78, task:0x7f994b3efea0, task_type:"N9oceanbase7storage10checkpoint16ObEmptyShellTaskE", scheduled_time:1755161946825383, pushed_time:1755161946825449, delay:5000000}, elapsed_time=227932711, period=5000000) [2025-08-14 09:02:54.758252] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=5] gc stale ls task succ [2025-08-14 09:02:54.758228] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3927] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:54.758394] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:54.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.753796] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:981) [712][T1001_GCCollect][T1][YB42AC4D0007-00063C495FA9A613-0-0] [lt=154811264][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1001"}, record_ret=-4283, ret=-4283) [2025-08-14 09:02:54.759226] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:199) [1029][ObStorageHADiag][T1][YB42AC4D0007-00063C4966799D69-0-0] [lt=8][errcode=-4283] result set close failed(ret=-4283) [2025-08-14 09:02:54.759247] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:951) [1029][ObStorageHADiag][T1][YB42AC4D0007-00063C4966799D69-0-0] [lt=21][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2025-08-14 09:02:54.759268] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:981) [1029][ObStorageHADiag][T1][YB42AC4D0007-00063C4966799D69-0-0] [lt=10][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"select time_to_usec(gmt_create) as create_ts from __all_storage_ha_error_diagnose_history where gmt_create > usec_to_time(0) order by gmt_create limit 100"}, record_ret=-4012, ret=-4012) [2025-08-14 09:02:54.759278] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:54.759284] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:763) [1029][ObStorageHADiag][T0][YB42AC4D0007-00063C4966799D69-0-0] [lt=12] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"select time_to_usec(gmt_create) as create_ts from __all_storage_ha_error_diagnose_history where gmt_create > usec_to_time(0) order by gmt_create limit 100"}, process_time=334010633) [2025-08-14 09:02:54.759295] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1018) [1029][ObStorageHADiag][T0][YB42AC4D0007-00063C4966799D69-0-0] [lt=12][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"select time_to_usec(gmt_create) as create_ts from __all_storage_ha_error_diagnose_history where gmt_create > usec_to_time(0) order by gmt_create limit 100"}, aret=-4012, ret=-4012) [2025-08-14 09:02:54.759311] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1808) [1029][ObStorageHADiag][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=select time_to_usec(gmt_create) as create_ts from __all_storage_ha_error_diagnose_history where gmt_create > usec_to_time(0) order by gmt_create limit 100) [2025-08-14 09:02:54.759325] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1079) [1029][ObStorageHADiag][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2025-08-14 09:02:54.759336] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1748) [1029][ObStorageHADiag][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1753069619, tenant_id=1) [2025-08-14 09:02:54.759348] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [1029][ObStorageHADiag][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4012] query failed(ret=-4012, conn=0x7f98de527780, start=1755161840746067, sql=select time_to_usec(gmt_create) as create_ts from __all_storage_ha_error_diagnose_history where gmt_create > usec_to_time(0) order by gmt_create limit 100) [2025-08-14 09:02:54.759365] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [1029][ObStorageHADiag][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4012] read failed(ret=-4012) [2025-08-14 09:02:54.760682] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=8522] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:54.762624] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:54.762836] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7277230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:54.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:54.762866] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7272330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a5090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:54.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:54.762883] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919822b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:54.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:54.762897] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:54.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:54.762910] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d71b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:54.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:54.762923] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:54.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:54.762935] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a27b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:54.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:54.762946] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a40b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:54.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:54.762958] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7faf30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:54.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:54.763350] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=12] gc stale ls task succ [2025-08-14 09:02:54.765108] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=11] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:54.757470] WDIAG [SQL.DAS] do_local_das_task (ob_data_access_service.cpp:375) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=7][errcode=-4038] start local das task failed(ret=-4038) [2025-08-14 09:02:54.769994] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:54.771267] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=7558][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:54.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.771308] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:54.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.771525] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=23] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:54.771551] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=23] temporary file module use no blocks [2025-08-14 09:02:54.771557] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=5] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:54.771574] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=16] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:54.771591] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=16] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623076211229, flush_io_finished_ret_:0, flush_io_finished_round_:26342, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:54.772254] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:54.772886] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7352330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196fc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:54.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:54.777391] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=41] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:54.779314] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=1155] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:02:54.782334] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=24] gc stale ls task succ [2025-08-14 09:02:54.782913] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aca90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:54.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:54.782945] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f734fc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891918610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:54.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:54.793009] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:54.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:54.793039] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:54.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:54.803080] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:54.794, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:54.809809] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:54.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.813122] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:54.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:54.813153] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:54.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:54.813166] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:54.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:54.813179] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7397b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:54.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:54.813192] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918299b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891916210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:54.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:54.820661] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:54.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.826864] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=455050063, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:54.826891] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:54.828476] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:54.830268] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=627091749, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=253148, succ_cluster_heartbeat_count=2207, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161000606550000, v:0}, local_cluster_delta=1174223715, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:02:54.830346] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=38] get_max_applied_scn(scn={val:1755161547738516001, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:50, palf_committed_end_lsn_:{lsn:5282737655}, palf_committed_end_scn_:{val:1755161547738516001, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161547738516001, v:0}}) [2025-08-14 09:02:54.830382] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31] get_min_unreplayed_log_info(lsn={lsn:5282737655}, scn={val:1755100947825033001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161624848121, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:13063726, ret_code_:0}}, next_to_submit_lsn_:{lsn:5282737655}, next_to_submit_scn_:{val:1755100947825033001, v:0}, base_lsn_:{lsn:4831543296}, base_scn_:{val:1755063445733337001, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7605000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:5282737655}, end_lsn:{lsn:5282737655}, read_buf:{buf_len_:3747840, buf_:0x7f98f7605000}, block_size:67104768, log_storage_:0x7f993e529bf0, log_storage_:{log_tail:{lsn:5282737655}, readable_log_tail:{lsn:5282737655}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:5234171904}, min_scn:{val:1755152244802535004, v:0}, curr_block_id:78, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:206, min_block_id:42, max_block_id:79, curr_writable_block_id:78}, logical_block_size_:67104768, curr_block_writable_size_:18539017, block_header_serialize_buf_:0x7f993e52a8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:85, read_io_size:301367296, read_disk_cost_ts:2364604, start_lsn:{lsn:4831543296}}}}}}) [2025-08-14 09:02:54.831046] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=85] get wrs ts(ls_id={id:1}, delta=627091699, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:54.831066] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:54.832222] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=10] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:54.832342] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:54.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.832577] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=425091139, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:54.832608] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:54.832918] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=11] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:02:54.833187] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1001}, delta=430104508, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:54.833201] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:54.833326] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:54.825, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:54.833355] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:54.826, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:54.833367] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:54.820, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:54.833379] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72879b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:54.828, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:54.865860] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:02:54.865887] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:02:54.920720] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:54.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.926952] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] get wrs ts(ls_id={id:1}, delta=455150613, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:54.926975] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:54.928533] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:54.930931] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=627191686, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:54.930962] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:54.932294] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=28] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:54.932384] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:54.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.932654] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=425191136, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:54.932676] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:54.933297] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27] get wrs ts(ls_id={id:1001}, delta=430204505, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:54.933314] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:54.982888] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=27] PNIO [ratelimit] time: 1755162174982886, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007627, add_bytes: 0 [2025-08-14 09:02:54.990029] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:54.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.012616] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:55.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.012654] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:55.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.020781] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:54.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.021786] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:55.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.026417] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get_max_applied_scn(scn={val:1755161719775179002, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:45, palf_committed_end_lsn_:{lsn:8461098908}, palf_committed_end_scn_:{val:1755161719775179002, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161719775179002, v:0}}) [2025-08-14 09:02:55.026445] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] get_min_unreplayed_log_info(lsn={lsn:8461098908}, scn={val:1755100947600130001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161920778323, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:23084275, ret_code_:0}}, next_to_submit_lsn_:{lsn:8461098908}, next_to_submit_scn_:{val:1755100947600130001, v:0}, base_lsn_:{lsn:7515734016}, base_scn_:{val:1753840867171901045, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6e05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:8461098908}, end_lsn:{lsn:8461098908}, read_buf:{buf_len_:3747840, buf_:0x7f98f6e05000}, block_size:67104768, log_storage_:0x7f99297a3bf0, log_storage_:{log_tail:{lsn:8461098908}, readable_log_tail:{lsn:8461098908}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8455200768}, min_scn:{val:1755160384851871002, v:0}, curr_block_id:126, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:215, min_block_id:93, max_block_id:127, curr_writable_block_id:126}, logical_block_size_:67104768, curr_block_writable_size_:61206628, block_header_serialize_buf_:0x7f99297a48b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:198, read_io_size:699822080, read_disk_cost_ts:4178035, start_lsn:{lsn:7515734016}}}}}}) [2025-08-14 09:02:55.027106] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=74] get wrs ts(ls_id={id:1}, delta=455251029, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:55.027123] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.028598] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:02:55.028636] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:55.033513] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=425291372, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:55.033535] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.033853] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:54.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.034072] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=627294049, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:55.034090] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.034120] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1001}, delta=430304741, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:55.034132] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.034214] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B92A8-0-0] [lt=29][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162175033297) [2025-08-14 09:02:55.034251] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B92A8-0-0] [lt=31][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162175033297}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162175033509}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162175033509}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:55.034342] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [316][pnio1][T1][YB42AC4D0007-00063C49607B92A8-0-0] [lt=13][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1, delta=4303435, last_succ_cluster_heartbeat_tstamp_=1755162170730906) [2025-08-14 09:02:55.034914] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=627295790, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:55.034934] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.094559] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:55.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.120854] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:55.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.127042] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=455351022, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:55.127064] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.128657] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:55.132742] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=425391385, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:55.132762] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.133339] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1001}, delta=430404754, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:55.133351] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.133927] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:55.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.135049] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=627396177, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:55.135067] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.215692] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:55.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.220881] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:55.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.223822] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:55.223856] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:55.223875] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:55.223895] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:55.223907] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:55.223926] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:55.223952] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:55.223966] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:55.223978] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:55.223992] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:55.224166] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:55.224181] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:55.227092] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=455451788, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:55.227114] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.228732] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:55.232828] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=425491125, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:55.232847] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.233438] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=430505567, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:55.233449] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.234017] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:55.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.235028] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=627495934, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:55.235047] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.236610] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:55.236633] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=23] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3976) [2025-08-14 09:02:55.236672] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=38][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:55.236686] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:55.236695] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3977) [2025-08-14 09:02:55.236702] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:55.236886] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:55.236897] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3978) [2025-08-14 09:02:55.236905] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:55.241814] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:55.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.244693] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:02:55.244751] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:02:55.244794] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:02:55.244836] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:02:55.244857] INFO [COMMON] print_io_status (ob_io_manager.cpp:2639) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:3788, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:9819, total:9825) [2025-08-14 09:02:55.244879] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, param_config_:{memory_limit:536870912, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=865568, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:02:55.244918] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2826) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=34] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:3788, [delay/us]: prepare:0, schedule:0, submit:600000000, device:9819, total:9825) [2025-08-14 09:02:55.244940] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:02:55.244955] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:55.244968] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:55.244978] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SERVER] net_in=0kB/s, net_out=0kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=1280000kB/s [2025-08-14 09:02:55.246578] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=7] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:55.246596] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162175246575], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:55.246622] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162175246575], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:55.246637] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162175246575]}) [2025-08-14 09:02:55.246650] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=11] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:55.246659] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162175246649], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:55.246670] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162175246649]}) [2025-08-14 09:02:55.246839] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:55.246851] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:55.246871] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162175246836], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:55.246883] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162175246836]}) [2025-08-14 09:02:55.251886] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:55.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.266026] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:55.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.266060] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:55.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.273553] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:55.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.303762] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:55.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.320941] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:55.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.327176] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=455551146, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:55.327201] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.328776] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:55.332897] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=425591482, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:55.332921] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.333534] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1001}, delta=430604851, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:55.333552] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.334417] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:55.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.335134] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=627596275, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:55.335156] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.420995] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:55.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.426927] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [758][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB90E0-0-0] [lt=37][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1001, svr="172.77.0.7:2882", version={val:1755161719775179002, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162175426573) [2025-08-14 09:02:55.426954] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [758][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB90E0-0-0] [lt=26][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1001, req={req_server:"172.77.0.7:2882", version:{val:1755161719775179002, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162175426573}, twrs={inited:true, tenant_id:1001, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162175326567}, server_version_for_stat:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162175326567}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:55.427045] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [316][pnio1][T1001][YB42AC4D0007-00063C4960CB90E0-0-0] [lt=9][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1001, delta=3686774, last_succ_cluster_heartbeat_tstamp_=1755162171740270) [2025-08-14 09:02:55.427403] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=455650524, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:55.427427] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.428056] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=455650524, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:55.428077] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.428837] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:55.432901] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1}, delta=425691275, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:55.432931] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.433534] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1001}, delta=430705704, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:55.433547] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.434471] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:55.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.435166] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=627696085, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:55.435186] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.496330] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:55.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.518127] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:55.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.518169] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:55.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.521055] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:55.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.526636] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:55.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.528160] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=455752228, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:55.528186] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.528896] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:55.532992] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=425791487, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:55.533015] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.533600] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1001}, delta=430804856, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:55.533626] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.534533] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:55.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.535200] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=627796298, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:55.535223] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.598624] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:55.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.621122] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:55.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.628274] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=29] get wrs ts(ls_id={id:1}, delta=455851772, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:55.628303] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.628924] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:55.633053] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=425891313, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:55.633075] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.633683] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1001}, delta=430905740, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:55.633703] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.634196] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:55.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.634588] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:55.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.635264] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=627896108, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:55.635284] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.701448] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEA7-0-0] [lt=10] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:02:55.714747] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=-4016] task executed too delay(delay_time=935162, *p_task_={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:0, time_interval:250.00ms, expected_run_time:09:02:55.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.719364] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:55.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.719827] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=17867] tx gc loop thread is running(MTL_ID()=1) [2025-08-14 09:02:55.719843] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15] try gc retain ctx [2025-08-14 09:02:55.719852] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] try to retry start_working [2025-08-14 09:02:55.720962] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72760b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.719, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.721561] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:55.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.722046] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=24] tx gc loop thread is running(MTL_ID()=1001) [2025-08-14 09:02:55.722064] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=17] try gc retain ctx [2025-08-14 09:02:55.722072] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=8] try to retry start_working [2025-08-14 09:02:55.722118] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:55.722147] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:55.725407] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:55.728170] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=14] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:02:55.729253] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7626] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:55.736225] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fb3b0, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fb690, time_wheel:0x7f99589fd990, thread_pool:0x7f9959fff170, is_running:1, time_interval:2.00s, expected_run_time:09:02:55.698, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.736286] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:55.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.736498] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:55.725448] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:55.736685] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11217][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:55.736732] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:55.736744] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:55.736763] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:55.736780] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:55.736793] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:55.736804] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:55.736817] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:55.736992] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:55.737007] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:55.728998] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:55.719811] WDIAG [SHARE] get_batch_row_keys (ob_storage_ha_diagnose_operator.cpp:76) [1029][ObStorageHADiag][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4012] fail to execute sql(sql=select time_to_usec(gmt_create) as create_ts from __all_storage_ha_error_diagnose_history where gmt_create > usec_to_time(0) order by gmt_create limit 100, ret=-4012, tenant_id=1) [2025-08-14 09:02:55.736443] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:213) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-08-14 09:02:55.737741] INFO [COMMON] handle (memory_dump.cpp:544) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=1294] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-08-14 09:02:55.738476] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:55.738518] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=41] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3979) [2025-08-14 09:02:55.738530] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:55.738546] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:55.738568] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=22] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3980) [2025-08-14 09:02:55.738575] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:55.738765] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=6] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:55.738781] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:227) [1031][PxTargetMgr0][T1002][Y0-0000000000000000-0-0] [lt=12] refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3981) [2025-08-14 09:02:55.738791] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:55.739267] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=627979619, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:55.739289] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.741180] INFO [SERVER] process (obmp_connect.cpp:504) [837][T1002_MysqlQueu][T1002][Y0-00063C4963099C4A-0-0] [lt=7525] MySQL LOGIN(direct_client_ip="172.77.0.1", client_ip=172.77.0.1, tenant_name=mdsk, tenant_id=1002, user_name=root, host_name=%, sessid=3221590970, proxy_sessid=0, client_sessid=3221590970, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-08-14 09:02:55.741559] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=14] update_check_range(min_check_version=2540, max_check_version=2540, global_version=2643) [2025-08-14 09:02:55.742069] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:54.776077] EDIAG [STORAGE] prepare_mds_tables (ob_multiple_merge.cpp:1662) [14945][T1_TimerWK8_MergeLoop][T1][YB42AC4D0007-00063C4978199C2C-0-0] [lt=10][errcode=-4012] LOGGER COST TOO MUCH TIME, cost: 1957081, owner: ObLog, click_count: 7, time dist:[FORMAT_END=1937332, ObMalloc#start=1, ObMalloc#end=19743, ALLOC_END=2, APPEND_END=3, ALLOC_BLOCK#start=6, ALLOC_BLOCK#end=1], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1d7eac00 0x9519b17 0x9361c6d 0x94fbe0f 0x94b4970 0x9766028 0x9765ae5 0x976579e 0x9764de3 0x975ab8f 0x97f7c68 0x97f78b3 0x97f7636 0x97f5260 0x97f4a4a 0x97f4859 0x97f4305 0x1aee6975 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:55.745287] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:55.745383] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] finish update min start scn(max_decided_scn={val:1755161547738516001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1755161493768447002, v:0}, keep_alive_scn_:{val:1755161493768447002, v:0}, update_ts_:1755162170730986}) [2025-08-14 09:02:55.747148] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:55.747169] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162175747142], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:55.747188] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162175747142], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:55.747204] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162175747142]}) [2025-08-14 09:02:55.747218] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=63, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:02:55.747242] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=20] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:55.747249] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162175747240], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:55.747427] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:55.747439] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:55.747453] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162175747423], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:55.747466] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162175747423]}) [2025-08-14 09:02:55.748038] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.748069] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.748082] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739be30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.748095] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.748178] INFO [SERVER] process (obmp_connect.cpp:504) [839][T1002_MysqlQueu][T1002][Y0-00063C4965499C46-0-0] [lt=12757] MySQL LOGIN(direct_client_ip="172.77.0.1", client_ip=172.77.0.1, tenant_name=mdsk, tenant_id=1002, user_name=ds_user, host_name=%, sessid=3221590972, proxy_sessid=0, client_sessid=3221590972, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=2073103, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-08-14 09:02:55.739713] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=7] finish update min start scn(max_decided_scn={val:1755161719775179002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1755161287778691002, v:0}, keep_alive_scn_:{val:1755161287778691002, v:0}, update_ts_:1755162170741302}) [2025-08-14 09:02:55.749795] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:55.749831] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:55.749867] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=403] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:55.749911] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:55.750185] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:55.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.750218] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:55.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.750473] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:274) [863][T1002_ArcSrv][T1001][YB42AC4D0007-00063C495E699C16-0-0] [lt=46][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=570) [2025-08-14 09:02:55.750488] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:953) [863][T1002_ArcSrv][T1001][YB42AC4D0007-00063C495E699C16-0-0] [lt=14][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2025-08-14 09:02:55.750506] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [863][T1002_ArcSrv][T1001][YB42AC4D0007-00063C495E699C16-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:55.751447] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:55.752317] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [863][T1002_ArcSrv][T1001][YB42AC4D0007-00063C495E699C16-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:55.753419] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:55.753562] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [863][T1002_ArcSrv][T1001][YB42AC4D0007-00063C495E699C16-0-0] [lt=23][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:00:15.767279] EDIAG [STORAGE] init_and_open_iter (ob_mds_row_iterator.cpp:232) [805][T1002_TimerWK1_MediumLoop][T1002][YB42AC4D0007-00063C495F7A19FC-0-0] [lt=954237][errcode=-4012] LOGGER COST TOO MUCH TIME, cost: 7982985, owner: ObLog, click_count: 5, time dist:[FORMAT_END=7982973, ObMalloc#start=2, ObMalloc#end=3, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1bc4ae6e 0x9766310 0x9765ae5 0x976579e 0x9764de3 0x975ab8f 0x97f7c68 0x97f78b3 0x9800989 0x980049e 0x97fe9af 0x1c353d12 0x1aee74b9 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:55.754633] WDIAG [STORAGE] init (ob_mds_row_iterator.cpp:75) [805][T1002_TimerWK1_MediumLoop][T1002][YB42AC4D0007-00063C495F7A19FC-0-0] [lt=7982985][errcode=-4012] fail to init and open iter(ret=-4012) [2025-08-14 09:02:55.757005] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:55.757270] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=57] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:55.757501] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] get wrs ts(ls_id={id:1}, delta=455942956, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:55.757523] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.758288] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=17] gc stale ls task succ [2025-08-14 09:02:55.758402] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [762][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB90E3-0-0] [lt=37][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1001, svr="172.77.0.7:2882", version={val:1755161719775179002, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162175727658) [2025-08-14 09:02:55.758430] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [762][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB90E3-0-0] [lt=22][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1001, req={req_server:"172.77.0.7:2882", version:{val:1755161719775179002, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162175727658}, twrs={inited:true, tenant_id:1001, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162175756929}, server_version_for_stat:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162175756929}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:55.758647] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7edc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:54.797811] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] [thread pool stat](pool_stat={pool:0x564d6e8922c0, last_check_time:1755162174797810, last_idle_time:102477764956, last_thread_count:12}, *pool={name:TimerWK, this:0x564d6e8922c0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:10, threads_idle_time:102477764956, tenant_id:500}, qsize=8, current_idle_time=102477764956) [2025-08-14 09:02:55.758676] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ebe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.758690] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.758687] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=959420] [thread pool stat](pool_stat={pool:0x7f99667cbcf0, last_check_time:1755162175758686, last_idle_time:34784979205, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x7f99667cbcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:34784979205, tenant_id:500}, qsize=-1, current_idle_time=34784979205) [2025-08-14 09:02:55.758735] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=46] [thread pool stat](pool_stat={pool:0x7f99667ccaf0, last_check_time:1755162175758735, last_idle_time:26559336914, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x7f99667ccaf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:26559336914, tenant_id:500}, qsize=-1, current_idle_time=26559336914) [2025-08-14 09:02:55.758747] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] [thread pool stat](pool_stat={pool:0x7f9969de56b0, last_check_time:1755162175758747, last_idle_time:102938681542, last_thread_count:17}, *pool={name:TimerWK, this:0x7f9969de56b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:16, threads_idle_time:102938681542, tenant_id:1}, qsize=9, current_idle_time=102938681542) [2025-08-14 09:02:55.758770] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=22] [thread pool stat](pool_stat={pool:0x7f99356b0e70, last_check_time:1755162175758770, last_idle_time:102366343177, last_thread_count:15}, *pool={name:TimerWK, this:0x7f99356b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:12, threads_idle_time:102366343177, tenant_id:1001}, qsize=10, current_idle_time=102366343177) [2025-08-14 09:02:55.758781] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] [thread pool stat](pool_stat={pool:0x7f99230b0e70, last_check_time:1755162175758781, last_idle_time:102534969158, last_thread_count:17}, *pool={name:TimerWK, this:0x7f99230b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:16, threads_idle_time:102534969158, tenant_id:1002}, qsize=15, current_idle_time=102534969158) [2025-08-14 09:02:55.758870] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=425976942, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:55.758886] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.759852] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=430990311, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:55.759874] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.760271] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=15] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:55.760549] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=455942956, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:55.749857] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:746) [14963][T1_TimerWK9_PsCacheEvict][T1][Y0-0000000000000000-0-0] [lt=17] ps cache evict(tenant_id=1, stmt_id_map_.size()=9, stmt_info_map_.size()=9, op.get_used_size()=21249, get_mem_high()=48318300, expired_stmt_ids.count()=0, closed_stmt_ids.count()=9) [2025-08-14 09:02:55.760565] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.760984] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:55.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.761008] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:55.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.762538] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:55.764271] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=16] gc stale ls task succ [2025-08-14 09:02:55.764521] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=14] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:55.764582] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=23] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:55.764775] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4015] clock out of order(cur_ts=1755162175764774, cur_ts_=1755162175718135, delta=46639) [2025-08-14 09:02:55.765714] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=19] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:55.765742] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=15] current monitor number(seq_=-1) [2025-08-14 09:02:55.762578] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:55.765817] WDIAG [RPC.OBMYSQL] async_write_data (ob_sql_sock_session.cpp:162) [837][T1002_MysqlQueu][T1002][Y0-00063C4963099C4A-0-0] [lt=79][errcode=-4009] sock has error(ret=-4009) [2025-08-14 09:02:55.762317] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=20] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:55.767296] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=4966] temporary file module use no blocks [2025-08-14 09:02:55.767306] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:55.767319] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:55.769964] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.750, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.770000] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.770014] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728d1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7291110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.767335] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623077200323, flush_io_finished_ret_:0, flush_io_finished_round_:26309, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:55.764750] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [863][T1002_ArcSrv][T1001][YB42AC4D0007-00063C495E699C16-0-0] [lt=24][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:55.768675] WDIAG [SERVER] flush_buffer (obmp_packet_sender.cpp:1070) [837][T1002_MysqlQueu][T1002][Y0-00063C4963099C4A-0-0] [lt=12][errcode=-4009] write response fail(ret=-4009) [2025-08-14 09:02:55.770991] WDIAG [SERVER] response (obmp_base.cpp:61) [837][T1002_MysqlQueu][T1002][Y0-00063C4963099C4A-0-0] [lt=2309][errcode=-4009] failed to flush_buffer(ret=-4009) [2025-08-14 09:02:55.771002] WDIAG [RPC.FRAME] run (ob_sql_processor.cpp:46) [837][T1002_MysqlQueu][T1002][Y0-00063C4963099C4A-0-0] [lt=11][errcode=-4009] response rpc result fail(ret=-4009, tmp_ret=-4009) [2025-08-14 09:02:55.770305] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [14963][T1_TimerWK9][T1][Y0-0000000000000000-0-0] [lt=10695][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f994e1a6878, timer:0x7f98de5b0cd0, task:0x7f994b3b7c40, task_type:"N9oceanbase3sql24ObPsCacheEliminationTaskE", scheduled_time:1755162051761347, pushed_time:1755162051761672, delay:5000000}, delay_in_thread_pool=122996525, threashold=500000) [2025-08-14 09:02:55.771331] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [863][T1002_ArcSrv][T1001][YB42AC4D0007-00063C495E699C16-0-0] [lt=5836][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:55.772643] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=1014289] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:55.772664] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=22] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:55.772663] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [863][T1002_ArcSrv][T1001][YB42AC4D0007-00063C495E699C16-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:55.772677] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:55.772687] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:55.772696] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:55.772707] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:55.772716] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:55.772725] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:55.772733] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:55.772743] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:02:55.772753] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:02:55.772778] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=37] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:55.772795] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] temporary file module use no blocks [2025-08-14 09:02:55.772804] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:55.772714] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:2005) [14963][T1_TimerWK9_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=765] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-08-14 09:02:55.772817] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:55.772832] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623077211397, flush_io_finished_ret_:0, flush_io_finished_round_:26343, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:55.772863] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [726][T1001_ObXAHbWor][T1001][Y0-0000000000000000-0-0] [lt=942529] XA scheduler heartbeat task statistics(avg_time=1) [2025-08-14 09:02:55.771013] WDIAG [RPC.FRAME] handlePacketQueue (ob_req_qhandler.cpp:79) [837][T1002_MysqlQueu][T1002][Y0-00063C4963099C4A-0-0] [lt=9][errcode=-4009] process rpc fail(ret=-4009) [2025-08-14 09:02:55.773432] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [863][T1002_ArcSrv][T1001][YB42AC4D0007-00063C495E699C16-0-0] [lt=17][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:55.774654] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:55.776738] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [863][T1002_ArcSrv][T1001][YB42AC4D0007-00063C495E699C16-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:55.778325] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:55.779290] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [863][T1002_ArcSrv][T1001][YB42AC4D0007-00063C495E699C16-0-0] [lt=20][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:55.770358] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:55.782502] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:02:55.782705] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aa990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.782742] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.782757] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919afe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.782770] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d62b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.782783] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.782796] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.782820] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.782833] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.782846] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7397a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.790304] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [863][T1002_ArcSrv][T1001][YB42AC4D0007-00063C495E699C16-0-0] [lt=22][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:55.794205] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [863][T1002_ArcSrv][T1001][YB42AC4D0007-00063C495E699C16-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:55.794346] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.794385] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.794399] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.795245] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [863][T1002_ArcSrv][T1001][YB42AC4D0007-00063C495E699C16-0-0] [lt=21][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:55.800268] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [863][T1002_ArcSrv][T1001][YB42AC4D0007-00063C495E699C16-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:55.804437] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.805405] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=32919] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25897967616, fast_delete_log_mode=false) [2025-08-14 09:02:55.808851] INFO handle (memory_dump.cpp:671) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=15] statistics: tenant_cnt: 6, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 168 270 0 1 2 110 1 0 1 3 7 0 0 1 4 1 1 0 1 5 86 6 0 1 6 13 0 0 1 7 22 2 0 1 8 53 1 0 1 12 17 3 0 1 13 10 1 0 1 14 1 1 0 1 15 1 1 0 1 16 7 8 0 1 18 4 3 0 1 20 2 1 0 1 22 1 1 0 1 24 1 0 0 1 25 6 0 0 500 0 557 276 0 500 7 193 5 0 500 8 41 1 0 500 9 2 1 0 500 10 7 2 0 500 16 56 4 0 500 17 56 7 0 500 22 38 64 0 500 23 27 10 0 508 0 2 7 0 508 8 5 1 0 509 0 2 7 0 509 8 5 1 0 1001 0 147 253 0 1001 2 26 1 0 1001 3 5 0 0 1001 4 1 1 0 1001 5 85 6 0 1001 6 11 0 0 1001 7 30 2 0 1001 8 36 1 0 1001 12 17 3 0 1001 13 3 1 0 1001 14 1 1 0 1001 16 6 3 0 1001 18 4 3 0 1001 20 4 1 0 1001 25 8 0 0 1002 0 261 278 0 1002 2 198 1 0 1002 3 4 0 0 1002 4 1 1 0 1002 5 101 6 0 1002 6 14 0 0 1002 7 6 2 0 1002 8 73 1 0 1002 12 18 3 0 1002 13 7 1 0 1002 14 1 2 0 1002 16 9 8 0 1002 18 9 3 0 1002 20 2 1 0 1002 21 1 0 0 1002 24 1 0 0 1002 25 7 0 0 cost_time: 67325 [2025-08-14 09:02:55.808887] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=15] [MEMORY] tenant: 1, limit: 2147483648 hold: 1013174272 cache_hold: 56623104 cache_used: 56623104 cache_item_count: 27 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 444993536 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 230686720 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 1163264 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 483328 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 53714944 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 1458176 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 14204928 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 111149056 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35651584 limit= 429496720 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 8003584 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= PS_CACHE_CTX_ID hold_bytes= 65536 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 12730368 limit= 9223372036854775807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 3481600 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3964928 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 212992 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 884736 limit= 9223372036854775807 [2025-08-14 09:02:55.814510] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.802, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.815901] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [863][T1002_ArcSrv][T1001][YB42AC4D0007-00063C495E699C16-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162175747240]}) [2025-08-14 09:02:55.821624] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:55.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.824572] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.814, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.824622] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.817, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.824637] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.824650] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.819, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.824664] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.813, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.829057] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=8088][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:55.834636] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919831b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.828, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.836328] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:55.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.836440] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=13] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:02:55.837374] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=628097791, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:55.837402] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.844696] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.835, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.854769] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.847, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.854802] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.844, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.854813] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:55.844, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:55.858913] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=426117605, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:55.858936] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.859958] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=456084775, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=252303, succ_cluster_heartbeat_count=1904, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161003593738002, v:0}, local_cluster_delta=1172266216, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:02:55.860592] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=40] get wrs ts(ls_id={id:1}, delta=456084691, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:55.860620] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.881218] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c0b90, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571c0e70, time_wheel:0x7f9959fef990, thread_pool:0x7f9959fef170, is_running:1, time_interval:3.00s, expected_run_time:09:02:55.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.891529] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EEC7-0-0] [lt=16] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-08-14 09:02:55.921648] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:55.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.929088] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:55.936391] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:55.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:55.937463] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=628198470, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:55.937489] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.940013] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:02:55.940049] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=25] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:02:55.960573] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=456185011, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:55.960598] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.990495] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=26] PNIO [ratelimit] time: 1755162175990492, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007606, add_bytes: 0 [2025-08-14 09:02:55.992452] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:55.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.014199] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:56.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.014248] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:56.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.016128] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [863][T1002_ArcSrv][T1001][YB42AC4D0007-00063C495E699C16-0-0] [lt=22] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162175747240]}) [2025-08-14 09:02:56.021715] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:55.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.029147] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:56.036454] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:55.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.036960] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161101770923, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755161572801635) [2025-08-14 09:02:56.037617] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] get wrs ts(ls_id={id:1}, delta=628297446, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:56.037644] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:56.060686] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=456284914, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:56.060711] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:56.094581] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:56.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.121778] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:56.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.129207] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:56.136516] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:56.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.137160] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B92B1-0-0] [lt=22][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162176137020) [2025-08-14 09:02:56.137184] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B92B1-0-0] [lt=23][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162176137020}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162176036983}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162176036983}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:56.137306] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [316][pnio1][T1][YB42AC4D0007-00063C49607B92B1-0-0] [lt=13][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1, delta=1102964, last_succ_cluster_heartbeat_tstamp_=1755162175034341) [2025-08-14 09:02:56.137632] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=628397608, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:56.137650] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:56.138273] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=628397608, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:56.138289] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:56.160690] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=27] get wrs ts(ls_id={id:1}, delta=456385304, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:56.160715] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:56.215243] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:56.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.221839] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:56.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.229283] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:56.236550] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:56.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.237181] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:56.237222] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:56.237257] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:56.237276] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:56.237288] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:56.237313] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:56.237331] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:56.237346] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:56.237358] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:56.237371] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:56.237551] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:56.237565] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:56.238391] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] get wrs ts(ls_id={id:1}, delta=628499204, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:56.238414] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:56.238863] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:56.238886] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=22] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3982) [2025-08-14 09:02:56.238935] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=48][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:56.238950] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:56.238959] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=10] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3983) [2025-08-14 09:02:56.238966] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:56.239161] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:56.239173] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=12] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3984) [2025-08-14 09:02:56.239181] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:56.243779] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=64][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:56.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.245760] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:02:56.245826] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:02:56.245882] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:02:56.245925] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:02:56.245949] INFO [COMMON] print_io_status (ob_io_manager.cpp:2639) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4091, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:490943, total:490948) [2025-08-14 09:02:56.245973] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, param_config_:{memory_limit:536870912, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=865568, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:02:56.246025] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2826) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=46] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4091, [delay/us]: prepare:0, schedule:0, submit:600000000, device:490943, total:490948) [2025-08-14 09:02:56.246051] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:02:56.246070] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:56.246084] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:56.246096] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SERVER] net_in=1kB/s, net_out=1kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=1280000kB/s [2025-08-14 09:02:56.247696] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:56.247719] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162176247692], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:56.247735] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162176247692], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:56.247750] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162176247692]}) [2025-08-14 09:02:56.247757] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:56.247765] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:56.247773] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=5] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:56.247783] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162176247772], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:56.247995] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:56.248018] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=16] get number(ret=-4038, service_type_=0, range=1, base_id=1755162176247196000, start_id=0, end_id=140295968341544) [2025-08-14 09:02:56.248038] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:56.248052] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162176247991], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:56.248069] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162176247991]}) [2025-08-14 09:02:56.253853] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:56.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.260821] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=456484583, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:56.260847] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:56.265509] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:56.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.265553] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:56.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.321900] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:56.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.329344] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:56.336592] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:56.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.338503] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=628599457, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:56.338528] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:56.360858] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] get wrs ts(ls_id={id:1}, delta=456584917, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:56.360881] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:56.421959] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:56.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.429392] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:56.436664] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:56.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.438556] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28] get wrs ts(ls_id={id:1}, delta=628699849, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:56.438589] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:56.460947] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=456684281, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:56.460970] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:56.495170] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:56.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.516854] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:56.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.516887] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:56.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.522022] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:56.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.529439] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:56.536733] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:56.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.538134] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B92B5-0-0] [lt=34][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162176538006) [2025-08-14 09:02:56.538161] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B92B5-0-0] [lt=25][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162176538006}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162176437963}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162176437963}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:56.538635] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=628799455, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:56.538662] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:56.539557] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29] get wrs ts(ls_id={id:1}, delta=628800520, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:56.539582] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:56.561020] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=456784944, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:56.561052] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=30] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:56.579874] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=14841][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fe030, caller:ob_table_lock_service.cpp:start:276, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE6$_1273, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe310, time_wheel:0x7f99535f3990, thread_pool:0x7f99535f3170, is_running:1, time_interval:10.00s, expected_run_time:09:02:56.426, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.617008] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [863][T1002_ArcSrv][T1001][YB42AC4D0007-00063C495E699C16-0-0] [lt=24] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162176247772]}) [2025-08-14 09:02:56.621727] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ebbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:56.615, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:56.622060] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:56.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.629497] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:56.636783] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:56.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.636792] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:56.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.639316] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=628900441, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:56.639339] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:56.716183] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [863][T1002_ArcSrv][T1001][YB42AC4D0007-00063C495E699C16-0-0] [lt=34] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:56.597275] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:56.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.719623] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=122145][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:56.597|207=122.33ms|210=0us|222=1us|total=122.37ms) [2025-08-14 09:02:56.719651] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4389] timer task use too much time(end=1755162176719650, start=1755162176597272, delta=122378) [2025-08-14 09:02:56.719663] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:56.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.720671] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=456943819, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:56.721274] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:56.721329] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:56.722091] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:56.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.720694] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:56.716206] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [863][T1002_ArcSrv][T1001][YB42AC4D0007-00063C495E699C16-0-0] [lt=22][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:56.726432] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [863][T1002_ArcSrv][T1001][YB42AC4D0007-00063C495E699C16-0-0] [lt=10209][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:02:56.726447] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [863][T1002_ArcSrv][T1001][YB42AC4D0007-00063C495E699C16-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162176247772]}) [2025-08-14 09:02:56.726459] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1507) [14963][T1_TimerWK9_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=118] start lib cache evict(tenant_id=1, mem_hold=53714944, mem_limit=107374180, cache_obj_num=308, cache_node_num=308) [2025-08-14 09:02:56.726482] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1524) [14963][T1_TimerWK9_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=20] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=53714944, mem_limit=107374180, cache_obj_num=308, cache_node_num=308) [2025-08-14 09:02:56.726499] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2792) [14963][T1_TimerWK9_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=12] schedule next cache evict task(evict_interval=5000000) [2025-08-14 09:02:55.751175] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=31] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136826952869001, v:0}, clog_base_lsn:{lsn:5099962368}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1755071646732636000, v:0}, all_id_meta:{id_meta:[{limited_id:1755161555795565000, latest_log_ts:{val:1755161545796642001, v:0}}, {limited_id:22000001, latest_log_ts:{val:1755135645787806003, v:0}}, {limited_id:60000001, latest_log_ts:{val:1755135646400653001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:50, palf_env_:0x7f9950bf4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-08-14 09:02:56.729198] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:56.729509] INFO [STORAGE] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:123) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] wead read timestamp is too old(timestamp={val:1755161749741193000, v:0}, gts_scn={val:1755161762721798002, v:0}, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1755161749741193000, v:0}}) [2025-08-14 09:02:56.728257] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1832) [863][T1002_ArcSrv][T1001][YB42AC4D0007-00063C495E699C16-0-0] [lt=10][errcode=-4012] get gts fail(ret=-4012, timeout_us=1000000, request_time=[mts=1755162175750470]) [2025-08-14 09:02:56.731447] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:56.730033] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1844) [863][T1002_ArcSrv][T1001][YB42AC4D0007-00063C495E699C16-0-0] [lt=1772][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1755162204672358, request_time=[mts=1755162175750470]) [2025-08-14 09:02:56.731854] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:662) [863][T1002_ArcSrv][T1001][YB42AC4D0007-00063C495E699C16-0-0] [lt=1816][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f98f6439360, tx_id:{txid:0}, state:1, addr:"172.77.0.7:2882", tenant_id:1001, session_id:3221590976, assoc_session_id:3221590976, client_sid:3221590976, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1755162175718135, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180067074, seq_base:1755162175425703, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}) [2025-08-14 09:02:56.732200] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=29] get wrs ts(ls_id={id:1001}, delta=432004010, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:56.732221] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:55.750489] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=9293] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136829134139001, v:0}, clog_base_lsn:{lsn:8186781696}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1753840867372367034, v:0}, all_id_meta:{id_meta:[{limited_id:1755161729775179000, latest_log_ts:{val:1755161719775179001, v:0}}, {limited_id:18000001, latest_log_ts:{val:1755157440846556001, v:0}}, {limited_id:60000001, latest_log_ts:{val:1755135657266496001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:46, palf_env_:0x7f99321fa030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-08-14 09:02:56.732305] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:01:57.725568] EDIAG [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [411][sql_nio4][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] LOGGER COST TOO MUCH TIME, cost: 58924126, owner: ObLog, click_count: 5, time dist:[FORMAT_END=6385, ObMalloc#start=1, ObMalloc#end=27917, ALLOC_END=58889822, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x98911bb 0x9890e1a 0x98905d9 0x27a595c5 0x94da7e4 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:56.732296] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25] get wrs ts(ls_id={id:1}, delta=426990641, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:56.732324] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:56.732998] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1001}, delta=432004010, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:56.733012] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:56.737750] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:56.737783] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:56.737802] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:56.737816] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:56.737824] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:56.737838] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:56.737851] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:56.737867] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:56.737874] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:56.737884] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:56.737896] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:02:56.738067] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:56.738077] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:56.733111] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919abd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:56.721, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:56.738749] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=5613][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919abb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:56.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:56.738780] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:56.726, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:56.738792] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fdc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:56.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:56.738801] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:56.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:01:57.725568] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [411][sql_nio4][T0][Y0-0000000000000000-0-0] [lt=32] sm conn init succ(conn.sessid_=3221590965, sess.client_addr_="172.77.0.1:48076") [2025-08-14 09:02:56.738859] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1090) [411][sql_nio4][T0][Y0-0000000000000000-0-0] [lt=58924126] accept one succ(*s={this:0x7f98f72bc030, session_id:3221590965, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:280}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:56.738940] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1008) [411][sql_nio4][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x7f98f72bc030, session_id:3221590965, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:280}, err:5, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:56.738968] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [411][sql_nio4][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221590965, proxy_sessid=0, ret=0) [2025-08-14 09:02:56.738976] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:983) [411][sql_nio4][T0][Y0-0000000000000000-0-0] [lt=7] sock ref clean, do destroy(*s={this:0x7f98f72bc030, session_id:3221590965, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:280}, err:5, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:02:56.741154] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=975973] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:56.741195] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=32] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:55.745564] WDIAG [STORAGE] prepare_read_tables (ob_multiple_merge.cpp:1604) [14945][T1_TimerWK8_MergeLoop][T1][YB42AC4D0007-00063C4978199C2C-0-0] [lt=1957081][errcode=-4012] fail to prepare mds tables(ret=-4012, refresh=false, get_table_param=0x7f97efaced40, access_param={iter_param:{table_id:88888, tablet_id:{id:50345}, ls_id:{id:1}, cg_idx:0, read_info:{schema_column_count:4, schema_rowkey_cnt:2, rowkey_cnt:4, trans_col_index:-1, mview_old_new_col_index:-1, group_idx_col_index:-1, seq_read_column_count:2, max_col_index:5, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 4, 5], data:0x7f98c5026740, count:4, allocator:0x7f97efad31e8, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 3], data:0x7f98c5026760, count:4, allocator:0x7f97efad31e8, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=16 {type:"TINYINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"CHAR", collation:"binary", coercibility:"INVALID"} order=0, column_id=18 {type:"CHAR", collation:"binary", coercibility:"INVALID"} order=0, column_id=19 {type:"CHAR", collation:"binary", coercibility:"INVALID"} order=0], data:0x7f98c5026700, count:4, allocator:0x7f97efad31e8, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:4, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}], data:0x7f98c50267b0, count:4, allocator:0x7f97efad31e8, capacity:4, init_cnt:4}, has_all_column_group:true, need_truncate_filter:false}, rowkey_read_info:{is_inited:true, compat_version:2, is_oracle_mode:true, is_cs_replica_compat:0, is_delete_insert_table:0, schema_column_count:4, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:6}, cols_desc:{ObIArray:[cnt:4, column_id=16 {type:"TINYINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"CHAR", collation:"binary", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9956fc6150, count:4, allocator:0x564d6ef37d68, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:true, rowkey_cnt:4, is_inited:true, is_oracle_mode:true}, memtable_cols_index:{rowkey_mode:true, for_memtable:true, schema_rowkey_cnt:2, column_cnt:6}}, out_cols_project:[0, 1, 2, 3], agg_cols_project:[], group_by_cols_project:[], pushdown_filter:NULL, op:null, sstable_index_filter:null, cg_read_infos:null, output_exprs:NULL, aggregate_exprs:NULL, output_sel_mask:[true, true, true, true], is_multi_version_minor_merge:false, need_scn:false, is_same_schema_column:true, pd_storage_flag:{pd_flag:0}, vectorized_enabled:false, has_virtual_columns:false, has_lob_column_out:false, is_for_foreign_check:false, limit_prefetch:true, is_mds_query:true, is_non_unique_local_index:false, ss_rowkey_prefix_cnt:0, table_scan_opt:{io_read_batch_size:0, io_read_gap_size:0, storage_rowsets_size:1}, auto_split_filter_type:18446744073709551615, auto_split_filter:null, auto_split_params:NULL, is_tablet_spliting:false, is_column_replica_table:false, is_delete_insert:false, need_update_tablet_param:0x7f97efad2fc8}, padding_cols:[], projector_size:0, output_exprs:NULL, op_filters:null, row2exprs_projector:null, output_sel_mask:[true, true, true, true], is_inited:true}) [2025-08-14 09:02:56.741465] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=980309] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:56.741498] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=20] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:56.741632] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:56.741650] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=18] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3985) [2025-08-14 09:02:56.741659] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:56.741673] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:56.741681] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3986) [2025-08-14 09:02:56.741688] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:01:55.791151] EDIAG [RS.COMPACTION] try_reload (ob_zone_merge_manager.cpp:134) [1060][T1002_FrzInfoDe][T1002][YB42AC4D0007-00063C496239A612-0-0] [lt=39958][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2936680, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2936674, ObMalloc#start=1, ObMalloc#end=4, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x11298961 0x11298612 0x112564aa 0x1126faa7 0x9a2203f 0x112b4140 0x1128c31e 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:56.741920] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=6] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:56.741936] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=14] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3987) [2025-08-14 09:02:56.741964] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:56.742327] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:56.730724] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2510) [14963][T1_TimerWK9_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=10] Dumping All Cache Objs(alloc_obj_list.count()=309, alloc_obj_list=[{obj_id:117, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:81511}, {obj_id:134, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48856}, {obj_id:139, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62914}, {obj_id:140, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58459}, {obj_id:147, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:150, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:154, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:160, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:45168}, {obj_id:181, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73460}, {obj_id:230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47192}, {obj_id:239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51274}, {obj_id:246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51414}, {obj_id:247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:52851}, {obj_id:260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52686}, {obj_id:263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46488}, {obj_id:277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105415}, {obj_id:294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71241}, {obj_id:1094, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:77398}, {obj_id:6297, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:71304}, {obj_id:12127, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47024}, {obj_id:12134, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47361}, {obj_id:12144, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72485}, {obj_id:12146, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:61373}, {obj_id:12147, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48997}, {obj_id:12162, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48972}, {obj_id:12165, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60282}, {obj_id:12169, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:79471}, {obj_id:12171, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:89656}, {obj_id:12173, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63527}, {obj_id:12174, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:12175, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48067}, {obj_id:12178, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120160}, {obj_id:12179, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97242}, {obj_id:12182, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:12186, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:61908}, {obj_id:12190, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72480}, {obj_id:12200, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54975}, {obj_id:12201, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66113}, {obj_id:12202, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:12203, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70267}, {obj_id:12207, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60418}, {obj_id:12209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55488}, {obj_id:12211, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59010}, {obj_id:12248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69220}, {obj_id:12249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63913}, {obj_id:12281, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:76397}, {obj_id:12282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54529}, {obj_id:12298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:96140}, {obj_id:12872, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:12890, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:107641}, {obj_id:15138, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47368}, {obj_id:15139, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48232}, {obj_id:15140, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62071}, {obj_id:15141, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:61352}, {obj_id:18083, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:72597}, {obj_id:18085, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:108972}, {obj_id:18086, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55046}, {obj_id:18087, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50527}, {obj_id:18151, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53935}, {obj_id:18204, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52360}, {obj_id:18206, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:108557}, {obj_id:18277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:51627}, {obj_id:18279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69286}, {obj_id:18548, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48728}, {obj_id:18549, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48728}, {obj_id:18550, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:18731, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47936}, {obj_id:18826, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:112576}, {obj_id:18838, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:112169}, {obj_id:20964, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62496}, {obj_id:20985, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93885}, {obj_id:23922, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47190}, {obj_id:23923, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50804}, {obj_id:23924, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47364}, {obj_id:23925, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49648}, {obj_id:23927, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:77887}, {obj_id:23928, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57476}, {obj_id:23931, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:23932, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:82816}, {obj_id:23933, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71684}, {obj_id:23935, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62823}, {obj_id:23938, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:23939, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52125}, {obj_id:23941, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:23947, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57387}, {obj_id:23966, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46856}, {obj_id:23999, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:24029, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73936}, {obj_id:24030, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:94524}, {obj_id:24031, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53268}, {obj_id:24036, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72111}, {obj_id:24037, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47781}, {obj_id:24086, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:91789}, {obj_id:24394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:24395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:24550, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60153}, {obj_id:24551, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:24555, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:24563, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55892}, {obj_id:24564, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53476}, {obj_id:24567, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50408}, {obj_id:24571, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:139753}, {obj_id:24575, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46437}, {obj_id:24576, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47944}, {obj_id:24577, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:150419}, {obj_id:24581, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72517}, {obj_id:24582, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:24583, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:24584, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:107236}, {obj_id:24586, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:81404}, {obj_id:24594, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49590}, {obj_id:24595, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175390}, {obj_id:24596, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:131093}, {obj_id:24622, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:110353}, {obj_id:24638, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:76516}, {obj_id:24639, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242094}, {obj_id:24640, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78318}, {obj_id:24643, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:128842}, {obj_id:24644, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:146442}, {obj_id:24717, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60153}, {obj_id:24718, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:112576}, {obj_id:28061, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:109447}, {obj_id:29779, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46475}, {obj_id:29781, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46848}, {obj_id:29782, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47408}, {obj_id:29801, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93450}, {obj_id:29802, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93085}, {obj_id:29814, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49718}, {obj_id:30383, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:13745}, {obj_id:30386, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1244037}, {obj_id:30387, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53353}, {obj_id:30391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:123151}, {obj_id:30392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51899}, {obj_id:30399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226857}, {obj_id:30400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2690007}, {obj_id:30401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:978411}, {obj_id:30402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:86847}, {obj_id:30403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213002}, {obj_id:30404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:168384}, {obj_id:30406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:30411, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:87421}, {obj_id:30412, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:87693}, {obj_id:30413, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:295630}, {obj_id:30417, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72880}, {obj_id:30418, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:119440}, {obj_id:30419, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225297}, {obj_id:30426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:80919}, {obj_id:30429, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:81927}, {obj_id:30430, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:267979}, {obj_id:30434, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:147323}, {obj_id:30435, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179975}, {obj_id:30436, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:588368}, {obj_id:30440, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:80150}, {obj_id:30441, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:84367}, {obj_id:30442, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:260176}, {obj_id:30447, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:77611}, {obj_id:30452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67402}, {obj_id:30453, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:98189}, {obj_id:30454, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198516}, {obj_id:30583, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52313}, {obj_id:31363, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:94489}, {obj_id:31617, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73758}, {obj_id:31618, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:79724}, {obj_id:31620, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67451}, {obj_id:31842, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:137689}, {obj_id:31843, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:172995}, {obj_id:31845, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93748}, {obj_id:31847, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:81787}, {obj_id:31848, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62940}, {obj_id:31849, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59055}, {obj_id:31850, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71483}, {obj_id:31851, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65300}, {obj_id:31852, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51380}, {obj_id:32277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:255867}, {obj_id:32278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:117726}, {obj_id:33260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:68046}, {obj_id:33261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60199}, {obj_id:33264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:95387}, {obj_id:33271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65339}, {obj_id:33274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62187}, {obj_id:33277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188759}, {obj_id:33280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64127}, {obj_id:33282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78179}, {obj_id:33286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208811}, {obj_id:33287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63755}, {obj_id:33292, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:61007}, {obj_id:33295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62979}, {obj_id:34025, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:155792}, {obj_id:34668, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:129620}, {obj_id:34669, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65224}, {obj_id:34689, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196763}, {obj_id:34700, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:111215}, {obj_id:34781, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:81518}, {obj_id:34782, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:82696}, {obj_id:34783, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:181694}, {obj_id:34784, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:35496, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:68867}, {obj_id:35499, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67383}, {obj_id:36119, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60473}, {obj_id:36121, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:37269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49718}, {obj_id:37490, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:317182}, {obj_id:37493, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:75507}, {obj_id:37494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62272}, {obj_id:37496, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:75208}, {obj_id:37498, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55868}, {obj_id:37500, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65795}, {obj_id:37505, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70558}, {obj_id:37507, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70370}, {obj_id:37509, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60427}, {obj_id:37511, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60473}, {obj_id:37513, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:37738, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57000}, {obj_id:37744, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:77594}, {obj_id:37828, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:37829, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:92320}, {obj_id:37830, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:95620}, {obj_id:37831, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:37832, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73385}, {obj_id:37866, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:37867, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52842}, {obj_id:37874, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:37875, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:92646}, {obj_id:37876, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:37878, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:61911}, {obj_id:37879, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56852}, {obj_id:37881, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70942}, {obj_id:37892, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53087}, {obj_id:37932, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46840}, {obj_id:38487, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52594}, {obj_id:38578, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:165858}, {obj_id:38581, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72251}, {obj_id:39637, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:369344}, {obj_id:39639, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:75507}, {obj_id:39640, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62272}, {obj_id:39643, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:75208}, {obj_id:39645, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55868}, {obj_id:39647, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65795}, {obj_id:39651, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70558}, {obj_id:39653, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70370}, {obj_id:39655, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60427}, {obj_id:39657, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60473}, {obj_id:39660, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:39796, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:39848, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59948}, {obj_id:40078, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71304}, {obj_id:40094, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46832}, {obj_id:40103, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50178}, {obj_id:40104, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64672}, {obj_id:40616, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:101964}, {obj_id:40617, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174736}, {obj_id:41446, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97938}, {obj_id:41449, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:357307}, {obj_id:41451, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:139689}, {obj_id:41452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69437}, {obj_id:41455, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:41456, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:170962}, {obj_id:41458, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:75208}, {obj_id:41461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55868}, {obj_id:41463, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65795}, {obj_id:41466, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70704}, {obj_id:41467, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70558}, {obj_id:41469, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70370}, {obj_id:41471, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60427}, {obj_id:41473, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60473}, {obj_id:41475, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66883}, {obj_id:41476, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:111696}, {obj_id:41482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55916}, {obj_id:41483, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48752}, {obj_id:41484, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:94689}, {obj_id:41485, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:962398}, {obj_id:41486, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:332316}, {obj_id:41487, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:41488, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120352}, {obj_id:41489, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64868}, {obj_id:41490, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55535}, {obj_id:41491, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:65439}, {obj_id:41492, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:113961}, {obj_id:41495, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:4, added_to_lc:true, mem_used:47392}, {obj_id:41496, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47424}, {obj_id:41497, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47472}, {obj_id:41498, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47672}, {obj_id:41503, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54843}, {obj_id:41504, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:81974}, {obj_id:41505, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60826}, {obj_id:41506, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64096}, {obj_id:41507, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66959}, {obj_id:41514, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46312}, {obj_id:41522, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48416}, {obj_id:41523, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:109427}, {obj_id:41525, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:110786}, {obj_id:42033, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:42208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:159832}, {obj_id:42209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70908}, {obj_id:42251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58088}, {obj_id:42252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:88656}, {obj_id:42253, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:121604}, {obj_id:42254, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63204}, {obj_id:42575, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48560}, {obj_id:42577, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50052}, {obj_id:42578, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65991}, {obj_id:42581, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:94075}, {obj_id:42583, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47392}, {obj_id:42591, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:109438}, {obj_id:42593, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:42596, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:42597, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:94243}, {obj_id:42599, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:42605, tenant_id:1, log_del_time:43622867181517, real_del_time:9223372036854775807, ref_count:1, added_to_lc:false, mem_used:25816}]) [2025-08-14 09:02:56.742370] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2800) [14963][T1_TimerWK9_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=11624] schedule next cache evict task(evict_interval=5000000) [2025-08-14 09:02:56.742380] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [14963][T1_TimerWK9][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f994e1a4118, timer:0x7f996713de90, task:0x7f994b3f0540, task_type:"N9oceanbase3sql26ObPlanCacheEliminationTaskE", scheduled_time:1755162053755795, pushed_time:1755162053756290, delay:5000000}, delay_in_thread_pool=122014790, threashold=500000) [2025-08-14 09:02:56.742732] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:943) [863][T1002_ArcSrv][T1001][YB42AC4D0007-00063C495E699C16-0-0] [lt=53][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f98e4ebf338, id:3221590976, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1001, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f98f6439360}) [2025-08-14 09:02:56.742770] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:647) [863][T1002_ArcSrv][T1001][YB42AC4D0007-00063C495E699C16-0-0] [lt=35][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs, exec_ctx)(ret=-4283, session_id=3221590976, *tx_desc={this:0x7f98f6439360, tx_id:{txid:0}, state:1, addr:"172.77.0.7:2882", tenant_id:1001, session_id:3221590976, assoc_session_id:3221590976, client_sid:3221590976, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1755162175718135, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180067074, seq_base:1755162175425703, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}) [2025-08-14 09:02:56.744064] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] get wrs ts(ls_id={id:1}, delta=629004194, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:56.744107] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=40] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=2, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:56.744200] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 444993536 used= 389625696 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 70 washed_blocks= 287 washed_size= 70221824 [MEMORY] request_cached_chunk_cnt= 4 [MEMORY] hold= 389625696 used= 379379459 count= 32871 avg_used= 11541 mod=SUMMARY [2025-08-14 09:02:56.744249] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=16] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 230686720 used= 228884480 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 block_cnt= 110 chunk_cnt= 110 mod=Memstore [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:02:56.744266] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= EXECUTE_CTX_ID hold= 1163264 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 15 washed_size= 13516800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:56.744283] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:02:56.744298] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 53714944 used= 47487424 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 84 washed_blocks= 334 washed_size= 126640128 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 47487424 used= 43098388 count= 3994 avg_used= 10790 mod=SUMMARY [2025-08-14 09:02:56.744315] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= WORK_AREA hold= 1458176 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 13 washed_blocks= 66 washed_size= 25804800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:56.744330] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 14204928 used= 10970224 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 277 washed_size= 31932416 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10970224 used= 6078924 count= 18573 avg_used= 327 mod=SUMMARY [2025-08-14 09:02:56.744346] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 111149056 used= 108896256 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 block_cnt= 211 chunk_cnt= 53 mod=CoStack [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 mod=SUMMARY [2025-08-14 09:02:56.744365] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 56623104 used= 56623104 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 56623104 used= 0 count= 27 avg_used= 0 mod=SUMMARY [2025-08-14 09:02:56.744381] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 34365376 limit= 429496720 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 34365376 used= 31086608 count= 475 avg_used= 65445 mod=SUMMARY [2025-08-14 09:02:56.744396] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 8003584 used= 4947392 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 10 washed_blocks= 23 washed_size= 12967936 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4947392 used= 4793344 count= 604 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:56.744413] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:56.744441] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 65536 used= 31024 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 2031616 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 31024 used= 27058 count= 20 avg_used= 1352 mod=SUMMARY [2025-08-14 09:02:56.744456] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= RPC_CTX_ID hold= 12730368 used= 4153344 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4153344 used= 2738560 count= 173 avg_used= 15829 mod=SUMMARY [2025-08-14 09:02:56.744474] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 3481600 used= 2391232 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 10 washed_size= 4907008 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2391232 used= 2367040 count= 36 avg_used= 65751 mod=SUMMARY [2025-08-14 09:02:56.744484] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:56.744494] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 3964928 used= 3080192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 3 washed_size= 229376 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3080192 used= 2983936 count= 376 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:56.744512] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= SCHEMA_SERVICE hold= 212992 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1884160 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:00:24.758579] EDIAG [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:301) [273][TimerWK1_SqlMemTimer][T1001][Y0-0000000000000000-0-0] [lt=37307][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1953918, owner: ObLog, click_count: 5, time dist:[FORMAT_END=981047, ObMalloc#start=1, ObMalloc#end=972867, ALLOC_END=3, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0xfa06846 0xfa05f62 0xfa05c84 0xfa05a50 0x17a566c0 0x17a5a245 0x125a3021 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:56.744522] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:56.744530] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:56.744556] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=20] [MEMORY] tenant_id= 1 ctx_id= MERGE_NORMAL_CTX_ID hold= 884736 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 6 washed_blocks= 9 washed_size= 11698176 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:56.745845] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:56.747457] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=30] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:56.747484] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:56.748477] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:56.748531] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:56.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.748946] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:56.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.749020] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=19] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:56.749030] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162176749017], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:56.749042] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162176749017], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:56.749053] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162176749017]}) [2025-08-14 09:02:56.749062] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=7] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:56.749068] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162176749061], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:56.749273] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:56.749281] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:56.749291] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162176749270], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:56.749309] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162176749270]}) [2025-08-14 09:02:56.753392] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:56.726936] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [723][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=12] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=16, total_destructed_detector=16, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f99333562b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-08-14 09:02:56.753441] WDIAG [DETECT] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [723][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=26460][errcode=0] cost too much time:ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t), (this=|threshold=100.00ms|start at 09:02:56.753|total=953.85ms) [2025-08-14 09:02:56.757818] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:56.758032] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:221) [411][sql_nio4][T0][Y0-0000000000000000-0-0] [lt=10] mark session id unused(conn.sessid_=3221590965) [2025-08-14 09:02:56.759176] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:56.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.752251] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:985) [408][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=24] can close safely, do destroy(*s={this:0x7f98f72dc030, session_id:3221590970, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:211}, err:5, last_decode_time:1755162167725699, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:219, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-08-14 09:02:56.760677] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=16] gc stale ls task succ [2025-08-14 09:02:56.761098] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=5720] get wrs ts(ls_id={id:1}, delta=456985101, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:56.761114] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=2, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:56.762644] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=14] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:56.762694] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant: 500, limit: 9223372036854775807 hold: 698925056 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 330993664 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 35307520 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 85983232 limit= 9223372036854775807 [MEMORY] ctx_id= LIBEASY hold_bytes= 3145728 limit= 9223372036854775807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 25165824 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 3022848 limit= 9223372036854775807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 37396480 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 78925824 limit= 9223372036854775807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 98983936 limit= 9223372036854775807 [2025-08-14 09:02:56.762728] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 330993664 used= 301298992 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 495 washed_blocks= 2221 washed_size= 914604032 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 301298992 used= 293883141 count= 12662 avg_used= 23209 mod=SUMMARY [2025-08-14 09:02:56.762752] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 500 ctx_id= GLIBC hold= 35307520 used= 21963120 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 187 washed_blocks= 819 washed_size= 369442816 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 21963120 used= 10364917 count= 131698 avg_used= 78 mod=SUMMARY [2025-08-14 09:02:56.762773] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 500 ctx_id= CO_STACK hold= 85983232 used= 83091456 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 83091456 used= 82936896 count= 161 avg_used= 515136 block_cnt= 161 chunk_cnt= 41 mod=CoStack [MEMORY] hold= 83091456 used= 82936896 count= 161 avg_used= 515136 mod=SUMMARY [2025-08-14 09:02:56.762800] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=21] [MEMORY] tenant_id= 500 ctx_id= LIBEASY hold= 3145728 used= 3070816 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1048576 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3070816 used= 2859784 count= 223 avg_used= 12824 mod=SUMMARY [2025-08-14 09:02:56.762810] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 500 ctx_id= LOGGER_CTX_ID hold= 25165824 used= 14565376 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 14565376 used= 14558208 count= 7 avg_used= 2079744 mod=SUMMARY [2025-08-14 09:02:56.762830] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 500 ctx_id= RPC_CTX_ID hold= 3022848 used= 1458176 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 56 washed_blocks= 69 washed_size= 114417664 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1458176 used= 963072 count= 60 avg_used= 16051 mod=SUMMARY [2025-08-14 09:02:56.762841] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 500 ctx_id= PKT_NIO hold= 37396480 used= 35890688 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 46 washed_blocks= 67 washed_size= 90816512 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 35890688 used= 34936544 count= 120 avg_used= 291137 mod=SUMMARY [2025-08-14 09:02:56.762871] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 500 ctx_id= SCHEMA_SERVICE hold= 78925824 used= 68920960 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 22 washed_size= 5767168 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 68920960 used= 67658879 count= 4858 avg_used= 13927 mod=SUMMARY [2025-08-14 09:02:56.762881] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 500 ctx_id= UNEXPECTED_IN_500 hold= 98983936 used= 97224016 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 97224016 used= 86822400 count= 3017 avg_used= 28777 mod=SUMMARY [2025-08-14 09:02:56.762893] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant: 508, limit: 1073741824 hold: 15613952 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 5128192 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10485760 limit= 9223372036854775807 [2025-08-14 09:02:56.762900] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 508 ctx_id= DEFAULT_CTX_ID hold= 5128192 used= 3765328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3765328 used= 3723912 count= 84 avg_used= 44332 mod=SUMMARY [2025-08-14 09:02:56.762919] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 508 ctx_id= CO_STACK hold= 10485760 used= 10321920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10321920 used= 10302720 count= 20 avg_used= 515136 mod=SUMMARY [2025-08-14 09:02:56.762953] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant: 509, limit: 1073741824 hold: 15613952 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 5128192 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10485760 limit= 9223372036854775807 [2025-08-14 09:02:56.762973] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 509 ctx_id= DEFAULT_CTX_ID hold= 5128192 used= 3765328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3765328 used= 3723912 count= 84 avg_used= 44332 mod=SUMMARY [2025-08-14 09:02:56.762992] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 509 ctx_id= CO_STACK hold= 10485760 used= 10321920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10321920 used= 10302720 count= 20 avg_used= 515136 mod=SUMMARY [2025-08-14 09:02:56.763033] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant: 1001, limit: 1073741824 hold: 606650368 cache_hold: 44040192 cache_used: 44040192 cache_item_count: 21 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 302309376 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 54525952 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 3080192 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 483328 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 45514752 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 5521408 limit= 157286400 [MEMORY] ctx_id= GLIBC hold_bytes= 15499264 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 73424896 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35651584 limit= 214748360 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 3252224 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 12582912 limit= 1073741800 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 5570560 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3866624 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 1179648 limit= 9223372036854775807 [2025-08-14 09:02:56.763063] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=14] [MEMORY] tenant_id= 1001 ctx_id= DEFAULT_CTX_ID hold= 302309376 used= 269051872 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 81 washed_blocks= 406 washed_size= 113410048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 269051872 used= 262935190 count= 3788 avg_used= 69412 mod=SUMMARY [2025-08-14 09:02:56.763072] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1001 ctx_id= MEMSTORE_CTX_ID hold= 54525952 used= 54099968 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 54099968 used= 54075216 count= 26 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:02:56.763078] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1001 ctx_id= EXECUTE_CTX_ID hold= 3080192 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 8 washed_size= 7405568 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:56.763085] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:02:56.763093] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= PLAN_CACHE_CTX_ID hold= 45514752 used= 36449984 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 77 washed_blocks= 201 washed_size= 132743168 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 36449984 used= 32916258 count= 3312 avg_used= 9938 mod=SUMMARY [2025-08-14 09:02:56.763099] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= WORK_AREA hold= 5521408 used= 0 limit= 157286400 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 9 washed_blocks= 40 washed_size= 17547264 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:56.763106] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= GLIBC hold= 15499264 used= 10889728 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 25 washed_blocks= 164 washed_size= 47415296 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10889728 used= 6020141 count= 18313 avg_used= 328 mod=SUMMARY [2025-08-14 09:02:56.763114] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= CO_STACK hold= 73424896 used= 71221248 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 32 washed_blocks= 34 washed_size= 2072576 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 71221248 used= 71088768 count= 138 avg_used= 515136 block_cnt= 138 chunk_cnt= 36 mod=CoStack [MEMORY] hold= 71221248 used= 71088768 count= 138 avg_used= 515136 mod=SUMMARY [2025-08-14 09:02:56.763124] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1001 ctx_id= KVSTORE_CACHE_ID hold= 44040192 used= 44040192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 44040192 used= 0 count= 21 avg_used= 0 mod=SUMMARY [2025-08-14 09:02:56.763131] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1001 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 33906624 limit= 214748360 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33906624 used= 30629648 count= 468 avg_used= 65447 mod=SUMMARY [2025-08-14 09:02:56.763137] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1001 ctx_id= TX_CALLBACK_CTX_ID hold= 3252224 used= 2342592 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 3 washed_blocks= 8 washed_size= 3039232 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2342592 used= 2269696 count= 286 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:56.763144] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1001 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:56.763152] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1001 ctx_id= RPC_CTX_ID hold= 12582912 used= 2039808 limit= 1073741800 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2039808 used= 1349248 count= 83 avg_used= 16256 mod=SUMMARY [2025-08-14 09:02:56.763161] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1001 ctx_id= TX_DATA_TABLE hold= 5570560 used= 2342784 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 8 washed_size= 2818048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2342784 used= 2319424 count= 30 avg_used= 77314 mod=SUMMARY [2025-08-14 09:02:56.763169] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1001 ctx_id= MDS_DATA_ID hold= 3866624 used= 2785280 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 5 washed_size= 4521984 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2785280 used= 2698240 count= 340 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:56.763181] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1001 ctx_id= MERGE_NORMAL_CTX_ID hold= 1179648 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 8 washed_blocks= 24 washed_size= 15597568 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:56.763196] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant: 1002, limit: 8589934592 hold: 1943941120 cache_hold: 48234496 cache_used: 48234496 cache_item_count: 23 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 1005416448 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 415236096 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 1114112 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 950272 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 211812352 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 1441792 limit= 429496725 [MEMORY] ctx_id= GLIBC hold_bytes= 12582912 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 149913600 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 37748736 limit= 1717986900 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 7766016 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 8077312 limit= 8589934500 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 5718016 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3047424 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 1032192 limit= 9223372036854775807 [2025-08-14 09:02:56.763220] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1002 ctx_id= DEFAULT_CTX_ID hold= 1005416448 used= 916397296 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 15 [MEMORY] hold= 185860096 used= 185534336 count= 114 avg_used= 1627494 block_cnt= 114 chunk_cnt= 104 mod=MysqlRequesReco [MEMORY] hold= 167792640 used= 167772192 count= 1 avg_used= 167772192 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 83927040 used= 83886080 count= 2 avg_used= 41943040 block_cnt= 2 chunk_cnt= 2 mod=LogGroupBuffer [MEMORY] hold= 67190784 used= 67109344 count= 4 avg_used= 16777336 block_cnt= 4 chunk_cnt= 4 mod=MrgMemPoolChk [MEMORY] hold= 916397296 used= 899198025 count= 9478 avg_used= 94872 mod=SUMMARY [2025-08-14 09:02:56.763230] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= MEMSTORE_CTX_ID hold= 415236096 used= 411992064 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 411992064 used= 411803568 count= 198 avg_used= 2079816 block_cnt= 198 chunk_cnt= 198 mod=Memstore [MEMORY] hold= 411992064 used= 411803568 count= 198 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:02:56.763250] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=18] [MEMORY] tenant_id= 1002 ctx_id= EXECUTE_CTX_ID hold= 1114112 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 7 washed_size= 7274496 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:56.763257] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1002 ctx_id= TRANS_CTX_MGR_ID hold= 950272 used= 933888 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1146880 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 933888 used= 921984 count= 2 avg_used= 460992 mod=SUMMARY [2025-08-14 09:02:56.763265] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1002 ctx_id= PLAN_CACHE_CTX_ID hold= 211812352 used= 201995008 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 176930880 used= 160228962 count= 13450 avg_used= 11912 block_cnt= 13450 chunk_cnt= 101 mod=SqlPhyPlan [MEMORY] hold= 201995008 used= 183048143 count= 17906 avg_used= 10222 mod=SUMMARY [2025-08-14 09:02:56.763271] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= WORK_AREA hold= 1441792 used= 0 limit= 429496725 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 14 washed_blocks= 54 washed_size= 27918336 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:56.763278] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1002 ctx_id= GLIBC hold= 12582912 used= 11425280 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 11425280 used= 6322556 count= 19216 avg_used= 329 mod=SUMMARY [2025-08-14 09:02:56.763286] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1002 ctx_id= CO_STACK hold= 149913600 used= 147087360 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 68 washed_blocks= 70 washed_size= 3178496 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 147087360 used= 146813760 count= 285 avg_used= 515136 block_cnt= 285 chunk_cnt= 73 mod=CoStack [MEMORY] hold= 147087360 used= 146813760 count= 285 avg_used= 515136 mod=SUMMARY [2025-08-14 09:02:56.763296] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1002 ctx_id= KVSTORE_CACHE_ID hold= 48234496 used= 48234496 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 48234496 used= 0 count= 23 avg_used= 0 mod=SUMMARY [2025-08-14 09:02:56.763302] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1002 ctx_id= META_OBJ_CTX_ID hold= 37748736 used= 35741632 limit= 1717986900 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 35741632 used= 32457488 count= 496 avg_used= 65438 mod=SUMMARY [2025-08-14 09:02:56.763309] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1002 ctx_id= TX_CALLBACK_CTX_ID hold= 7766016 used= 6934400 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 17 washed_size= 6914048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 6934400 used= 6729728 count= 848 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:56.763316] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1002 ctx_id= LOB_CTX_ID hold= 147456 used= 16384 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 16384 used= 15872 count= 2 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:56.763324] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1002 ctx_id= RPC_CTX_ID hold= 8077312 used= 3784704 limit= 8589934500 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 6 washed_blocks= 6 washed_size= 10797056 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3784704 used= 2490368 count= 160 avg_used= 15564 mod=SUMMARY [2025-08-14 09:02:56.763333] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1002 ctx_id= TX_DATA_TABLE hold= 5718016 used= 4521856 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 9 washed_blocks= 16 washed_size= 13156352 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4521856 used= 4480128 count= 40 avg_used= 112003 mod=SUMMARY [2025-08-14 09:02:56.763341] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1002 ctx_id= MDS_DATA_ID hold= 3047424 used= 1998848 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 4 washed_size= 1146880 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1998848 used= 1936384 count= 244 avg_used= 7936 mod=SUMMARY [2025-08-14 09:02:56.763347] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1002 ctx_id= MDS_CTX_ID hold= 147456 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:56.763356] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1002 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:56.763362] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1002 ctx_id= MERGE_NORMAL_CTX_ID hold= 1032192 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 12 washed_size= 13647872 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:02:56.757335] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=30] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:56.745121] WDIAG [STORAGE] init (ob_multiple_merge.cpp:182) [14945][T1_TimerWK8_MergeLoop][T1][YB42AC4D0007-00063C4978199C2C-0-0] [lt=994225][errcode=-4012] fail to prepare read tables(ret=-4012) [2025-08-14 09:02:56.763830] INFO [CLOG] operator() (ob_log_replay_service.cpp:1678) [14963][T1_TimerWK9_ReplayProcessSt][T1][Y0-0000000000000000-0-0] [lt=14] get_replay_process success(id={id:1}, submitted_log_size=451194359, unsubmitted_log_size=451194359, replayed_log_size=451194359, unreplayed_log_size=0) [2025-08-14 09:02:56.764293] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3224] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:56.764629] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=17] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:56.765806] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=18] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:56.767279] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=14] gc stale ls task succ [2025-08-14 09:02:56.771252] WDIAG ev_invoke_pending (ev.c:2136) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=10529][errcode=0] epoll handler too much time :10549 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:56.771287] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=3133] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:56.771307] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] temporary file module use no blocks [2025-08-14 09:02:56.771314] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:56.771324] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:56.771937] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=7243] temporary file module use no blocks [2025-08-14 09:02:56.771945] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:56.771951] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=6] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:56.771958] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=7] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623078204365, flush_io_finished_ret_:0, flush_io_finished_round_:26332, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:56.771998] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11131][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:56.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.770003] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:155) [408][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8064] unlock session of tenant(conn.sessid_=3221590970, proxy_sessid=0, conn.tenant_id_=1002) [2025-08-14 09:02:56.772111] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:244) [408][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=2104] connection close(sessid=3221590970, proxy_sessid=0, tenant_id=1002, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-08-14 09:02:56.772281] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9058] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:56.772335] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:56.758049] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:244) [411][sql_nio4][T0][Y0-0000000000000000-0-0] [lt=16] connection close(sessid=3221590965, proxy_sessid=0, tenant_id=0, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:0], disconnect_state=0) [2025-08-14 09:02:56.771335] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623078210242, flush_io_finished_ret_:0, flush_io_finished_round_:26310, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:57.751027] EDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [982][T1002_L0_G0][T1002][YB42AC4D0007-00063C4962CA3040-0-0] [lt=998862][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1010444, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1010439, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x974f472 0x974e159 0x974e084 0x948fab3 0x97257c0 0x97256f4 0x9725691 0x20b8d66c 0x20c1ea16 0x1f11562c 0x9703542 0x9700852 0x970011d 0x15f6f4c7 0x15ebaffc 0x15eba874 0x9343ecf 0x9341ca3 0x92e8803 0x92e30c4 0x92df309 0x92db9ba 0x92d9f5e 0x9828a7d 0x9298239 0x9285f13 0x927efad 0x926c154 0x12c5a0c9 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:56.775293] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=20] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:56.775312] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=16] temporary file module use no blocks [2025-08-14 09:02:56.775322] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:56.775335] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:56.775349] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623078212621, flush_io_finished_ret_:0, flush_io_finished_round_:26344, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:01:40.765982] EDIAG [MDS] mds_range_query (ob_i_tablet_mds_interface.ipp:844) [806][T1002_TimerWK2_SSTableGC][T1002][YB42AC4D0007-00063C495F3A18A1-0-0] [lt=5004829][errcode=-4012] LOGGER COST TOO MUCH TIME, cost: 75899879, owner: ObLog, click_count: 5, time dist:[FORMAT_END=19961842, ObMalloc#start=1, ObMalloc#end=34, ALLOC_END=55937998, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1bc2a36e 0x9764e72 0x975ab8f 0x975a314 0x975972e 0x9758a1f 0x1aee8055 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:00:25.734026] EDIAG [STORAGE.TRANS] handle_timeout (ob_trans_part_ctx.cpp:629) [812][T1002_TransTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4012] LOGGER COST TOO MUCH TIME, cost: 1017163, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1017163, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b7578bd 0x1b754499 0x1b88503c 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:56.778422] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:56.744546] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728be10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:56.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:56.778964] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34361][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72847b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:56.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:56.778991] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:56.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:56.779004] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891806f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:56.751, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:56.779016] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:56.750, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:56.779029] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73539b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7396b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:56.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:56.753532] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:56.779680] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=26106][errcode=0] epoll handler too much time :26153 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:56.780393] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:174) [14963][T1_TimerWK9_ReplayProcessSt][T1][Y0-0000000000000000-0-0] [lt=10] dump tenant replay process(tenant_id=1, unsubmitted_log_size(MB)=430, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=430, replayed_log_size(MB)=430, last_submitted_log_size(MB)=430, last_replayed_log_size(MB)=430, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-08-14 09:02:56.780412] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [14963][T1_TimerWK9][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f994e1a49d8, timer:0x7f9959ffbdb0, task:0x7f994f0044d0, task_type:"N9oceanbase10logservice17ReplayProcessStatE", scheduled_time:1755162057738245, pushed_time:1755162057758376, delay:10000000}, delay_in_thread_pool=118984027, threashold=500000) [2025-08-14 09:01:40.765982] WDIAG [MDS] mds_range_query (ob_i_tablet_mds_interface.ipp:844) [806][T1002_TimerWK2_SSTableGC][T1002][YB42AC4D0007-00063C495F3A18A1-0-0] [lt=5004829][errcode=-4012] fail to init range scan iter(ret=-4012, scan_param={tablet_id:{id:101047}, ls_id:{id:1}, column_ids:[16, 17, 18, 19], index_id:88888, key_ranges:[{"range":"table_id:88888,group_idx:0,index_ordered_idx:0,({"TINYINT":3},MIN;{"TINYINT":3},MAX)"}], ss_key_ranges:[], range_array_pos:[], timeout:1755161750753477, scan_flag:{scan_order:0, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_bare_row_scan:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:1, is_select_follower:0, enable_lob_prefetch:0, is_bare_row_scan:0, mr_mv_scan:0, is_plain_insert_gts_opt:0, reserved:0}, sql_mode:134217760, reserved_cell_count:4, schema_version:9527, query_begin_schema_version:-1, limit_offset:{offset_:0, limit_:-1}, for_update:false, wait:-1, frozen_version:9223372036854775807, is_get:false, output_exprs:NULL, op_filters:NULL, trans_desc:NULL, snapshot:{this:0x7f99013cf150, valid:true, source:2, core:{version:{val:4611686018427387903, v:0}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, table_param:{table_id:88888, output_projector:[0, 1, 2, 3], aggregate_projector:[], group_by_projector:[], output_sel_mask:[true, true, true, true], pad_col_projector:[], read_param_version:1, main_read_info:{schema_column_count:4, schema_rowkey_cnt:2, rowkey_cnt:4, trans_col_index:-1, mview_old_new_col_index:-1, group_idx_col_index:-1, seq_read_column_count:2, max_col_index:5, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 4, 5], data:0x7f9898766740, count:4, allocator:0x7f99013cf458, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 3], data:0x7f9898766760, count:4, allocator:0x7f99013cf458, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=16 {type:"TINYINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"CHAR", collation:"binary", coercibility:"INVALID"} order=0, column_id=18 {type:"CHAR", collation:"binary", coercibility:"INVALID"} order=0, column_id=19 {type:"CHAR", collation:"binary", coercibility:"INVALID"} order=0], data:0x7f9898766700, count:4, allocator:0x7f99013cf458, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:4, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}], data:0x7f98987667b0, count:4, allocator:0x7f99013cf458, capacity:4, init_cnt:4}, has_all_column_group:true, need_truncate_filter:false}, use_lob_locator:false, rowid_version:0, rowid_projector:[], enable_lob_locator_v2:false, is_fts_index:false, parser_name:"", parser_properties:"", is_vec_index:false, is_column_replica_table:false, is_normal_cgs_at_the_end:false, is_mlog_table:false}, sample_info:{method:0, percent:"1.000000000000000000e+02", seed:-1, table_id:0, scope:0, force_block:false}, need_scn:false, need_switch_param:false, is_mds_query:true, fb_read_tx_uncommitted:false, external_file_format:{"TYPE":"INVALID"}, external_file_location:"", tx_seq_base:-1, auto_split_filter_type:18446744073709551615, is_tablet_spliting:false, need_update_tablet_param:false, mds_collector:{exist_uncommitted_node:false, exist_new_committed_node:false}}) [2025-08-14 09:02:56.781017] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=21] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:56.781033] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:56.781042] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:56.781050] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:56.781059] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:56.781069] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:56.781077] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:56.781085] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:56.781092] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:02:56.781102] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:02:56.781111] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:02:56.782567] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=26] gc stale ls task succ [2025-08-14 09:02:56.817245] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:56.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:56.817286] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d22b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eaf90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:56.808, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:56.822157] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:56.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.827314] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:56.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:56.827351] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:56.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:56.831480] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:02:56.832393] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=427090055, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:56.832417] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:56.833081] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1001}, delta=432103424, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:56.833103] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:56.837164] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=17] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:02:56.840704] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13] LogLoopThread round_cost_time(us)(round_cost_time=12) [2025-08-14 09:02:56.844033] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=629103317, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:56.844051] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:56.848600] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:56.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.848689] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:56.831, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:56.848717] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:56.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:56.858752] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ac10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:56.849, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:56.861271] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=457085737, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:56.861298] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:56.861389] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [758][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB90EC-0-0] [lt=28][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1001, svr="172.77.0.7:2882", version={val:1755161719775179002, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162176860593) [2025-08-14 09:02:56.861420] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [758][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB90EC-0-0] [lt=25][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1001, req={req_server:"172.77.0.7:2882", version:{val:1755161719775179002, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162176860593}, twrs={inited:true, tenant_id:1001, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162176860631}, server_version_for_stat:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162176860631}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:56.861529] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [316][pnio1][T1001][YB42AC4D0007-00063C4960CB90EC-0-0] [lt=14][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1001, delta=1102744, last_succ_cluster_heartbeat_tstamp_=1755162175758784) [2025-08-14 09:02:56.862347] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=27] get wrs ts(ls_id={id:1}, delta=457085737, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:56.862369] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:56.865331] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a5df8, timer:0x7f99200ff180, task:0x7f99200ff1c0, task_type:"N9oceanbase8observer18ObTableLoadService15ObHeartBeatTaskE", scheduled_time:1755162055729056, pushed_time:0, delay:10000000}, running_task_token={this:0x7f991b7a5fb8, timer:0x7f99200ff180, task:0x7f99200ff1f0, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162050755138, pushed_time:1755162084760013, delay:1000000}, total_delay=121136274, threashold=1000000, tenant_id=1002) [2025-08-14 09:02:56.870306] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:56.852, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:56.870340] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:56.861, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:56.888356] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=32][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a66b8, timer:0x7f9966fddee0, task:0x7f994f1b1e78, task_type:"N9oceanbase5share23ObStorageIOUsageRepoter23TenantIOUsageReportTaskE", scheduled_time:1755161694839037, pushed_time:0, delay:5000000}, running_task_token={this:0x7f994e1a65d8, timer:0x7f9966fddee0, task:0x7f994f1bbcc0, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161694824965, pushed_time:1755161925772608, delay:5000000}, total_delay=482049318, threashold=1000000, tenant_id=1) [2025-08-14 09:02:56.905266] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919859b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:56.896, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:56.922208] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:56.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.932427] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25] get wrs ts(ls_id={id:1}, delta=427190339, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:56.932451] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:56.933090] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1001}, delta=432204768, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:56.933111] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:56.940789] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:02:56.940822] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=27] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:02:56.944733] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=629205739, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:56.944753] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:56.948646] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=53][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:56.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.962428] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=457187048, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:56.962453] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:56.994701] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:56.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.017781] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:57.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.017826] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:57.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.022245] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:56.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.032450] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=427290558, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:57.032475] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:57.033067] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1001}, delta=432304985, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:57.033086] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:57.044341] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161101770923, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755161572801635) [2025-08-14 09:02:57.044941] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] get wrs ts(ls_id={id:1}, delta=629304894, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:57.044961] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:57.048697] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:56.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.062516] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=457286838, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:57.062540] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:57.098254] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:57.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.122321] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:57.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.132520] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1}, delta=427390695, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:57.132553] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=29] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:57.133157] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1001}, delta=432405123, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:57.133178] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:57.144503] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B92BB-0-0] [lt=28][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162177144356) [2025-08-14 09:02:57.144541] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B92BB-0-0] [lt=38][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162177144356}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162177044365}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162177044365}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:02:57.144759] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [316][pnio1][T1][YB42AC4D0007-00063C49607B92BB-0-0] [lt=25][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1, delta=606482, last_succ_cluster_heartbeat_tstamp_=1755162176538277) [2025-08-14 09:02:57.144939] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=629404983, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:57.144956] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:57.145547] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=629404983, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:57.145562] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:57.148722] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:57.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.162531] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] get wrs ts(ls_id={id:1}, delta=457386354, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:57.162555] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:57.219049] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:57.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.222384] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:57.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.232555] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=427490407, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:57.232579] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:57.233184] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26] get wrs ts(ls_id={id:1001}, delta=432504836, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:57.233201] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:57.238299] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:57.238335] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:57.238348] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:57.238379] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:57.238393] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:57.238408] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:57.238430] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:57.238444] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:57.238465] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:57.238483] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:57.238672] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:57.238691] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:57.242256] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=6] dummy location not credible, need refresh(tenant_id_=1, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:57.242281] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=24] waiting for refresh location cache for target_monitor(tenant_id_=1, refresh_ctrl=3988) [2025-08-14 09:02:57.242312] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1) [2025-08-14 09:02:57.242326] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8] dummy location not credible, need refresh(tenant_id_=1001, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:57.242332] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=6] waiting for refresh location cache for target_monitor(tenant_id_=1001, refresh_ctrl=3989) [2025-08-14 09:02:57.242336] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1001) [2025-08-14 09:02:57.242497] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:182) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5] dummy location not credible, need refresh(tenant_id_=1002, server_="172.77.0.7:2882", dummy_cache_leader_="172.77.0.7:2882", role=2) [2025-08-14 09:02:57.242512] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:233) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=15] waiting for refresh location cache for target_monitor(tenant_id_=1002, refresh_ctrl=3990) [2025-08-14 09:02:57.242517] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [1031][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] refresh dummy location cache(ret=0, tenant_id_=1002) [2025-08-14 09:02:57.245630] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] get wrs ts(ls_id={id:1}, delta=629506754, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:02:57.245654] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:57.246327] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:57.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.248784] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:57.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.249041] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:02:57.249107] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:02:57.249161] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:02:57.249213] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:02:57.249250] INFO [COMMON] print_io_status (ob_io_manager.cpp:2639) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4082, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:474324, total:474329) [2025-08-14 09:02:57.249278] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, param_config_:{memory_limit:536870912, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=865568, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:02:57.249339] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2826) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=55] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4082, [delay/us]: prepare:0, schedule:0, submit:600000000, device:474324, total:474329) [2025-08-14 09:02:57.249358] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:02:57.249375] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:57.249395] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:57.249406] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SERVER] net_in=1kB/s, net_out=1kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=1280000kB/s [2025-08-14 09:02:57.249539] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=6] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:57.249557] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162177249534], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:57.249576] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162177249534], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:57.249593] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162177249534]}) [2025-08-14 09:02:57.249605] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:57.249624] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:02:57.249635] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:57.249643] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162177249634], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:57.249655] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162177249634]}) [2025-08-14 09:02:57.249846] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:57.249860] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:57.249874] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162177249842], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:57.249886] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162177249842]}) [2025-08-14 09:02:57.256399] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:57.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.262585] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=457486586, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:02:57.269296] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:57.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.269339] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:57.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.322445] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:57.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.332554] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=427591274, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:02:57.332576] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:57.333159] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1001}, delta=432604643, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:02:57.333172] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:57.348858] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:57.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.422508] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:57.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.457258] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:57.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.491735] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:57.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.511455] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:57.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.511500] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:57.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.522608] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:57.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.592553] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:57.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.621977] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:57.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.622645] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=52][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:57.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.639401] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571634a0, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957163780, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:5.00s, expected_run_time:09:02:57.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.639445] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:57.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.657377] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:57.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.713453] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:57.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.720361] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:57.723406] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:57.725056] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:57.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.725128] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:57.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.729045] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:57.734290] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:02:57.735732] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:57.724, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:57.736312] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:57.740279] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:57.740315] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:57.740328] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:57.740346] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:57.740357] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:57.740381] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:57.740410] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:57.740422] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:57.740432] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:57.740444] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:57.740600] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:57.740613] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:57.742499] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:57.742550] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:57.745256] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:57.745860] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:57.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.748431] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:57.748474] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:57.748502] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:57.750103] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:57.750135] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=31][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162177750099], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:57.750158] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162177750099], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:57.750189] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=22] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162177750099]}) [2025-08-14 09:02:57.750206] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162177750099]}) [2025-08-14 09:02:57.750222] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=13] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:02:57.750245] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162177750221], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:57.750260] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162177750221]}) [2025-08-14 09:02:57.750270] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162177750221]}) [2025-08-14 09:02:57.750467] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:02:57.750482] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:57.755452] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6132] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:57.755921] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:57.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:57.755963] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:57.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:57.755996] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:57.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.757255] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:57.758281] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:57.759147] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=16] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:57.763840] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=14] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:57.763889] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=12] gc stale ls task succ [2025-08-14 09:02:57.764032] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:57.764770] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=11] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:57.262610] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:57.765961] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a4578, timer:0x7f9966a63f10, task:0x7f9922e05028, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr17RefreshConfigTaskE", scheduled_time:1755162055747671, pushed_time:0, delay:10000000}, running_task_token={this:0x7f991b7a4498, timer:0x7f9966a63f10, task:0x7f9922e05010, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr11TableGCTaskE", scheduled_time:1755162048800605, pushed_time:1755162048800872, delay:20000}, total_delay=122018289, threashold=1000000, tenant_id=1002) [2025-08-14 09:02:57.766281] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:57.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:57.766628] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:57.766646] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:02:57.768297] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=11] gc stale ls task succ [2025-08-14 09:02:57.768653] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=20] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:02:57.772800] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=18] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:57.772836] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=29] temporary file module use no blocks [2025-08-14 09:02:57.772846] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:57.772870] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=23] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:57.772890] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=20] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623079211745, flush_io_finished_ret_:0, flush_io_finished_round_:26333, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:57.772988] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=5592] current monitor number(seq_=-1) [2025-08-14 09:02:57.773256] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=1889] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:57.773278] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=19] temporary file module use no blocks [2025-08-14 09:02:57.773283] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=5] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:57.773296] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:02:57.773320] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=24] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623079212997, flush_io_finished_ret_:0, flush_io_finished_round_:26311, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:57.775723] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:57.775790] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:57.775914] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=24] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:02:57.775932] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] temporary file module use no blocks [2025-08-14 09:02:57.776685] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:57.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:57.776728] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:57.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:57.776740] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d36b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ebb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:57.767, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:57.776751] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:57.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:57.779261] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:57.782631] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=24] gc stale ls task succ [2025-08-14 09:02:57.786776] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:57.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:57.786809] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72739b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a5e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:57.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:57.786822] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72779b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eab10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:57.777, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:57.786848] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728bc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:57.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:57.797759] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919902b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:57.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:57.797792] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:57.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:57.797806] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:57.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:57.807919] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7277730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:57.794, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:57.807955] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:57.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:57.828126] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:57.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:57.837821] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=19] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:02:57.838295] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:57.820, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:57.848346] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=49][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:57.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:57.857526] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:57.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.866060] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a5ed8, timer:0x7f99200ff180, task:0x7f99200ff1d8, task_type:"N9oceanbase8observer18ObTableLoadService8ObGCTaskE", scheduled_time:1755162056715823, pushed_time:0, delay:30000000}, running_task_token={this:0x7f991b7a5fb8, timer:0x7f99200ff180, task:0x7f99200ff1f0, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162050755138, pushed_time:1755162084760013, delay:1000000}, total_delay=121150235, threashold=1000000, tenant_id=1002) [2025-08-14 09:02:57.969221] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196aa90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:57.952, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:57.969278] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=54][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:57.953, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:57.989358] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891992fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196de90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:57.970, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:57.999682] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:57.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:58.029939] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:58.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:58.057651] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:57.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:58.240803] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:58.240854] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:58.240866] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:58.240883] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:58.240894] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:58.240911] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:58.240928] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:58.240940] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:58.240950] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:58.240963] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:58.241116] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:58.241130] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:58.249475] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:58.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:58.258321] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:58.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:58.259360] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:02:58.259417] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:02:58.259466] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:02:58.259512] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:02:58.259531] INFO [COMMON] print_io_status (ob_io_manager.cpp:2639) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4054, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:505204, total:505209) [2025-08-14 09:02:58.259551] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, param_config_:{memory_limit:536870912, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=865568, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:02:58.259591] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2826) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=36] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4054, [delay/us]: prepare:0, schedule:0, submit:600000000, device:505204, total:505209) [2025-08-14 09:02:58.259569] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=63][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:58.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:58.259610] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:02:58.259631] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:58.259645] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:02:58.279831] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:58.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:58.310086] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=51][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:58.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:58.458469] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:58.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:58.491561] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:58.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:58.521805] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:58.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:58.605406] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEA9-0-0] [lt=26] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:02:58.640096] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:58.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:58.658624] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:58.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:58.720554] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=36] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:58.721531] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:58.731321] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:56.998109] EDIAG eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1732828, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1732824, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x27a8a8c8 0x97b7e1a 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:57.753666] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:56.998109] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=23] PNIO [ratelimit] time: 1755162176998107, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007615, add_bytes: 0 [2025-08-14 09:02:58.733609] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=1732828] PNIO [ratelimit] time: 1755162178733608, bytes: 0, bw: 0.000000 MB/s, add_ts: 1735501, add_bytes: 0 [2025-08-14 09:02:58.739244] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:58.742403] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:58.742573] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=49][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8030, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8310, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:10.00s, expected_run_time:09:02:58.730, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:58.742620] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:58.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:58.744293] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=31] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:57.766224] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:57.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:57.953644] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] EASY SLOW: thread: 7f99540fb300, lastrun: 1755162177.753656 cost: 0.199984 loop:26554, slowcnt: 1 [2025-08-14 09:02:57.742600] INFO [STORAGE.TRANS] statistics (ob_location_adapter.cpp:70) [1031][PxTargetMgr0][T1][Y0-0000000000000000-0-0] [lt=4] location adapter statistics(renew_access=12, total_access=124, error_count=0, renew_rate=9.600000083e-02) [2025-08-14 09:02:57.765385] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:288) [860][T1002_ArcFetche][T1002][YB42AC4D0007-00063C495E399C16-0-0] [lt=25] ObArchiveFetcher is running(thread_index=0) [2025-08-14 09:02:57.775941] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:02:58.746979] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:58.747892] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:58.747926] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:58.747950] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:58.748395] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:02:58.748431] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:02:58.748442] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:02:58.748460] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:02:58.748470] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:02:58.748487] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:02:58.748515] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:02:58.748532] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:02:58.748542] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:02:58.748555] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:02:58.748725] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:02:58.748738] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:58.751265] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:58.753809] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:58.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:58.755506] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:58.759057] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=17] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:58.733537] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:233) [827][T1002_RFLWorker][T1002][YB42AC4D0007-00063C495D099C16-0-0] [lt=636] ObRemoteFetchWorker is running(thread_index=0) [2025-08-14 09:02:58.742336] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=45] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:02:58.763270] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:58.763962] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=28] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:58.764879] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:02:58.765988] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=23] gc stale ls task succ [2025-08-14 09:02:57.825127] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:57.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:56.780444] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:147) [14963][T1_TimerWK9_][T1][Y0-0000000000000000-0-0] [lt=13] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-08-14 09:02:58.769330] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=17] gc stale ls task succ [857]OB_MALLOC COST TOO MUCH TIME, cost_time=1326526, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:02:57.766555] EDIAG [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:171) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1986053, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1986052, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1b538df0 0x1b4f8d72 0x1b4fbf53 0x1b5014e0 0x1b5009d1 0x1b50387c 0x1b8905de 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:59.260502] EDIAG [COMMON] print_io_status (ob_io_manager.cpp:2639) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1456151, owner: ObLog, click_count: 3, time dist:[FORMAT_END=5, ALLOC_END=1456144, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1f9eb590 0x97c91ec 0x1f981a55 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:59.275800] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=57][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1440950, owner: ObLog, click_count: 3, time dist:[FORMAT_END=25, ALLOC_END=1440925, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:59.327812] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1389090, owner: ObLog, click_count: 3, time dist:[FORMAT_END=23, ALLOC_END=1389067, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:57.768646] EDIAG [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=26307][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2947002, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1979279, ALLOC_END=967723, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1fce4e91 0x1fce455d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [567]OB_MALLOC COST TOO MUCH TIME, cost_time=1413885, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=1930330, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1930327], seq:[0] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=1938262, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1938255], seq:[0] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=1931447, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1931435], seq:[0] [2025-08-14 09:02:58.770909] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:58.780371] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:58.781047] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:58.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:58.801526] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:58.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:58.927603] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:58.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:59.023691] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:59.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:59.027665] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:58.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:59.127728] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:59.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:59.227771] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:59.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:59.260355] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:02:59.260408] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:02:59.260444] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:02:59.260486] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:02:59.746570] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:57.766555] INFO [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:171) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=8] handle gts request statistics(total_rt=1906, total_cnt=16, avg_rt=1.121176470588235361e+02, avg_cnt=1.600000000000000089e+00) [2025-08-14 09:02:59.755656] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=1986053][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162177750462], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:59.755673] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162177750462], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:02:59.755687] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162177750462]}) [2025-08-14 09:02:59.766422] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=21] gc stale ls task succ [2025-08-14 09:02:59.772362] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=19] gc stale ls task succ [2025-08-14 09:02:59.772470] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:59.260502] INFO [COMMON] print_io_status (ob_io_manager.cpp:2639) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4092, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:476115, total:476120) [471]OB_MALLOC COST TOO MUCH TIME, cost_time=1910034, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=32896, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1910032], seq:[0] [729]OB_MALLOC COST TOO MUCH TIME, cost_time=994858, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=1974047, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1930344], seq:[0, 1] [2025-08-14 09:02:58.788315] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1974081, owner: ObLog, click_count: 5, time dist:[FORMAT_END=19, ObMalloc#start=1, ObMalloc#end=1974059, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [403]OB_MALLOC COST TOO MUCH TIME, cost_time=1982109, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=58, WASH_BLOCK_END=130, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1938280], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:58.780322] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1982133, owner: ObLog, click_count: 5, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=1982121, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:58.928244] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [571][T1_ArcSender][T1][YB42AC4D0007-00063C495BA99C16-0-0] [lt=13][errcode=-4018] LOGGER COST TOO MUCH TIME, cost: 1834520, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1834519, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3b4b 0x113d34aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:58.927319] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=29][errcode=-4018] LOGGER COST TOO MUCH TIME, cost: 1835475, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1835475, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3b4b 0x113d34aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [776]OB_MALLOC COST TOO MUCH TIME, cost_time=1982524, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1931463], seq:[0, 1] [2025-08-14 09:02:58.788348] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [776][T1002_StandbySc][T1002][YB42AC4D0007-00063C495DFA032D-0-0] [lt=20][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1982551, owner: ObLog, click_count: 5, time dist:[FORMAT_END=9, ObMalloc#start=0, ObMalloc#end=1982540, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:57.766428] EDIAG [STORAGE.TRANS] refresh_elr_tenant_config_ (ob_tx_elr_util.cpp:42) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3002985, owner: ObLog, click_count: 3, time dist:[FORMAT_END=3002984, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b878a47 0x1b8787a8 0x1b878197 0x986bac9 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:57.768600] EDIAG [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3004083, owner: ObLog, click_count: 3, time dist:[FORMAT_END=3004081, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x12377491 0x1236e8e9 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [332]OB_MALLOC COST TOO MUCH TIME, cost_time=1988896, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1988890], seq:[0] [2025-08-14 09:02:59.275800] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=57][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:59.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:00.718955] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1440950][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:59.275|207=1.44s|210=1us|222=2us|total=1.44s) [2025-08-14 09:03:00.718984] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=-4389] timer task use too much time(end=1755162180718982, start=1755162179275798, delta=1443184) [2025-08-14 09:02:59.327812] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:59.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:00.719023] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1389090][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:59.327|207=1.39s|210=0us|222=1us|total=1.39s) [2025-08-14 09:03:00.719031] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] timer task use too much time(end=1755162180719031, start=1755162179327811, delta=1391220) [2025-08-14 09:02:57.768646] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=26307] blacklist_loop exec finished(cost_time=27, is_enabled=true, send_cnt=0) [2025-08-14 09:02:59.755057] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:59.772418] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=631] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:59.755699] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts statistics(tenant_id=1002, gts_rpc_cnt=0, get_gts_cache_cnt=75, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:03:00.729099] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=971926] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162177750462]}) [2025-08-14 09:03:00.729119] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1002}) [2025-08-14 09:03:00.729130] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1002}) [2025-08-14 09:03:00.720666] WDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=964149][errcode=0] epoll handler too much time :965616 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:00.720780] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=946928][errcode=0] epoll handler too much time :948370 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:00.748948] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:00.755500] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24416] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:58.788315] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=17] gc stale ls task succ [2025-08-14 09:03:00.765406] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=1974081] gc stale ls task succ [2025-08-14 09:02:58.780322] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:00.718914] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=1456151] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, param_config_:{memory_limit:536870912, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=865568, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:03:00.765474] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=1982133][errcode=0] epoll handler too much time :1985162 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:00.765517] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=33] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:58.746562] EDIAG [RS.COMPACTION] try_idle (ob_major_merge_scheduler.cpp:226) [1057][T1_MergeSchedul][T1][YB42AC4D0007-00063C4961599CC3-0-0] [lt=31][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1995579, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1995578, ALLOC_END=0, APPEND_END=0, ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=60], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1125d502 0x1125cf81 0x1125a4c3 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:57.866716] EDIAG [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3861980, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=1, FORMAT_END=3861973, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1bf2ec62 0x986ebd7 0x986bee6 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:00:25.738272] EDIAG [SQL.PC] need_check_schema_version (ob_plan_cache_value.cpp:2334) [813][T1002_DupTbLeas][T1001][YB42AC4D0007-00063C496039A468-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1982870, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=29901, ALLOC_BLOCK#end=7525, FORMAT_END=1945442, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0x15385470 0x9433023 0x92b85e2 0x92b41d6 0x92b202a 0x92d2008 0x96a777d 0x96a6679 0x96a5033 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x1ee93940 0x1ee93063 0x1d53bd1b 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [332]OB_MALLOC COST TOO MUCH TIME, cost_time=2947746, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1988911], seq:[0, 1] [2025-08-14 09:02:58.788508] EDIAG do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2947779, owner: ObLog, click_count: 5, time dist:[FORMAT_END=10, ObMalloc#start=0, ObMalloc#end=2947762, ALLOC_END=7, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x202c530b 0x202c6888 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:58.745845] EDIAG [ELECT] operator() (election_acceptor.cpp:137) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2990415, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2990414, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xfa03487 0x9835ae8 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [374]OB_MALLOC COST TOO MUCH TIME, cost_time=2947244, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=2947235], seq:[0] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=2947257, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2947253], seq:[0, 1] [2025-08-14 09:02:58.749830] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2986861, owner: ObLog, click_count: 7, time dist:[FORMAT_END=27, ObMalloc#start=0, ObMalloc#end=39540, ObMalloc#start=30, ObMalloc#end=2947263, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [419]OB_MALLOC COST TOO MUCH TIME, cost_time=2890733, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=2890722, ALLOC_CHUNK_END=5], seq:[0, 1] [2025-08-14 09:02:58.746590] EDIAG [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=969180][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2997655, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2997653, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0xfd250b6 0xfd247d3 0xfd244ea 0xfd242a2 0x1d19629b 0x1d17baa3 0x1d17ad94 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:58.760302] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2984042, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2984041, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [419]OB_MALLOC COST TOO MUCH TIME, cost_time=2919355, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2890751, WASH_BLOCK_END=99, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=28501], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:58.848171] EDIAG try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2919395, owner: ObLog, click_count: 5, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=2919378, ALLOC_END=4, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x26499505 0x97074a3 0x26498fa9 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:57.841286] EDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3927461, owner: ObLog, click_count: 3, time dist:[FORMAT_END=3927455, ALLOC_END=5, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0xfc0e73d 0x993f560 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [808]OB_MALLOC COST TOO MUCH TIME, cost_time=2898096, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=2898089], seq:[0] [808]OB_MALLOC COST TOO MUCH TIME, cost_time=2898112, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2898110], seq:[0, 1] [2025-08-14 09:02:58.873273] EDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 2898147, owner: ObLog, click_count: 5, time dist:[FORMAT_END=23, ObMalloc#start=1, ObMalloc#end=2898119, ALLOC_END=3, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x9757ced 0x97569df 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [466]OB_MALLOC COST TOO MUCH TIME, cost_time=2883955, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=2883949], seq:[0] [466]OB_MALLOC COST TOO MUCH TIME, cost_time=2883964, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2883963], seq:[0, 1] [2025-08-14 09:02:58.887671] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=59][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2884000, owner: ObLog, click_count: 5, time dist:[FORMAT_END=29, ObMalloc#start=0, ObMalloc#end=2883970, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [651]OB_MALLOC COST TOO MUCH TIME, cost_time=2867886, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=2867881], seq:[0] [651]OB_MALLOC COST TOO MUCH TIME, cost_time=2867894, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2867893], seq:[0, 1] [2025-08-14 09:02:58.903857] EDIAG [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EEC9-0-0] [lt=26][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2867916, owner: ObLog, click_count: 5, time dist:[FORMAT_END=14, ObMalloc#start=1, ObMalloc#end=2867899, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1dc20366 0x1dc1fa40 0x1dd7a832 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [404]OB_MALLOC COST TOO MUCH TIME, cost_time=2806022, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=2806017], seq:[0] [404]OB_MALLOC COST TOO MUCH TIME, cost_time=2806032, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2806031], seq:[0, 1] [2025-08-14 09:02:58.965887] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=260868][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2806060, owner: ObLog, click_count: 5, time dist:[FORMAT_END=17, ObMalloc#start=1, ObMalloc#end=2806037, ALLOC_END=5, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [474]OB_MALLOC COST TOO MUCH TIME, cost_time=2778126, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=2778122], seq:[0] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=2778133, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2778132], seq:[0, 1] [2025-08-14 09:02:58.993849] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2778179, owner: ObLog, click_count: 5, time dist:[FORMAT_END=38, ObMalloc#start=1, ObMalloc#end=2778139, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [276]OB_MALLOC COST TOO MUCH TIME, cost_time=2685015, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=2685010], seq:[0] [276]OB_MALLOC COST TOO MUCH TIME, cost_time=2685024, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2685023], seq:[0, 1] [2025-08-14 09:02:59.087262] EDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=885792][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 2685059, owner: ObLog, click_count: 5, time dist:[FORMAT_END=28, ObMalloc#start=0, ObMalloc#end=2685029, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x9757ced 0x97569df 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [372]OB_MALLOC COST TOO MUCH TIME, cost_time=2523773, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=2523769], seq:[0] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=2523779, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2523779], seq:[0, 1] [2025-08-14 09:02:59.248908] EDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] LOGGER COST TOO MUCH TIME, cost: 2523814, owner: ObLog, click_count: 5, time dist:[FORMAT_END=27, ObMalloc#start=1, ObMalloc#end=2523785, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x13b06ffb 0x958a39e 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [768]OB_MALLOC COST TOO MUCH TIME, cost_time=2132801, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=2132796], seq:[0] [768]OB_MALLOC COST TOO MUCH TIME, cost_time=2132808, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2132808], seq:[0, 1] [2025-08-14 09:02:59.640855] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2132869, owner: ObLog, click_count: 5, time dist:[FORMAT_END=52, ObMalloc#start=1, ObMalloc#end=2132815, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [398]OB_MALLOC COST TOO MUCH TIME, cost_time=2051510, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=2051505], seq:[0] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=2051517, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2051516], seq:[0, 1] [2025-08-14 09:02:59.722373] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2051542, owner: ObLog, click_count: 5, time dist:[FORMAT_END=15, ObMalloc#start=2, ObMalloc#end=2051522, ALLOC_END=3, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [382]OB_MALLOC COST TOO MUCH TIME, cost_time=2051206, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=2051197], seq:[0] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=2051216, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2051215], seq:[0, 1] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=2044760, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=2044751, ALLOC_CHUNK_END=4], seq:[0, 1] [2025-08-14 09:02:59.722781] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2051244, owner: ObLog, click_count: 5, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=2051222, ALLOC_END=11, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [556]OB_MALLOC COST TOO MUCH TIME, cost_time=2038384, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=2038374, ALLOC_CHUNK_END=5], seq:[0, 1] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=2039863, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=2039854, ALLOC_CHUNK_END=4], seq:[0, 1] [386]OB_MALLOC COST TOO MUCH TIME, cost_time=2036797, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=2036788, ALLOC_CHUNK_END=4], seq:[0, 1] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=2030909, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=2030900, ALLOC_CHUNK_END=4], seq:[0, 1] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=2029020, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=2029005, ALLOC_CHUNK_END=10], seq:[0, 1] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=2022214, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=2022205, ALLOC_CHUNK_END=4], seq:[0, 1] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=2022014, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=2022006, ALLOC_CHUNK_END=3], seq:[0, 1] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=2021815, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=2021806, ALLOC_CHUNK_END=4], seq:[0, 1] [388]OB_MALLOC COST TOO MUCH TIME, cost_time=2031666, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=2031657, ALLOC_CHUNK_END=4], seq:[0, 1] [387]OB_MALLOC COST TOO MUCH TIME, cost_time=2021743, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=2021733, ALLOC_CHUNK_END=5], seq:[0, 1] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=2021743, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=2021726, ALLOC_CHUNK_END=12], seq:[0, 1] [383]OB_MALLOC COST TOO MUCH TIME, cost_time=2021721, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=2021711, ALLOC_CHUNK_END=5], seq:[0, 1] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=2020264, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=2020255, ALLOC_CHUNK_END=4], seq:[0, 1] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=2016216, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=2016207, ALLOC_CHUNK_END=4], seq:[0, 1] [864]OB_MALLOC COST TOO MUCH TIME, cost_time=2014786, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=2014777, ALLOC_CHUNK_END=4], seq:[0, 1] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=2011292, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=2011282, ALLOC_CHUNK_END=5], seq:[0, 1] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=2010200, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=2010191, ALLOC_CHUNK_END=4], seq:[0, 1] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=2002669, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=2002659, ALLOC_CHUNK_END=5], seq:[0, 1] [717]OB_MALLOC COST TOO MUCH TIME, cost_time=1995026, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1995018, ALLOC_CHUNK_END=4], seq:[0, 1] [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=1994643, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1994634, ALLOC_CHUNK_END=4], seq:[0, 1] [832]OB_MALLOC COST TOO MUCH TIME, cost_time=1994633, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1994625, ALLOC_CHUNK_END=4], seq:[0, 1] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=1737614, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1737605, ALLOC_CHUNK_END=4], seq:[0, 1] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=1058087, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1058077, ALLOC_CHUNK_END=4], seq:[0, 1] [857]OB_MALLOC COST TOO MUCH TIME, cost_time=1056745, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1056736, ALLOC_CHUNK_END=5], seq:[0, 1] [1275]OB_MALLOC COST TOO MUCH TIME, cost_time=1031898, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1031879, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [322]OB_MALLOC COST TOO MUCH TIME, cost_time=1024468, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=17, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1024438, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=1032030, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=20476, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1011527, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [2025-08-14 09:02:58.928244] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [571][T1_ArcSender][T1][YB42AC4D0007-00063C495BA99C16-0-0] [lt=13][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:03:00.718995] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1426644, expected_run_ts_us_=1755162179292349, ObClockGenerator::getRealClock()=1755162180718994, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:59.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:58.927319] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=29][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:03:00.766538] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=14] gc stale ls task succ [2025-08-14 09:02:57.766428] INFO [STORAGE.TRANS] refresh_elr_tenant_config_ (ob_tx_elr_util.cpp:42) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=9] refresh tenant config success(tenant_id=1001, *this={last_refresh_ts:1755162177764832, can_tenant_elr:true}) [2025-08-14 09:03:00.773985] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=3002985] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:00.765557] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=46489][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:59.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:00.774331] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=8728][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:00.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:00.769325] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24431] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:57.768600] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=9] finish to resize log external storage handler(current_ts=1755162176788719, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-08-14 09:03:00.773998] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=6] tx gc loop thread is running(MTL_ID()=1001) [861]OB_MALLOC COST TOO MUCH TIME, cost_time=1011576, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1011551], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:00.780498] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=6488] try gc retain ctx [2025-08-14 09:03:00.780505] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=7] try to retry start_working [2025-08-14 09:03:00.780518] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=5] get_max_applied_scn(scn={val:1755161719775179002, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:45, palf_committed_end_lsn_:{lsn:8461098908}, palf_committed_end_scn_:{val:1755161719775179002, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161719775179002, v:0}}) [2025-08-14 09:03:00.780541] WDIAG [COORDINATOR] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [471][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] cost too much time:ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_(), (this=|threshold=1.00s|start at 09:03:00.780|total=1.95s) [2025-08-14 09:03:00.804812] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:00.792, task_priority:1, with_handle_protected:1}) [861]OB_MALLOC COST TOO MUCH TIME, cost_time=1011588, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1011587], seq:[0, 1] [2025-08-14 09:03:00.819106] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:00.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:01.229396] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=6] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:01.229424] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162181229389], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:01.229445] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162181229389], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:01.229464] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162181229389]}) [2025-08-14 09:03:01.229476] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=40, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:02:58.927245] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=11][errcode=-4018] LOGGER COST TOO MUCH TIME, cost: 2847186, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1835586, ObMalloc#start=1, ObMalloc#end=1011596, ALLOC_END=3, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3b4b 0x113d34aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:01.229487] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162181229389]}) [2025-08-14 09:03:01.229500] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=10] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:01.229513] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162181229499], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:01.229522] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162181229499]}) [2025-08-14 09:03:01.229528] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6] gts statistics(tenant_id=1001, gts_rpc_cnt=0, get_gts_cache_cnt=81, get_gts_with_stc_cnt=3331, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:03:01.229535] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162181229499]}) [2025-08-14 09:03:01.229743] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:03:01.229753] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=9] get number(ret=-4038, service_type_=0, range=1, base_id=1755162181228508000, start_id=0, end_id=140295968341544) [2025-08-14 09:03:01.229763] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:02:55.754677] WDIAG [MDS] init (ob_mds_range_query_iterator.h:165) [805][T1002_TimerWK1_MediumLoop][T1002][YB42AC4D0007-00063C495F7A19FC-0-0] [lt=20][errcode=-4012] fail to init mds sstable iter(ret=-4012, scan_param={tablet_id:{id:60336}, ls_id:{id:1}, column_ids:[16, 17, 18, 19], index_id:88888, key_ranges:[{"range":"table_id:88888,group_idx:0,index_ordered_idx:0,({"TINYINT":3},MIN;{"TINYINT":3},MAX)"}], ss_key_ranges:[], range_array_pos:[], timeout:1755161765724988, scan_flag:{scan_order:0, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_bare_row_scan:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:1, is_select_follower:0, enable_lob_prefetch:0, is_bare_row_scan:0, mr_mv_scan:0, is_plain_insert_gts_opt:0, reserved:0}, sql_mode:134217760, reserved_cell_count:4, schema_version:9527, query_begin_schema_version:-1, limit_offset:{offset_:0, limit_:-1}, for_update:false, wait:-1, frozen_version:9223372036854775807, is_get:false, output_exprs:NULL, op_filters:NULL, trans_desc:NULL, snapshot:{this:0x7f99013508d0, valid:true, source:2, core:{version:{val:4611686018427387903, v:0}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, table_param:{table_id:88888, output_projector:[0, 1, 2, 3], aggregate_projector:[], group_by_projector:[], output_sel_mask:[true, true, true, true], pad_col_projector:[], read_param_version:1, main_read_info:{schema_column_count:4, schema_rowkey_cnt:2, rowkey_cnt:4, trans_col_index:-1, mview_old_new_col_index:-1, group_idx_col_index:-1, seq_read_column_count:2, max_col_index:5, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 4, 5], data:0x7f9845530740, count:4, allocator:0x7f99013516b8, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 3], data:0x7f9845530760, count:4, allocator:0x7f99013516b8, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=16 {type:"TINYINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"CHAR", collation:"binary", coercibility:"INVALID"} order=0, column_id=18 {type:"CHAR", collation:"binary", coercibility:"INVALID"} order=0, column_id=19 {type:"CHAR", collation:"binary", coercibility:"INVALID"} order=0], data:0x7f9845530700, count:4, allocator:0x7f99013516b8, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:4, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}], data:0x7f98455307b0, count:4, allocator:0x7f99013516b8, capacity:4, init_cnt:4}, has_all_column_group:true, need_truncate_filter:false}, use_lob_locator:false, rowid_version:0, rowid_projector:[], enable_lob_locator_v2:false, is_fts_index:false, parser_name:"", parser_properties:"", is_vec_index:false, is_column_replica_table:false, is_normal_cgs_at_the_end:false, is_mlog_table:false}, sample_info:{method:0, percent:"1.000000000000000000e+02", seed:-1, table_id:0, scope:0, force_block:false}, need_scn:false, need_switch_param:false, is_mds_query:true, fb_read_tx_uncommitted:false, external_file_format:{"TYPE":"INVALID"}, external_file_location:"", tx_seq_base:-1, auto_split_filter_type:18446744073709551615, is_tablet_spliting:false, need_update_tablet_param:false, mds_collector:{exist_uncommitted_node:false, exist_new_committed_node:false}}) [2025-08-14 09:03:01.732510] WDIAG [COORDINATOR] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [471][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] cost too much time:ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_failure(), (this=|threshold=1.00s|start at 09:03:01.732|total=2.90s) [2025-08-14 09:03:01.732538] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [471][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:00.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:01.732559] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [471][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=19][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:03:01.732|total=2.90s) [2025-08-14 09:03:00.919173] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:00.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:01.733509] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=813101][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:00.919|207=814.33ms|210=0us|222=2us|total=814.36ms) [2025-08-14 09:03:01.733533] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4389] timer task use too much time(end=1755162181733532, start=1755162180919170, delta=814362) [2025-08-14 09:03:01.056843] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=791629][errcode=0] EASY SLOW: thread: 7f99543f7300, lastrun: 1755162179.745187 cost: 1.311653 loop:26532, slowcnt: 13 [2025-08-14 09:03:01.229777] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162181229738], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:01.733575] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=503031] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162181229738]}) [2025-08-14 09:03:01.733591] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1002}) [2025-08-14 09:03:01.733600] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1002}) [2025-08-14 09:03:01.026110] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:01.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:01.733750] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=706564][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:01.261|207=707.64ms|210=0us|222=2us|total=707.67ms) [2025-08-14 09:03:01.733777] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=-4389] timer task use too much time(end=1755162181733777, start=1755162181026107, delta=707670) [2025-08-14 09:03:01.733793] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=441442, expected_run_ts_us_=1755162181292349, ObClockGenerator::getRealClock()=1755162181733792, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:01.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:01.733812] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:01.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:00.765483] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2826) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=46491] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:1634, [delay/us]: prepare:0, schedule:0, submit:600000000, device:476115, total:476120) [2025-08-14 09:03:01.733928] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=966965] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:03:01.733943] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:03:01.733951] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:03:00.773819] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:01.734120] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=958848][errcode=0] epoll handler too much time :960309 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:00.773835] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=14] gc stale ls task succ [2025-08-14 09:03:00.774006] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=923249] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:03:00.773778] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [564][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=23] XA scheduler heartbeat task statistics(avg_time=1) [2025-08-14 09:02:57.866716] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:01.734567] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=3861980] tx gc loop thread is running(MTL_ID()=1) [2025-08-14 09:03:01.734573] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=6] try gc retain ctx [2025-08-14 09:03:01.734578] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=4] try to retry start_working [2025-08-14 09:03:01.735199] WDIAG pop_task (ob_timer_service.cpp:565) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9931bb3a78, timer:0x7f9966f69f20, task:0x7f99333c4e90, task_type:"N9oceanbase7storage21ObTenantFreezeInfoMgr24UpdateLSResvSnapshotTaskE", scheduled_time:1755161699726703, pushed_time:0, delay:10000000}, running_task_token={this:0x7f9931bb3998, timer:0x7f9966f69f20, task:0x7f99333c4e78, task_type:"N9oceanbase7storage21ObTenantFreezeInfoMgr10ReloadTaskE", scheduled_time:1755161691777945, pushed_time:1755161691782442, delay:3000000}, total_delay=482008494, threashold=1000000, tenant_id=1001) [2025-08-14 09:03:00.774015] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=919254] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:03:01.733856] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=18] [COMPACTION DAG STATUS] (start_time=1755162169738930, end_time=1755162181733855, dag_status={}) [2025-08-14 09:03:01.733779] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] [COMPACTION DAG STATUS] (start_time=1755162169739794, end_time=1755162181733778, dag_status={}) [2025-08-14 09:02:58.788508] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=23] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:01.740821] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:0, time_interval:250.00ms, expected_run_time:09:03:01.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:01.740834] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:03:01.740|total=3.96s) [2025-08-14 09:03:01.740845] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] task executed too delay(delay_time=3932952, *p_task_={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:0, time_interval:500.00ms, expected_run_time:09:03:01.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:58.749830] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741229] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=2986861][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:58.749|207=2.99s|210=0us|222=2us|total=2.99s) [2025-08-14 09:03:01.741251] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task use too much time(end=1755162181741250, start=1755162178749827, delta=2991423) [2025-08-14 09:03:01.741259] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] task scheduled too delay(scheduled_delay=3003821, expected_run_ts_us_=1755162178737437, ObClockGenerator::getRealClock()=1755162181741258, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.737, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741271] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.737, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741283] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=3001767, expected_run_ts_us_=1755162178739515, ObClockGenerator::getRealClock()=1755162181741282, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e71b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.739, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741296] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e71b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.739, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741305] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=3003692, expected_run_ts_us_=1755162178737613, ObClockGenerator::getRealClock()=1755162181741305, ret=0, ret="OB_SUCCESS", *this={this:0x7f98918119b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.737, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741314] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918119b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.737, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741323] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=3000251, expected_run_ts_us_=1755162178741071, ObClockGenerator::getRealClock()=1755162181741323, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72774b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741330] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72774b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741341] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=2992471, expected_run_ts_us_=1755162178748870, ObClockGenerator::getRealClock()=1755162181741341, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fbe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741350] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fbe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741360] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=2982448, expected_run_ts_us_=1755162178758911, ObClockGenerator::getRealClock()=1755162181741360, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741369] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741378] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=2983763, expected_run_ts_us_=1755162178757614, ObClockGenerator::getRealClock()=1755162181741378, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741388] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741397] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=2983563, expected_run_ts_us_=1755162178757833, ObClockGenerator::getRealClock()=1755162181741397, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741406] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741416] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=2974622, expected_run_ts_us_=1755162178766793, ObClockGenerator::getRealClock()=1755162181741416, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919aef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741425] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919aef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741437] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=2978995, expected_run_ts_us_=1755162178762441, ObClockGenerator::getRealClock()=1755162181741436, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7270a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a4190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.762, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741445] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7270a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a4190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.762, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741454] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=2976870, expected_run_ts_us_=1755162178764584, ObClockGenerator::getRealClock()=1755162181741454, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741463] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741474] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=2973422, expected_run_ts_us_=1755162178768051, ObClockGenerator::getRealClock()=1755162181741474, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741485] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741495] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=2977857, expected_run_ts_us_=1755162178763637, ObClockGenerator::getRealClock()=1755162181741494, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741504] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741514] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=2964697, expected_run_ts_us_=1755162178776816, ObClockGenerator::getRealClock()=1755162181741513, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f734de30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891919810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741523] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f734de30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891919810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741526] INFO pn_print_diag_info (group.c:755) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=14] PNIO client:0x7f995f4dd048_"172.77.0.7:49788"_"172.77.0.7:2882"_213_1755161597760925_1, write_queue=0/0, write=542/174814, read=528/120480, doing=0, done=549, write_time=22163148, read_time=32, process_time=25122174 [2025-08-14 09:03:01.741535] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=2968994, expected_run_ts_us_=1755162178772540, ObClockGenerator::getRealClock()=1755162181741535, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741544] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741553] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=2968652, expected_run_ts_us_=1755162178772901, ObClockGenerator::getRealClock()=1755162181741553, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741563] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741573] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=2967753, expected_run_ts_us_=1755162178773819, ObClockGenerator::getRealClock()=1755162181741572, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d59b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.773, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741584] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d59b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.773, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741594] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=2964298, expected_run_ts_us_=1755162178777295, ObClockGenerator::getRealClock()=1755162181741593, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efa10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.777, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741603] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efa10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.777, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741613] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=2952106, expected_run_ts_us_=1755162178789507, ObClockGenerator::getRealClock()=1755162181741613, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eba10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741622] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eba10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741634] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=2952420, expected_run_ts_us_=1755162178789213, ObClockGenerator::getRealClock()=1755162181741634, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741644] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741663] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] task scheduled too delay(scheduled_delay=2945648, expected_run_ts_us_=1755162178796014, ObClockGenerator::getRealClock()=1755162181741663, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741672] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741681] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=2951177, expected_run_ts_us_=1755162178790504, ObClockGenerator::getRealClock()=1755162181741681, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.790, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741692] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.790, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741701] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=2914531, expected_run_ts_us_=1755162178827169, ObClockGenerator::getRealClock()=1755162181741701, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.827, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741710] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.827, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741718] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=2904222, expected_run_ts_us_=1755162178837496, ObClockGenerator::getRealClock()=1755162181741718, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739af30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196bc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.837, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741728] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739af30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196bc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.837, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741737] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=2900442, expected_run_ts_us_=1755162178841295, ObClockGenerator::getRealClock()=1755162181741737, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.841, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741745] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.841, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741754] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=2892439, expected_run_ts_us_=1755162178849314, ObClockGenerator::getRealClock()=1755162181741754, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.849, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741762] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.849, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741772] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=2891957, expected_run_ts_us_=1755162178849815, ObClockGenerator::getRealClock()=1755162181741772, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.849, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741781] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.849, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741789] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=2849517, expected_run_ts_us_=1755162178892272, ObClockGenerator::getRealClock()=1755162181741789, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.892, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741798] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.892, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741807] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=2824101, expected_run_ts_us_=1755162178917706, ObClockGenerator::getRealClock()=1755162181741807, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.917, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741820] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.917, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741829] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=2819060, expected_run_ts_us_=1755162178922769, ObClockGenerator::getRealClock()=1755162181741829, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.922, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741837] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:58.922, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741857] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=2013779, expected_run_ts_us_=1755162179728077, ObClockGenerator::getRealClock()=1755162181741857, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.728, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741865] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.728, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741876] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=2015923, expected_run_ts_us_=1755162179725952, ObClockGenerator::getRealClock()=1755162181741876, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f734c030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891807390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741884] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f734c030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891807390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741894] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=2003302, expected_run_ts_us_=1755162179738591, ObClockGenerator::getRealClock()=1755162181741894, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aae10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741902] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aae10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741911] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=2008159, expected_run_ts_us_=1755162179733752, ObClockGenerator::getRealClock()=1755162181741911, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741921] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741930] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=2007571, expected_run_ts_us_=1755162179734358, ObClockGenerator::getRealClock()=1755162181741929, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e67b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741938] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e67b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741946] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=2008208, expected_run_ts_us_=1755162179733738, ObClockGenerator::getRealClock()=1755162181741946, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741955] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741966] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1995283, expected_run_ts_us_=1755162179746682, ObClockGenerator::getRealClock()=1755162181741966, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919bfe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aa810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741980] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919bfe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aa810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741989] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1999821, expected_run_ts_us_=1755162179742168, ObClockGenerator::getRealClock()=1755162181741989, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.741985] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=23] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25898082304, fast_delete_log_mode=false) [2025-08-14 09:03:01.741997] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742006] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1996113, expected_run_ts_us_=1755162179745893, ObClockGenerator::getRealClock()=1755162181742006, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742025] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742034] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1996167, expected_run_ts_us_=1755162179745867, ObClockGenerator::getRealClock()=1755162181742034, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742043] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742052] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=2001244, expected_run_ts_us_=1755162179740807, ObClockGenerator::getRealClock()=1755162181742051, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742060] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742071] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1986112, expected_run_ts_us_=1755162179755958, ObClockGenerator::getRealClock()=1755162181742070, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742079] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742088] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1962296, expected_run_ts_us_=1755162179779792, ObClockGenerator::getRealClock()=1755162181742088, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72707b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a4010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742097] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72707b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a4010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742106] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1953347, expected_run_ts_us_=1755162179788758, ObClockGenerator::getRealClock()=1755162181742106, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742116] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742125] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1943252, expected_run_ts_us_=1755162179798873, ObClockGenerator::getRealClock()=1755162181742125, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739acb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196bb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.798, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742134] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739acb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196bb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.798, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742142] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1935482, expected_run_ts_us_=1755162179806660, ObClockGenerator::getRealClock()=1755162181742142, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.806, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742151] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.806, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742161] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1932254, expected_run_ts_us_=1755162179809907, ObClockGenerator::getRealClock()=1755162181742161, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.809, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742169] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.809, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742178] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1926449, expected_run_ts_us_=1755162179815728, ObClockGenerator::getRealClock()=1755162181742178, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.815, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742185] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.815, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742194] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1911251, expected_run_ts_us_=1755162179830942, ObClockGenerator::getRealClock()=1755162181742194, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742205] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742214] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1910168, expected_run_ts_us_=1755162179832045, ObClockGenerator::getRealClock()=1755162181742214, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.832, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742223] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.832, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742240] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1901049, expected_run_ts_us_=1755162179841191, ObClockGenerator::getRealClock()=1755162181742240, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.841, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742249] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.841, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742260] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1882788, expected_run_ts_us_=1755162179859472, ObClockGenerator::getRealClock()=1755162181742260, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.859, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742269] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:02:59.859, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742291] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1025347, expected_run_ts_us_=1755162180716943, ObClockGenerator::getRealClock()=1755162181742291, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73516b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196f510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:00.716, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742300] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73516b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196f510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:00.716, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742309] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1021948, expected_run_ts_us_=1755162180720360, ObClockGenerator::getRealClock()=1755162181742309, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:00.720, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742319] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:00.720, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742328] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1014602, expected_run_ts_us_=1755162180727726, ObClockGenerator::getRealClock()=1755162181742328, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:00.727, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742336] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:00.727, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742345] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1015119, expected_run_ts_us_=1755162180727226, ObClockGenerator::getRealClock()=1755162181742345, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:00.727, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742355] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:00.727, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742364] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1021190, expected_run_ts_us_=1755162180721174, ObClockGenerator::getRealClock()=1755162181742364, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72851b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:00.721, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742372] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72851b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:00.721, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742382] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1004581, expected_run_ts_us_=1755162180737800, ObClockGenerator::getRealClock()=1755162181742382, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:00.737, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742390] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:00.737, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742400] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1010925, expected_run_ts_us_=1755162180731475, ObClockGenerator::getRealClock()=1755162181742400, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:00.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742409] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:00.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742418] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=995471, expected_run_ts_us_=1755162180746946, ObClockGenerator::getRealClock()=1755162181742418, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:00.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742427] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:00.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742436] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=993710, expected_run_ts_us_=1755162180748725, ObClockGenerator::getRealClock()=1755162181742436, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891807210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:00.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742447] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891807210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:00.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742456] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=992869, expected_run_ts_us_=1755162180749587, ObClockGenerator::getRealClock()=1755162181742456, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:00.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742465] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:00.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742474] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=992901, expected_run_ts_us_=1755162180749573, ObClockGenerator::getRealClock()=1755162181742474, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:00.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742484] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:00.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742497] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=982853, expected_run_ts_us_=1755162180759643, ObClockGenerator::getRealClock()=1755162181742497, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d67b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:00.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742505] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d67b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:00.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742514] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=959729, expected_run_ts_us_=1755162180782785, ObClockGenerator::getRealClock()=1755162181742514, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891807090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:00.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742523] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891807090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:00.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742531] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=955362, expected_run_ts_us_=1755162180787169, ObClockGenerator::getRealClock()=1755162181742531, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739aa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:00.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742542] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739aa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:00.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742551] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=950944, expected_run_ts_us_=1755162180791606, ObClockGenerator::getRealClock()=1755162181742551, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:00.791, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742560] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:00.791, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742568] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=951195, expected_run_ts_us_=1755162180791373, ObClockGenerator::getRealClock()=1755162181742568, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ecf10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:00.791, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742577] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ecf10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:00.791, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.742588] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:00.773940] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=899000] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:03:01.732722] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] [COMPACTION DAG STATUS] (start_time=1755162170764928, end_time=1755162181732719, dag_status={}) [2025-08-14 09:02:58.897295] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [874][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E899C16-0-0] [lt=861935] ObArchiveSender is running(thread_index=2) [2025-08-14 09:03:01.748716] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:58.746590] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=969180] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:03:01.748808] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=2997655] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623079215142, flush_io_finished_ret_:0, flush_io_finished_round_:26345, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:02:58.760302] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:01.748888] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2984042][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:01.748904] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:01.748915] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:01.733216] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=25] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:03:00.765309] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=501976] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119797572, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755162138745740) [2025-08-14 09:03:01.755285] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:01.741541] INFO pn_print_diag_info (group.c:768) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=13] PNIO server:0x7f995f4dc048_"172.77.0.7:49788"_261_1755161597800497, write_queue=0/0, write=528/120480, read=542/174814, doing=5, done=537, write_time=1099007, read_time=84, process_time=60265223 [2025-08-14 09:03:01.757684] INFO pn_print_diag_info (group.c:775) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=16117] PNIO client_send:70010/27984345, client_queue_time=1246109868, cnt=1, server_send:69991/17176899, server_queue_time=63740453, cnt=1 [2025-08-14 09:03:01.756281] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=3004083] snapshot_traffic_info once [2025-08-14 09:02:58.848171] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=16] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:01.731341] WDIAG [STORAGE.TRANS] runTimerTask (ob_trans_timer.cpp:91) [812][T1002_TransTime][T1002][Y0-0000000000000000-0-0] [lt=1017163][errcode=-4012] handle timeout error(ret=-4012, context={this:0x7f991dadbad0, ref:5, trans_id:{txid:8095834}, tenant_id:1002, is_exiting:false, trans_expired_time:1755248149732313, cluster_version:17180067074, trans_need_wait_wrap:{receive_gts_ts_:[mts=0], need_wait_interval_us:0}, stc:[mts=1755161766747557], ctx_create_time:1755161749732313}) [2025-08-14 09:03:01.772036] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [812][T1002_TransTime][T1002][Y0-0000000000000000-0-0] [lt=40631][errcode=-4389] timer task use too much time(end=1755162181772035, start=1755162019755612, delta=162016423) [2025-08-14 09:03:01.773643] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5116] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:57.841286] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:8607757, max_lsn:1604090331, max_scn:{val:1755161756730847001, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:1556737504}, reuse_lsn:{lsn:1604090331}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:1562147291}}, last_submit_lsn:{lsn:1604090203}, last_submit_end_lsn:{lsn:1604090331}, last_submit_log_id:8607757, last_submit_log_pid:50, max_flushed_lsn:{lsn:1604090203}, max_flushed_end_lsn:{lsn:1604090331}, max_flushed_log_pid:50, committed_end_lsn:{lsn:1604090090}, last_slide_log_id:8607755, last_slide_scn:{val:1755161749741193000, v:0}, last_slide_lsn:{lsn:1604089968}, last_slide_end_lsn:{lsn:1604090090}, last_slide_log_pid:50, last_slide_log_accum_checksum:1446576759, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f990bfe4530}, [StateMgr]={this:0x7f990bfe89b0, self:"172.77.0.7:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"PENDING", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:51}, leader:"0.0.0.0:0", leader_epoch:-1, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:1604090331}, scan_disk_log_finished:true, last_check_start_id:8607756, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161901828039, palf_role_change_cb:0x7f990bfef470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:50, prev:{version:1, config:{config_version:{proposal_id:49, config_seq:45}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:50, config_seq:46}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_mode_pid:50}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_end_lsn:{lsn:1604088821}, prev_mode_pid:50}, reconfig_barrier:{prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_end_lsn:{lsn:1604088821}, prev_mode_pid:50}, persistent_config_version:{proposal_id:50, config_seq:46}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755161738854289, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755161738854289, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f990bfe5c70}, [ModeMgr]={palf_id:1, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:50, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:1543409664}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:51}, log_config_meta:{version:3, proposal_id:50, prev:{version:1, config:{config_version:{proposal_id:49, config_seq:45}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:50, config_seq:46}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_mode_pid:50}, log_snapshot_meta:{version:2, base_lsn:{lsn:1543409664}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:712704}, readable_log_tail:{lsn:712704}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:1610514432}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/meta", dir_fd:225, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66392064, block_header_serialize_buf_:0x7f990bfeabb8, flashback_version:0}, log_storage:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1755161901828095, this:0x7f990bfe90f0}, [Reconfirm]={palf_id:1, self:"172.77.0.7:2882", state:"RECONFIRM_MODE_META", new_proposal_id:51, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"172.77.0.7:2882", majority_max_accept_pid:50, majority_max_lsn:{lsn:1604090331}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:1755161901828373, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f990bfe8b30}) [2025-08-14 09:03:01.613887] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEAB-0-0] [lt=36] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:03:01.774935] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=7] gc stale ls task succ [2025-08-14 09:03:01.756901] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=987321] get wrs ts(ls_id={id:1}, delta=461977885, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:01.774970] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=11] gc stale ls task succ [2025-08-14 09:03:01.774981] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18048] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:02:58.873273] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a7678, timer:0x7f994b1dfb70, task:0x7f9920110450, task_type:"N9oceanbase10rootserver21ObMLogMaintenanceTaskE", scheduled_time:1755161754754755, pushed_time:0, delay:0}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=424118517, threashold=1000000, tenant_id=1002) [2025-08-14 09:03:01.775826] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=2898147][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a7758, timer:0x7f994b1dfb70, task:0x7f9920110530, task_type:"N9oceanbase10rootserver22ObMViewMaintenanceTaskE", scheduled_time:1755161754754764, pushed_time:0, delay:0}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=427021061, threashold=1000000, tenant_id=1002) [2025-08-14 09:03:01.775844] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a7bb8, timer:0x7f994b1dfb70, task:0x7f9920110628, task_type:"N9oceanbase10rootserver34ObMViewRefreshStatsMaintenanceTaskE", scheduled_time:1755161754754765, pushed_time:0, delay:0}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=427021078, threashold=1000000, tenant_id=1002) [2025-08-14 09:03:01.775877] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:01.775899] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=958977] gc stale ls task succ [2025-08-14 09:02:58.887671] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=59][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c0b90, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571c0e70, time_wheel:0x7f9959fef990, thread_pool:0x7f9959fef170, is_running:1, time_interval:3.00s, expected_run_time:09:02:58.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:01.776052] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=2884000][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:58.887|207=2.89s|210=0us|222=1us|total=2.89s) [2025-08-14 09:03:01.776068] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4389] timer task use too much time(end=1755162181776068, start=1755162178887669, delta=2888399) [2025-08-14 09:02:58.903857] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EEC9-0-0] [lt=26] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-08-14 09:03:01.776132] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EEC9-0-0] [lt=2867916][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571fc8b0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcb90, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:0, time_interval:3.00s, expected_run_time:09:03:01.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:01.776147] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EEC9-0-0] [lt=13][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:03:01.776|total=2.88s) [2025-08-14 09:02:58.965887] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=260868][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162178.780311 cost: 0.185574 loop:26522, slowcnt: 1 [2025-08-14 09:02:58.993849] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:58.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:01.776266] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=2778179][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:58.993|207=2.78s|210=0us|222=2us|total=2.78s) [2025-08-14 09:03:01.776277] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task use too much time(end=1755162181776276, start=1755162178993846, delta=2782430) [2025-08-14 09:03:01.776287] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] task scheduled too delay(scheduled_delay=2531653, expected_run_ts_us_=1755162179244633, ObClockGenerator::getRealClock()=1755162181776287, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:59.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:01.776300] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:59.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:59.087262] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=885792][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc4e38, timer:0x7f9970f40f80, task:0x564d6d07c140, task_type:"N9oceanbase7storage25ObRedefTableHeartBeatTaskE", scheduled_time:1755161637759743, pushed_time:0, delay:30000000}, running_task_token={this:0x7f9970fc5fb8, timer:0x7f9970f40f80, task:0x564d727182f0, task_type:"N9oceanbase5share16ObAshRefreshTaskE", scheduled_time:1755161611717613, pushed_time:1755161614757742, delay:120000000}, total_delay=541327517, threashold=1000000, tenant_id=500) [2025-08-14 09:03:01.776450] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72742b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:01.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.776463] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919840b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:01.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.776475] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180ef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:01.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.776484] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180ecb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:01.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.776494] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:01.767, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.776503] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:01.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:59.248908] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:03:01.776567] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=2523814][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:03:01.776579] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:03:01.776588] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:03:01.776597] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:03:01.776608] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:03:01.776618] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:03:01.776633] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:03:01.776646] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:03:01.776664] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:03:01.776672] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:03:01.776682] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:03:01.776696] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:03:01.776846] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:03:01.776856] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:59.640855] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:02:59.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:01.776969] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=2132869][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:59.640|207=2.14s|210=0us|222=2us|total=2.14s) [2025-08-14 09:03:01.776978] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] timer task use too much time(end=1755162181776977, start=1755162179640852, delta=2136125) [2025-08-14 09:02:59.722373] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:01.777036] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2051542][errcode=0] epoll handler too much time :2054680 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:59.722781] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:01.777147] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2051244][errcode=0] epoll handler too much time :2054376 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:02:58.927245] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=11][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:03:00.783638] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [770]OB_MALLOC COST TOO MUCH TIME, cost_time=2916741, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=2916737], seq:[0] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=2916758, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2916756], seq:[0, 1] [2025-08-14 09:02:58.859097] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=57][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:58.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:58.859097] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=57][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2916810, owner: ObLog, click_count: 5, time dist:[FORMAT_END=34, ObMalloc#start=1, ObMalloc#end=2916766, ALLOC_END=1, APPEND_END=7], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:01.780345] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=2916810][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:58.859|207=2.92s|210=1us|222=2us|total=2.92s) [2025-08-14 09:03:01.780374] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4389] timer task use too much time(end=1755162181780374, start=1755162178859094, delta=2921280) [2025-08-14 09:03:01.789259] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919925b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196d890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:01.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.789293] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:01.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.789318] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:01.777, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.799319] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728ca30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7291590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:01.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.806855] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:01.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:01.809372] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=50][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739ee10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:01.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.819431] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ebd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:01.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.819462] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739ec90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:01.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.829494] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72874b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:01.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.834183] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:03:01.834264] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:03:01.834310] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:03:01.834357] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:03:01.834390] INFO [COMMON] print_io_status (ob_io_manager.cpp:2639) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:0, norm_iops:0, bw:1591, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:28258, total:28265) [2025-08-14 09:03:01.839553] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:01.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.839583] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739dc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:01.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.872126] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=2919395] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:01.876141] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c0f70, caller:mds_tenant_service.cpp:mtl_start:117, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_145, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571c1250, time_wheel:0x7f9959fef990, thread_pool:0x7f9959fef170, is_running:1, time_interval:15.00s, expected_run_time:09:03:01.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:01.902411] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EECB-0-0] [lt=7] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-08-14 09:03:01.909941] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a59b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:01.895, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:01.933673] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:01.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:01.981320] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:01.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:01.997902] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:01.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.028947] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:02.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.033736] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:01.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.133771] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:02.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.181410] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:02.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.233829] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=6] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:02.233828] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:02.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.233861] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162182233824], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:02.233880] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162182233824], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:02.233897] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162182233824]}) [2025-08-14 09:03:02.233916] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=12] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:02.233927] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162182233913], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:02.233938] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162182233913]}) [2025-08-14 09:03:02.234124] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:03:02.234137] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:03:02.234152] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162182234118], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:02.234165] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162182234118]}) [2025-08-14 09:03:02.249418] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:02.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.259458] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:02.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.272471] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:02.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.277020] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:03:02.277056] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:03:02.277077] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:03:02.277097] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:03:02.277110] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:03:02.277129] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:03:02.277147] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:03:02.277161] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:03:02.277173] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:03:02.277187] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:03:02.277391] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:03:02.277406] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:03:02.302667] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:02.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.333899] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:02.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.381542] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:02.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.433970] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:02.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.490953] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:02.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.524102] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:02.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.534016] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:02.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.581669] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:02.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.634085] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:02.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.681491] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=4][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571634a0, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957163780, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:5.00s, expected_run_time:09:03:02.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.681528] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:03:02.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.731692] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:02.726, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:02.731740] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739eb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:02.721, task_priority:1, with_handle_protected:0}) [772]OB_MALLOC COST TOO MUCH TIME, cost_time=2976630, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2022220, WASH_BLOCK_END=954407, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:58.744441] EDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=976735][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3982777, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=2, FORMAT_END=1006103, ObMalloc#start=2, ObMalloc#end=2976651, ALLOC_END=2, APPEND_END=13], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ed35c 0x97aead0 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [391]OB_MALLOC COST TOO MUCH TIME, cost_time=2984523, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=2029027, WASH_BLOCK_END=955494], seq:[0, 1, 2] [2025-08-14 09:02:59.745200] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2984626, owner: ObLog, click_count: 7, time dist:[FORMAT_END=23, ObMalloc#start=0, ObMalloc#end=2984538, ObMalloc#start=59, ObMalloc#end=2, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [399]OB_MALLOC COST TOO MUCH TIME, cost_time=2976527, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2021751, WASH_BLOCK_END=954775], seq:[0, 1, 2] [2025-08-14 09:02:59.753504] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2976572, owner: ObLog, click_count: 5, time dist:[FORMAT_END=7, ObMalloc#start=1, ObMalloc#end=2976538, ALLOC_END=23, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [383]OB_MALLOC COST TOO MUCH TIME, cost_time=2976904, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2021729, WASH_BLOCK_END=955175], seq:[0, 1, 2] [2025-08-14 09:02:59.753556] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2976961, owner: ObLog, click_count: 5, time dist:[FORMAT_END=6, ObMalloc#start=0, ObMalloc#end=2976921, ALLOC_END=31, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:01.777051] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:02.732945] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=954447][errcode=0] epoll handler too much time :955896 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:02.732982] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:01.777161] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:02.733034] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=954428][errcode=0] epoll handler too much time :955875 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:02.733052] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:58.744441] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=976735][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:57.766|207=978.21ms|210=0us|222=2us|total=1.99s) [2025-08-14 09:03:02.733265] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=3982777][errcode=-4389] timer task use too much time(end=1755162182733265, start=1755162177766222, delta=4967043) [2025-08-14 09:03:02.733280] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=4977792, expected_run_ts_us_=1755162177755487, ObClockGenerator::getRealClock()=1755162182733279, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:57.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.733312] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:57.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.733333] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=4644016, expected_run_ts_us_=1755162178089316, ObClockGenerator::getRealClock()=1755162182733333, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:58.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.733347] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:58.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.733362] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=4529065, expected_run_ts_us_=1755162178204297, ObClockGenerator::getRealClock()=1755162182733362, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:58.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.733375] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:02:58.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.733402] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=0] task scheduled too delay(scheduled_delay=4012848, expected_run_ts_us_=1755162178720554, ObClockGenerator::getRealClock()=1755162182733402, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc190, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc470, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:10.00s, expected_run_time:09:02:58.720, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.733416] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc190, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc470, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:10.00s, expected_run_time:09:02:58.720, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.733430] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=3974295, expected_run_ts_us_=1755162178759134, ObClockGenerator::getRealClock()=1755162182733430, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:58.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.733574] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=143][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:02:58.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:01.778776] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=993634][errcode=0] epoll handler too much time :995146 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:02.733712] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=953480] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:02.734102] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:02.664, task_priority:1, with_handle_protected:1}) [385]OB_MALLOC COST TOO MUCH TIME, cost_time=3009527, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2044777, WASH_BLOCK_END=964750], seq:[0, 1, 2] [386]OB_MALLOC COST TOO MUCH TIME, cost_time=3002443, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2036806, WASH_BLOCK_END=965637], seq:[0, 1, 2] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=2702032, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1737622, WASH_BLOCK_END=964410], seq:[0, 1, 2] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=2987321, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2021827, WASH_BLOCK_END=965494], seq:[0, 1, 2] [2025-08-14 09:03:00.038251] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=571186][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2702092, owner: ObLog, click_count: 5, time dist:[FORMAT_END=29, ObMalloc#start=2, ObMalloc#end=2702041, ALLOC_END=19, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:58.767311] EDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=940758][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3971138, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=0, FORMAT_END=983731, ObMalloc#start=1, ObMalloc#end=2987381, ALLOC_END=20, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ed35c 0x97aead0 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [321]OB_MALLOC COST TOO MUCH TIME, cost_time=3005715, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2039884, WASH_BLOCK_END=965831], seq:[0, 1, 2] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=2996750, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2030916, WASH_BLOCK_END=965834], seq:[0, 1, 2] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=2987789, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2022020, WASH_BLOCK_END=965769], seq:[0, 1, 2] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=2981075, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2016224, WASH_BLOCK_END=964851], seq:[0, 1, 2] [388]OB_MALLOC COST TOO MUCH TIME, cost_time=2996691, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2031677, WASH_BLOCK_END=965014], seq:[0, 1, 2] [2025-08-14 09:02:59.743509] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2996732, owner: ObLog, click_count: 5, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=2996699, ALLOC_END=21, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [387]OB_MALLOC COST TOO MUCH TIME, cost_time=2986847, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2021755, WASH_BLOCK_END=965092], seq:[0, 1, 2] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=2985349, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2020271, WASH_BLOCK_END=965078], seq:[0, 1, 2] [864]OB_MALLOC COST TOO MUCH TIME, cost_time=2979856, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2014793, WASH_BLOCK_END=965063], seq:[0, 1, 2] [717]OB_MALLOC COST TOO MUCH TIME, cost_time=2960065, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1995033, WASH_BLOCK_END=965032], seq:[0, 1, 2] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=2976422, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2011299, WASH_BLOCK_END=965123], seq:[0, 1, 2] [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=2959771, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1994650, WASH_BLOCK_END=965120], seq:[0, 1, 2] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=2975430, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2010208, WASH_BLOCK_END=965222], seq:[0, 1, 2] [863]OB_MALLOC COST TOO MUCH TIME, cost_time=1009843, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=44923, WASH_BLOCK_END=964920], seq:[0, 1, 2] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=2967965, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2002679, WASH_BLOCK_END=965286], seq:[0, 1, 2] [2025-08-14 09:02:59.772744] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2968015, owner: ObLog, click_count: 5, time dist:[FORMAT_END=8, ObMalloc#start=0, ObMalloc#end=2967974, ALLOC_END=32, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [669]OB_MALLOC COST TOO MUCH TIME, cost_time=1009416, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=44386, WASH_BLOCK_END=965030], seq:[0, 1, 2] [832]OB_MALLOC COST TOO MUCH TIME, cost_time=2960021, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1994640, WASH_BLOCK_END=965381], seq:[0, 1, 2] [497]OB_MALLOC COST TOO MUCH TIME, cost_time=1004723, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=39650, WASH_BLOCK_END=965073], seq:[0, 1, 2] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=2023523, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1058094, WASH_BLOCK_END=965429], seq:[0, 1, 2] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=1004810, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=39680, WASH_BLOCK_END=965129], seq:[0, 1, 2] [857]OB_MALLOC COST TOO MUCH TIME, cost_time=2022234, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1056754, WASH_BLOCK_END=965480], seq:[0, 1, 2] [513]OB_MALLOC COST TOO MUCH TIME, cost_time=1004838, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=39656, WASH_BLOCK_END=965182], seq:[0, 1, 2] [1275]OB_MALLOC COST TOO MUCH TIME, cost_time=1997445, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1031907, WASH_BLOCK_END=965538], seq:[0, 1, 2] [273]OB_MALLOC COST TOO MUCH TIME, cost_time=986397, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=21163, WASH_BLOCK_END=965233], seq:[0, 1, 2] [322]OB_MALLOC COST TOO MUCH TIME, cost_time=1990022, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1024479, WASH_BLOCK_END=965543], seq:[0, 1, 2] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=1997587, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1032039, WASH_BLOCK_END=965548], seq:[0, 1, 2] [2025-08-14 09:03:00.745072] EDIAG [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1997640, owner: ObLog, click_count: 5, time dist:[FORMAT_END=22, ObMalloc#start=1, ObMalloc#end=1997595, ALLOC_END=21, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0xfebfd41 0x94edd90 0x94edb97 0x97a8fbd 0x97a8cc7 0x97a88d2 0x97a7481 0x97a6a43 0x97a42c5 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:01.743101] EDIAG [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1016241, owner: ObLog, click_count: 5, time dist:[FORMAT_END=990141, ObMalloc#start=1, ObMalloc#end=26095, ALLOC_END=3, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xff22be4 0x97a9cd7 0x97a9238 0x97a9307 0x97a8fe2 0x986bc46 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:59.737372] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=150][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3019892, owner: ObLog, click_count: 7, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=3002472, ObMalloc#start=23, ObMalloc#end=17383, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:59.729258] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3028065, owner: ObLog, click_count: 7, time dist:[FORMAT_END=13, ObMalloc#start=1, ObMalloc#end=3009545, ObMalloc#start=34, ObMalloc#end=18471, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [556]OB_MALLOC COST TOO MUCH TIME, cost_time=3021882, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2038390, WASH_BLOCK_END=965809, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=8257], seq:[0, 1, 2, 3, 4] [2025-08-14 09:02:59.735684] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [556][T1_OBJLockGC][T1][Y0-0000000000000000-0-0] [lt=27020][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3021929, owner: ObLog, click_count: 5, time dist:[FORMAT_END=34, ObMalloc#start=1, ObMalloc#end=3021892, ALLOC_END=2, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:59.752293] EDIAG [RS] skip_check_member_list_change_ (ob_ls_recovery_stat_handler.cpp:109) [777][T1002_TeRec][T1002][YB42AC4D0007-00063C495DDA0042-0-0] [lt=30][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3005799, owner: ObLog, click_count: 7, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=2987796, ObMalloc#start=17987, ObMalloc#end=3, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x1067fb2c 0x1067ecc2 0x1067e03d 0x1002f2a3 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:03.769780] EDIAG [PALF] check_parent_health (log_config_mgr.cpp:2875) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=0] LOGGER COST TOO MUCH TIME, cost: 58905726, owner: ObLog, click_count: 3, time dist:[FORMAT_END=58905725, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x9c5fb23 0xfad7c8f 0xfad7980 0xfad7757 0xfa69d59 0xfa69321 0x993f3fc 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:59.745200] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=31] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:02.734361] WDIAG ev_invoke_pending (ev.c:2136) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2984626][errcode=0] epoll handler too much time :2989172 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:02.734379] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:02.734427] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:02.734450] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:02.734443] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162182734422], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:02.734460] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162182734422], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:02.734474] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162182734422]}) [2025-08-14 09:03:02.734486] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:03:02.734497] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:03:02.734510] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=8] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:02.734517] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162182734508], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:02.734528] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162182734508]}) [2025-08-14 09:02:59.753504] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:02.734649] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2976572][errcode=0] epoll handler too much time :2981152 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:02.734679] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:02.734694] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:02.734731] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:03:02.734744] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:03:02.734758] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162182734726], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:02.734779] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162182734726]}) [2025-08-14 09:02:59.753556] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:02.735040] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2976961][errcode=0] epoll handler too much time :2981489 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:02.735058] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:02.735073] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:02.740835] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=2947779] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:02.732467] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [WRS] update tenant weak read server version(tenant_id=1001, server_version={version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162181754126}, version_delta=462957287) [2025-08-14 09:03:02.743399] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:02.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:00.038251] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=571186][errcode=0] EASY SLOW: thread: 7f9954779300, lastrun: 1755162179.722770 cost: 0.315479 loop:26541, slowcnt: 3 [2025-08-14 09:02:58.767311] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=940758][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:57.825|207=942.18ms|210=0us|222=2us|total=1.93s) [2025-08-14 09:03:02.744474] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=3971138][errcode=-4389] timer task use too much time(end=1755162182744473, start=1755162177825125, delta=4919348) [2025-08-14 09:03:02.744487] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=3950551, expected_run_ts_us_=1755162178793934, ObClockGenerator::getRealClock()=1755162182744486, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:58.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.744506] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:02:58.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:59.743509] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:02.744792] WDIAG ev_invoke_pending (ev.c:2136) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2996732][errcode=0] epoll handler too much time :3001294 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:02.744818] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:02.744831] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:59.772744] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:02.745265] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2968015][errcode=0] epoll handler too much time :2972529 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:02.745280] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:02.745291] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:00.745072] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get_max_applied_scn(scn={val:1755161547738516001, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:50, palf_committed_end_lsn_:{lsn:5282737655}, palf_committed_end_scn_:{val:1755161547738516001, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161547738516001, v:0}}) [2025-08-14 09:03:02.745766] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1997640] get_min_unreplayed_log_info(lsn={lsn:5282737655}, scn={val:1755100947825033001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161624848121, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:13063726, ret_code_:0}}, next_to_submit_lsn_:{lsn:5282737655}, next_to_submit_scn_:{val:1755100947825033001, v:0}, base_lsn_:{lsn:4831543296}, base_scn_:{val:1755063445733337001, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7605000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:5282737655}, end_lsn:{lsn:5282737655}, read_buf:{buf_len_:3747840, buf_:0x7f98f7605000}, block_size:67104768, log_storage_:0x7f993e529bf0, log_storage_:{log_tail:{lsn:5282737655}, readable_log_tail:{lsn:5282737655}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:5234171904}, min_scn:{val:1755152244802535004, v:0}, curr_block_id:78, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:206, min_block_id:42, max_block_id:79, curr_writable_block_id:78}, logical_block_size_:67104768, curr_block_writable_size_:18539017, block_header_serialize_buf_:0x7f993e52a8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:85, read_io_size:301367296, read_disk_cost_ts:2364604, start_lsn:{lsn:4831543296}}}}}}) [2025-08-14 09:03:02.746410] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=68] get wrs ts(ls_id={id:1}, delta=635004797, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:02.746427] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=3, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:02.746611] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:01.743101] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=14] get_min_unreplayed_log_info(lsn={lsn:8461098908}, scn={val:1755100947600130001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161920778323, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:23084275, ret_code_:0}}, next_to_submit_lsn_:{lsn:8461098908}, next_to_submit_scn_:{val:1755100947600130001, v:0}, base_lsn_:{lsn:7515734016}, base_scn_:{val:1753840867171901045, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6e05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:8461098908}, end_lsn:{lsn:8461098908}, read_buf:{buf_len_:3747840, buf_:0x7f98f6e05000}, block_size:67104768, log_storage_:0x7f99297a3bf0, log_storage_:{log_tail:{lsn:8461098908}, readable_log_tail:{lsn:8461098908}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8455200768}, min_scn:{val:1755160384851871002, v:0}, curr_block_id:126, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:215, min_block_id:93, max_block_id:127, curr_writable_block_id:126}, logical_block_size_:67104768, curr_block_writable_size_:61206628, block_header_serialize_buf_:0x7f99297a48b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:198, read_io_size:699822080, read_disk_cost_ts:4178035, start_lsn:{lsn:7515734016}}}}}}) [2025-08-14 09:02:59.737372] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=150] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:02.761858] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3019892][errcode=0] epoll handler too much time :3024497 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:02.761876] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:02.761889] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:59.729258] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:02.761921] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3028065][errcode=0] epoll handler too much time :3032674 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:02.761934] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:02.761944] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:59.735684] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [556][T1_OBJLockGC][T1][Y0-0000000000000000-0-0] [lt=27020][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fafd0, caller:ob_table_lock_service.cpp:start:276, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE6$_1273, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fb2b0, time_wheel:0x7f994b19db50, thread_pool:0x7f994b19d330, is_running:1, time_interval:10.00s, expected_run_time:09:02:59.696, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.762207] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [556][T1_OBJLockGC][T1][Y0-0000000000000000-0-0] [lt=3021929][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:59.735|207=3.03s|210=0us|222=2us|total=3.03s) [2025-08-14 09:03:02.762240] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [556][T1_OBJLockGC][T1][Y0-0000000000000000-0-0] [lt=33][errcode=-4389] timer task use too much time(end=1755162182762240, start=1755162179735680, delta=3026560) [2025-08-14 09:03:01.834420] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, param_config_:{memory_limit:536870912, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=865568, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:03:02.762337] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2826) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=926508] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:2051, [delay/us]: prepare:0, schedule:0, submit:600000000, device:28258, total:28265) [2025-08-14 09:03:02.762359] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:03:02.762370] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:03:02.762379] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:03:02.732567] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [280][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:33, tid:660}]) [2025-08-14 09:03:02.758633] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10815] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=462968123, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=252317, succ_cluster_heartbeat_count=1905, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161003593738002, v:0}, local_cluster_delta=1179149564, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:03:02.762800] INFO [SHARE.LOCATION] refresh_cache_ (ob_tablet_location_refresh_service.cpp:609) [365][TbltRefreshSer][T0][YB42AC4D0007-00063C4960B99C42-0-0] [lt=13570] [REFRESH_TABLET_LOCATION] refresh cache start [2025-08-14 09:03:02.764418] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:02.750, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:02.764451] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:02.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:02.764463] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d80b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:02.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:02.764473] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:02.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:02.764484] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891992330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196d710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:02.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:02.764500] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:02.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:02:03.769780] INFO [PALF] check_parent_health (log_config_mgr.cpp:2875) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=20] re_register_parent reason(palf_id=1, self="172.77.0.7:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1755162119724899, last_first_register_time_us=1755158728414267, register_time_us=1755162119724899, curr_time_us=1755162123769779) [2025-08-14 09:03:02.074893] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [402][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=1222737][errcode=0] EASY SLOW: thread: 7f9953cfb300, lastrun: 1755162179.759095 cost: 2.315796 loop:26506, slowcnt: 23 [2025-08-14 09:03:02.760813] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=635012538, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=253171, succ_cluster_heartbeat_count=2211, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161000606550000, v:0}, local_cluster_delta=1182144504, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:03:02.768270] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:02.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.768316] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:02.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.770487] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:02.772894] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:02.775977] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=6] gc stale ls task succ [2025-08-14 09:03:02.776979] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:02.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.777274] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=13] gc stale ls task succ [2025-08-14 09:03:02.777320] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=7] gc stale ls task succ [2025-08-14 09:03:02.777350] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:02.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:02.777385] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:02.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:02.777585] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:03:02.777615] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:03:02.777629] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:03:02.777646] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:03:02.777667] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:03:02.777686] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:03:02.777711] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:03:02.777724] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:03:02.777736] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:03:02.777750] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:03:02.777916] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:03:02.777931] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:03:02.779523] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:02.779553] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:02.779579] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:02.781755] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:02.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.790343] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:02.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:02.790369] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72751b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:02.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:02.790382] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d31b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:02.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:02.790394] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73959b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:02.773, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:02.790405] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:02.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:02.790418] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72856b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:02.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:02.790429] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:02.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:02.802284] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919920b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196d590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:02.790, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:02.807184] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:02.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.844550] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:02.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.852597] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:02.840, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:02.852631] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d54b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ecd90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:02.846, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:02.852646] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ecc10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:02.848, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:02.866278] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:03:02.866335] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:03:02.866388] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:03:02.866441] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:03:02.866484] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:03:02.866509] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=1, sync_total=0, async_cnt= 1 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:03:02.866524] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:03:02.876672] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=17] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:02.944609] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:02.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:02.981882] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:02.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.019763] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:03.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.019798] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:03.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.028801] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:03.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.044678] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:02.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.090244] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:03.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.135393] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=675692][errcode=0] EASY SLOW: thread: 7f995407d300, lastrun: 1755162182.756359 cost: 0.379031 loop:26555, slowcnt: 3 [2025-08-14 09:03:03.144698] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:03.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.181999] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:03.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.210955] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:03.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.235004] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:03.235030] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162183234998], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:03.235060] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162183234998], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:03.235079] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162183234998]}) [2025-08-14 09:03:03.235097] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=13] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:03.235113] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162183235095], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:03.235133] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162183235095]}) [2025-08-14 09:03:03.235349] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:03:03.235363] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:03:03.235379] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162183235334], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:03.235398] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162183235334]}) [2025-08-14 09:03:03.244758] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:03.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.261259] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:03.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.261294] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:03.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.270943] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:03.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.278109] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:03:03.278146] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:03:03.278159] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:03:03.278178] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:03:03.278198] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:03:03.278218] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:03:03.278250] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:03:03.278264] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:03:03.278277] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:03:03.278291] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:03:03.278498] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:03:03.278520] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:03:03.301103] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:03.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.344786] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:03.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.382096] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:03.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.444854] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:03.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.512962] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:03.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.512997] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:03.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.522446] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:03.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.544897] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:03.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.582221] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:03.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.593450] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:03.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.644967] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:03.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.683670] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:03:03.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.714279] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:03.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:59.759100] EDIAG [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3952623, owner: ObLog, click_count: 6, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=2981087, ObMalloc#start=22, ObMalloc#end=8151, FREE_END=963350], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a1f867 0x27a135c3 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:59.755023] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [663][T1001_StandbySc][T1001][YB42AC4D0007-00063C495C7A0326-0-0] [lt=24][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3956702, owner: ObLog, click_count: 6, time dist:[FORMAT_END=13, ObMalloc#start=1, ObMalloc#end=2985358, ObMalloc#start=13751, ObMalloc#end=83, FREE_END=957496], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:03.717623] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=965563][errcode=0] epoll handler too much time :967036 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:03.717752] WDIAG ev_invoke_pending (ev.c:2136) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=3952623][errcode=0] epoll handler too much time :3958657 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:03.717750] WDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=959924][errcode=0] epoll handler too much time :961389 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:03.717770] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=11] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:03.717784] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=10] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:59.765171] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3951117, owner: ObLog, click_count: 6, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=2975441, ObMalloc#start=19, ObMalloc#end=7895, FREE_END=967749], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:00.720194] EDIAG [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2997611, owner: ObLog, click_count: 6, time dist:[FORMAT_END=22, ObMalloc#start=1, ObMalloc#end=2022243, ObMalloc#start=21, ObMalloc#end=7635, FREE_END=967689], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0xfebfd41 0x94edd90 0x94edb97 0x97a8fbd 0x97a8cc7 0x97a88d2 0x97a7481 0x97a6a43 0x97a42c5 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:59.780819] EDIAG [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3935982, owner: ObLog, click_count: 6, time dist:[FORMAT_END=8, ObMalloc#start=0, ObMalloc#end=2959779, ObMalloc#start=20, ObMalloc#end=7914, FREE_END=968261], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1f832f91 0x97c36fe 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:59.760529] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=18057][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3956630, owner: ObLog, click_count: 6, time dist:[FORMAT_END=22, ObMalloc#start=7, ObMalloc#end=2979864, ObMalloc#start=18, ObMalloc#end=7992, FREE_END=968727], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:56.931537] EDIAG [RS] get_valid_sts_after (ob_tenant_info_loader.cpp:454) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=0] LOGGER COST TOO MUCH TIME, cost: 6781383, owner: ObLog, click_count: 6, time dist:[FORMAT_END=3781588, ObMalloc#start=2, ObMalloc#end=2023531, ObMalloc#start=20, ObMalloc#end=7661, FREE_END=968581], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1001d942 0x996d843 0x996d15a 0x996cfd8 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:59.780403] EDIAG [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3937059, owner: ObLog, click_count: 6, time dist:[FORMAT_END=14, ObMalloc#start=1, ObMalloc#end=2960079, ObMalloc#start=18, ObMalloc#end=7970, FREE_END=968977], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1d189030 0x1d18285e 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:59.734258] EDIAG eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3984910, owner: ObLog, click_count: 6, time dist:[FORMAT_END=18, ObMalloc#start=1, ObMalloc#end=3005723, ObMalloc#start=18, ObMalloc#end=8264, FREE_END=970886], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x27a8a8c8 0x97b7e1a 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:57.891603] EDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 5824857, owner: ObLog, click_count: 8, time dist:[FORMAT_END=3841780, ObMalloc#start=1, ObMalloc#end=1004845, ObMalloc#start=33, ObMalloc#end=7607, ObMalloc#start=366, ObMalloc#end=37, FREE_END=970188], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x9757ced 0x97569df 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:58.733508] EDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=978361][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4984319, owner: ObLog, click_count: 8, time dist:[FORMAT_END=3001091, ObMalloc#start=2, ObMalloc#end=1004818, ObMalloc#start=19, ObMalloc#end=7652, ObMalloc#start=210, ObMalloc#end=49, FREE_END=970477], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:57.760838] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=44][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5955585, owner: ObLog, click_count: 10, time dist:[ALLOC_BLOCK#start=6, ALLOC_BLOCK#end=3, FORMAT_END=3972286, ObMalloc#start=1, ObMalloc#end=1004738, ObMalloc#start=19, ObMalloc#end=7678, ObMalloc#start=422, ObMalloc#end=39, FREE_END=970392], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:59.764064] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3955452, owner: ObLog, click_count: 6, time dist:[FORMAT_END=15, ObMalloc#start=1, ObMalloc#end=2976429, ObMalloc#start=50, ObMalloc#end=7930, FREE_END=971027], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:00.741760] EDIAG pn_print_diag_info (group.c:775) [322][pnio2][T0][Y0-0000000000000000-0-0] [lt=27519][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2979271, owner: ObLog, click_count: 6, time dist:[FORMAT_END=10838, ObMalloc#start=1, ObMalloc#end=1990039, ObMalloc#start=27, ObMalloc#end=7547, FREE_END=970819], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x27a90dc3 0x27a8aaf2 0x97b7c8d 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:59.753476] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3966114, owner: ObLog, click_count: 6, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=2986863, ObMalloc#start=21, ObMalloc#end=8023, FREE_END=971195], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:59.780840] EDIAG [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3938871, owner: ObLog, click_count: 6, time dist:[FORMAT_END=7, ObMalloc#start=0, ObMalloc#end=2960029, ObMalloc#start=19, ObMalloc#end=7698, FREE_END=971118], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1d189030 0x1d18285e 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:58.761118] EDIAG [RS.COMPACTION] try_idle (ob_major_merge_scheduler.cpp:226) [1275][T1001_MergeSche][T1001][YB42AC4D0007-00063C4963E99CBE-0-0] [lt=21][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4957522, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1981023, ObMalloc#start=1, ObMalloc#end=1997453, ObMalloc#start=6877, ObMalloc#end=859, FREE_END=971309], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1125d502 0x1125cf81 0x1125a4c3 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:59.743279] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3976888, owner: ObLog, click_count: 6, time dist:[FORMAT_END=13, ObMalloc#start=1, ObMalloc#end=2996758, ObMalloc#start=19, ObMalloc#end=8256, FREE_END=971841], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:03.722372] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=2997611] get_min_unreplayed_log_info(lsn={lsn:1604088821}, scn={val:1755100948296325001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873479, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:12492179, ret_code_:0}}, next_to_submit_lsn_:{lsn:1604088821}, next_to_submit_scn_:{val:1755100948296325001, v:0}, base_lsn_:{lsn:1409200128}, base_scn_:{val:1753840802540914000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:1604088821}, end_lsn:{lsn:1604088821}, read_buf:{buf_len_:3747840, buf_:0x7f98f6a05000}, block_size:67104768, log_storage_:0x7f990bfebbf0, log_storage_:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:42, read_io_size:147709952, read_disk_cost_ts:1061450, start_lsn:{lsn:1409200128}}}}}}) [2025-08-14 09:03:03.722462] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=83] get wrs ts(ls_id={id:1}, delta=433980064, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:03.722480] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:03.723848] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4192] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161101770923, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755161572801635) [2025-08-14 09:03:03.724389] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=635982741, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:03.724403] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:03.725021] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=635982741, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:03.725036] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:03.725084] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=977048][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:02.746|207=978.53ms|210=0us|222=2us|total=978.56ms) [2025-08-14 09:03:03.725116] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=31][errcode=-4389] timer task use too much time(end=1755162183725115, start=1755162182746549, delta=978566) [2025-08-14 09:03:03.725130] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=980496, expected_run_ts_us_=1755162182744633, ObClockGenerator::getRealClock()=1755162183725130, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:02.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.725149] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:02.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.725299] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=5824857][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a64f8, timer:0x7f9966fddee0, task:0x7f994f1b1510, task_type:"N9oceanbase5table20ObTableObjectPoolMgr28ObTableReqResEliminationTaskE", scheduled_time:1755161695734026, pushed_time:0, delay:5000000}, running_task_token={this:0x7f994e1a65d8, timer:0x7f9966fddee0, task:0x7f994f1bbcc0, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161694824965, pushed_time:1755161925772608, delay:5000000}, total_delay=487991273, threashold=1000000, tenant_id=1) [2025-08-14 09:03:03.725323] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a6418, timer:0x7f9966fddee0, task:0x7f994f1b14f8, task_type:"N9oceanbase5table20ObTableObjectPoolMgr27ObTableSessSysVarUpdateTaskE", scheduled_time:1755161695734641, pushed_time:0, delay:5000000}, running_task_token={this:0x7f994e1a65d8, timer:0x7f9966fddee0, task:0x7f994f1bbcc0, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161694824965, pushed_time:1755161925772608, delay:5000000}, total_delay=487990681, threashold=1000000, tenant_id=1) [2025-08-14 09:03:03.725335] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a4f18, timer:0x7f9966fddee0, task:0x7f994b3f6b30, task_type:"N9oceanbase3sql3dtl23ObDTLIntermResultGCTaskE", scheduled_time:1755161700757633, pushed_time:0, delay:10000000}, running_task_token={this:0x7f994e1a65d8, timer:0x7f9966fddee0, task:0x7f994f1bbcc0, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161694824965, pushed_time:1755161925772608, delay:5000000}, total_delay=482967702, threashold=1000000, tenant_id=1) [2025-08-14 09:03:03.725396] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4984319] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:03.725412] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:03.725452] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=5955585][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:57.760|207=5.96s|210=1us|222=1us|total=5.96s) [2025-08-14 09:03:03.725467] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] timer task use too much time(end=1755162183725466, start=1755162177760836, delta=5964630) [2025-08-14 09:03:03.725609] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3966114][errcode=0] epoll handler too much time :3972144 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:03.725617] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=3928] get wrs ts(ls_id={id:1}, delta=463946078, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:03.725627] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:03.725641] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:03.725633] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:03.726203] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3976888][errcode=0] epoll handler too much time :3982935 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:03.726207] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=463946078, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:03.726221] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:03.726220] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:03.726245] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:03.721143] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=2947002] blacklist_loop exec finished(cost_time=19, is_enabled=true, send_cnt=0) [2025-08-14 09:03:03.727540] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=3984910] PNIO [ratelimit] time: 1755162183727538, bytes: 0, bw: 0.000000 MB/s, add_ts: 3993282, add_bytes: 0 [2025-08-14 09:03:03.730036] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:03.722, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:03.736259] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:03.737247] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:03.737273] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162183737243], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:03.737297] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162183737243], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:03.737311] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162183737243]}) [2025-08-14 09:03:03.737328] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:03:03.737343] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:03:03.737356] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:03.737370] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162183737354], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:03.737395] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162183737354]}) [2025-08-14 09:03:03.737585] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:03:03.737605] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:03:03.737621] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162183737581], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:03.737640] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162183737581]}) [2025-08-14 09:03:03.737653] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162183737581]}) [2025-08-14 09:03:02.733765] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [723][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=19] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=16, total_destructed_detector=16, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f99333562b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-08-14 09:03:02.733765] EDIAG [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [723][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1002633, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1002621, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=8], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x973b640 0x973b132 0x973a32d 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:03.737949] WDIAG [DETECT] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [723][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=1002633][errcode=0] cost too much time:ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t), (this=|threshold=100.00ms|start at 09:03:03.737|total=1.93s) [2025-08-14 09:03:03.740303] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:03.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:03.740339] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891826cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891918190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:03.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:03.741172] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=3927461] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161974728707, last_flushed_end_lsn:{lsn:1604090331}}]) [2025-08-14 09:03:03.741217] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=37] current log_task status(palf_id=1, self="172.77.0.7:2882", log_id=8607756, log_task={header:{begin_lsn:{lsn:1604090090}, end_lsn:{lsn:1604090203}, log_id:8607756, min_scn:{val:1755161749741193001, v:0}, max_scn:{val:1755161749741193001, v:0}, data_len:57, proposal_id:50, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:1604090090}, data_checksum:2031283194, accum_checksum:759186326, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1755161750716165, freeze_ts:1755161750716173, submit_ts:1755161750716183, flushed_ts:1755161853765770, gen_to_freeze cost time:8, gen_to_submit cost time:18, submit_to_flush cost time:103049587}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161974728707, last_flushed_end_lsn:{lsn:1604090331}}]) [2025-08-14 09:03:03.742079] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=23] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:03.745245] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:03.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.745326] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:01.740855] EDIAG [ELECT] operator() (election_proposer.cpp:232) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2001881, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2001873, ObMalloc#start=0, ObMalloc#end=2, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xfa07f77 0xfa07a14 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:03.745748] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=36] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:03:03.745787] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=2001881][errcode=-4389] run this task use too much time(*p_task_={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:0, time_interval:500.00ms, expected_run_time:09:03:03.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.745808] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:03:03.745|total=2.00s) [2025-08-14 09:03:03.745824] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] task executed too delay(delay_time=4934215, *p_task_={this:0x7f996f40b7f0, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40bad0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:0, time_interval:10.00s, expected_run_time:09:03:08.802, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.746186] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:03.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.746260] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:03.746295] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:54.766632] WDIAG [SQL.DAS] execute_dist_das_task (ob_data_access_service.cpp:177) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=9141][errcode=-4038] do local das task failed(ret=-4038, task_arg={timeout_ts:1755162198753066, ctrl_svr:"172.77.0.7:2882", runner_svr:"172.77.0.7:2882", task_ops:[{parent:{tenant_id:1, task_id:50056313, op_type:2, errcode:-4038, can_part_retry:0, task_started:1, in_part_retry:0, in_stmt_retry:0, need_switch_param:0, trans_desc:{this:0x7f98c907e020, tx_id:{txid:21195872}, state:1, addr:"172.77.0.7:2882", tenant_id:1, session_id:1, assoc_session_id:1, client_sid:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1755162170771423, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999987, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1753069619, cluster_version:17180067074, seq_base:1755162170724889, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:2}, snapshot:{this:0x7f9956cd2798, valid:true, source:5, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:21195872}, scn:{branch:0, seq:2}}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, tablet_id:{id:154}, ls_id:{id:1}, tablet_loc:{tablet_id:{id:154}, ls_id:{id:1}, server:"172.77.0.7:2882", loc_meta:{table_loc_id:18446744073709551614, ref_table_id:154, related_table_ids:[101015, 101016], use_dist_das:0, select_leader:1, is_dup_table:0, is_weak_read:0, unuse_related_pruning:0, is_external_table:0, route_policy:0}, in_retry:0, partition_id:18446744073709551615, first_level_part_id:18446744073709551615}, related_ctdefs:[{ObDASBaseCtDef:{ObDASBaseDef:{op_type:2, children_cnt:0}, table_id:18446744073709551614, index_tid:101015, schema_version:1753069637599528, rowkey_cnt:4, spk_cnt:0, column_ids:[19, 16, 17, 18], column_types:[{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}], column_accuracys:[{length:64, precision:-1, scale:-1}, {length:-1, precision:25, scale:6}, {length:46, precision:-1, scale:-1}, {length:-1, precision:20, scale:0}], old_row_projector:[], new_row_projector:[3, 0, 1, 2], is_total_quantity_log:1, is_ignore:0, is_batch_stmt:0, is_insert_up:0, is_table_api:0, is_main_table_in_fts_ddl:0, tz_info:{id:-1, offset:28800, error_on_overlap_time:false}, table_param:{tenant_schema_version:1753073979411448, data_table:{table_id:101015, schema_version:1753069637599528, table_type:5, index_type:1, index_status:2, shadow_rowkey_column_num:0, doc_id_col_id:18446744073709551615, fulltext_col_id:18446744073709551615, index_name:"idx_server_module", fts_parser_name:"", fts_parser_properties:"", pk_name:"", columns:[{column_id:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:64, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:25, scale:6}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"EXT":"CURRENT_TIMESTAMP"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}, {column_id:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:46, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}, {column_id:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}], read_info:{schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, trans_col_index:-1, mview_old_new_col_index:-1, group_idx_col_index:-1, seq_read_column_count:4, max_col_index:3, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 2, 3], data:0x7f9827a3ae60, count:4, allocator:0x7f9827a1e6f0, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 3], data:0x7f9827a3ae80, count:4, allocator:0x7f9827a1e6f0, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9827a3ae20, count:4, allocator:0x7f9827a1e6f0, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, has_all_column_group:true, need_truncate_filter:false}, lob_inrow_threshold:4096}, col_descs:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], col_map:{is_inited_:true, use_array_:true, shadow_use_array_:false, has_:true, has_shadow_:false}}, encrypt_meta:[]}, table_rowkey_cids:[], table_rowkey_types:[]}, {ObDASBaseCtDef:{ObDASBaseDef:{op_type:2, children_cnt:0}, table_id:18446744073709551614, index_tid:101016, schema_version:1753069637600200, rowkey_cnt:4, spk_cnt:0, column_ids:[20, 16, 17, 18], column_types:[{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}], column_accuracys:[{length:64, precision:-1, scale:-1}, {length:-1, precision:25, scale:6}, {length:46, precision:-1, scale:-1}, {length:-1, precision:20, scale:0}], old_row_projector:[], new_row_projector:[4, 0, 1, 2], is_total_quantity_log:1, is_ignore:0, is_batch_stmt:0, is_insert_up:0, is_table_api:0, is_main_table_in_fts_ddl:0, tz_info:{id:-1, offset:28800, error_on_overlap_time:false}, table_param:{tenant_schema_version:1753073979411448, data_table:{table_id:101016, schema_version:1753069637600200, table_type:5, index_type:1, index_status:2, shadow_rowkey_column_num:0, doc_id_col_id:18446744073709551615, fulltext_col_id:18446744073709551615, index_name:"idx_server_event", fts_parser_name:"", fts_parser_properties:"", pk_name:"", columns:[{column_id:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:64, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:25, scale:6}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"EXT":"CURRENT_TIMESTAMP"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}, {column_id:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:46, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}, {column_id:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}], read_info:{schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, trans_col_index:-1, mview_old_new_col_index:-1, group_idx_col_index:-1, seq_read_column_count:4, max_col_index:3, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 2, 3], data:0x7f9827a3baf0, count:4, allocator:0x7f9827a1e6f0, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 3], data:0x7f9827a3bb10, count:4, allocator:0x7f9827a1e6f0, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9827a3bab0, count:4, allocator:0x7f9827a1e6f0, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, has_all_column_group:true, need_truncate_filter:false}, lob_inrow_threshold:4096}, col_descs:[column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], col_map:{is_inited_:true, use_array_:true, shadow_use_array_:false, has_:true, has_shadow_:false}}, encrypt_meta:[]}, table_rowkey_cids:[], table_rowkey_types:[]}], related_rtdefs:[{ObDASBaseRtDef:{ObDASBaseRtDef:{op_type:2, children_cnt:0}, timeout_ts:1755162198653066, sql_mode:4194304, prelock:false, tenant_schema_version:1753073979411448, is_for_foreign_key_check:false, affected_rows:0, is_immediate_row_conflict_check:true}, need_fetch_conflict:false, is_duplicated:false, direct_insert_task_id:0, use_put:false, ddl_task_id:0}, {ObDASBaseRtDef:{ObDASBaseRtDef:{op_type:2, children_cnt:0}, timeout_ts:1755162198653066, sql_mode:4194304, prelock:false, tenant_schema_version:1753073979411448, is_for_foreign_key_check:false, affected_rows:0, is_immediate_row_conflict_check:true}, need_fetch_conflict:false, is_duplicated:false, direct_insert_task_id:0, use_put:false, ddl_task_id:0}], task_status:1, related_tablet_ids:[{id:101015}, {id:101016}], das_task_node:, plan_line_id:0}, ins_ctdef:{ObDASBaseCtDef:{ObDASBaseDef:{op_type:2, children_cnt:0}, table_id:18446744073709551614, index_tid:154, schema_version:1753069637597680, rowkey_cnt:3, spk_cnt:0, column_ids:[16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33], column_types:[{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"LONGTEXT", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}], column_accuracys:[{length:-1, precision:25, scale:6}, {length:46, precision:-1, scale:-1}, {length:-1, precision:20, scale:0}, {length:64, precision:-1, scale:-1}, {length:64, precision:-1, scale:-1}, {length:256, precision:-1, scale:-1}, {length:512, precision:-1, scale:-1}, {length:256, precision:-1, scale:-1}, {length:536870911, precision:-1, scale:-1}, {length:256, precision:-1, scale:-1}, {length:512, precision:-1, scale:-1}, {length:256, precision:-1, scale:-1}, {length:512, precision:-1, scale:-1}, {length:256, precision:-1, scale:-1}, {length:512, precision:-1, scale:-1}, {length:256, precision:-1, scale:-1}, {length:512, precision:-1, scale:-1}, {length:512, precision:-1, scale:-1}], old_row_projector:[], new_row_projector:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17], is_total_quantity_log:1, is_ignore:0, is_batch_stmt:0, is_insert_up:0, is_table_api:0, is_main_table_in_fts_ddl:0, tz_info:{id:-1, offset:28800, error_on_overlap_time:false}, table_param:{tenant_schema_version:1753073979411448, data_table:{table_id:154, schema_version:1753069637597680, table_type:0, index_type:0, index_status:0, shadow_rowkey_column_num:0, doc_id_col_id:18446744073709551615, fulltext_col_id:18446744073709551615, index_name:"", fts_parser_name:"", fts_parser_properties:"", pk_name:"", columns:[{column_id:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:25, scale:6}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"EXT":"CURRENT_TIMESTAMP"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}, {column_id:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:46, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}, {column_id:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}, {column_id:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:64, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:64, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:256, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:512, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:256, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:24, meta_type:{type:"LONGTEXT", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:536870911, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:25, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:256, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:26, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:512, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:27, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:256, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:28, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:512, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:29, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:256, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:30, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:512, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:31, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:256, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:32, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:512, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:33, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:512, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}], read_info:{schema_column_count:18, schema_rowkey_cnt:3, rowkey_cnt:5, trans_col_index:-1, mview_old_new_col_index:-1, group_idx_col_index:-1, seq_read_column_count:3, max_col_index:19, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:18, 0, 1, 2, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19], data:0x7f9827a39e20, count:18, allocator:0x7f9827a1e6f0, capacity:18, init_cnt:18}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:18, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17], data:0x7f9827a39e70, count:18, allocator:0x7f9827a1e6f0, capacity:18, init_cnt:18}}, cols_desc:{ObIArray:[cnt:18, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=23 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=24 {type:"LONGTEXT", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=25 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=26 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=27 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=28 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=29 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=30 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=31 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=32 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=33 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0], data:0x7f9827a39d40, count:18, allocator:0x7f9827a1e6f0, capacity:18, init_cnt:18}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, has_all_column_group:true, need_truncate_filter:false}, lob_inrow_threshold:4096}, col_descs:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=23 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=24 {type:"LONGTEXT", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=25 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=26 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=27 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=28 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=29 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=30 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=31 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=32 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=33 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0], col_map:{is_inited_:true, use_array_:true, shadow_use_array_:false, has_:true, has_shadow_:false}}, encrypt_meta:[]}, table_rowkey_cids:[], table_rowkey_types:[]}, ins_rtdef:{ObDASBaseRtDef:{ObDASBaseRtDef:{op_type:2, children_cnt:0}, timeout_ts:1755162198653066, sql_mode:4194304, prelock:false, tenant_schema_version:1753073979411448, is_for_foreign_key_check:false, affected_rows:0, is_immediate_row_conflict_check:true}, need_fetch_conflict:false, is_duplicated:false, direct_insert_task_id:0, use_put:false, ddl_task_id:0}, insert_buffer:{mem_attr:tenant_id=1, label=DASInsertBuffer, ctx_id=0, prio=0, buffer_memory:427, buffer_size:1, datum_store:NULL}}], remote_info:NULL}) [2025-08-14 09:03:03.749772] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=65] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:03.750310] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:03.750338] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:03.750467] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:03.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:03.750516] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919845b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:03.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:03.723203] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=6781383] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:03:03.752162] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28914] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:03:03.755503] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:03.756741] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:03.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:02:54.766632] EDIAG [SQL.DAS] execute_dist_das_task (ob_data_access_service.cpp:177) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=9141][errcode=-4038] LOGGER COST TOO MUCH TIME, cost: 8968378, owner: ObLog, click_count: 5, time dist:[FORMAT_END=8968361, ObMalloc#start=2, ObMalloc#end=4, ALLOC_END=5, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x15f36e49 0x95f5ef0 0x9344f39 0x9343c25 0x9615b91 0x9613406 0x9613912 0x92df373 0x932ff51 0x932e8be 0x96ac524 0x96a672d 0x96a5033 0x97aa16f 0x124eac91 0x97a9944 0x261395f9 0xfa0e430 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:03.758711] WDIAG [SQL.DAS] execute_das_task (ob_data_access_service.cpp:108) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=8968378][errcode=-4038] failed to execute dist das task(ret=-4038) [2025-08-14 09:03:03.761606] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:03.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:03.761653] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:03.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:03.761679] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:03.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:03.761693] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:03.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:03.761706] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eacb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:03.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:03.763264] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:03.763297] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:03.764711] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:03.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.764741] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:03.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.770364] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:03.772367] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:03.767, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:03.776040] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=17] gc stale ls task succ [2025-08-14 09:03:03.778278] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6f78, timer:0x7f99329e1b10, task:0x7f98e4e091c0, task_type:"N9oceanbase5table21ObClearTTLHistoryTaskE", scheduled_time:1755162122736611, pushed_time:0, delay:30000000}, running_task_token={this:0x7f991b7a4c78, timer:0x7f99329e1b10, task:0x7f98e4e091f8, task_type:"N9oceanbase5table21ObTTLAllTaskSchedulerE", scheduled_time:1755162064731690, pushed_time:1755162092737016, delay:15000000}, total_delay=61041666, threashold=1000000, tenant_id=1002) [2025-08-14 09:03:03.778348] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=13] gc stale ls task succ [2025-08-14 09:03:03.778372] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=8] gc stale ls task succ [2025-08-14 09:03:03.778549] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:03.778578] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:03.778604] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:03.779362] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:03:03.779404] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:03:03.779418] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:03:03.779436] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:03:03.779452] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:03:03.779472] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:03:03.779501] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:03:03.779518] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:03:03.780047] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:03.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.782329] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:03.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.783319] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:03.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:03.783346] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:03.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:03.783362] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:03.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:03.783379] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:03.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:03.793394] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72716b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a4910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:03.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:03.793425] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:03.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:03.793440] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918100b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:03.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:03.793461] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:03.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:03.803472] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:03.793, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:03.803506] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180fe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:03.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:03.817271] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:03.805, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:03.827340] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73927b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:03.812, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:03.827374] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7399090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:03.815, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:03.827397] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:03.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:03.837414] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73954b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:03.827, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:03.847254] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:03.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.868827] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:03:03.868883] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:03:03.868929] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:03:03.868973] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:03:03.868996] INFO [COMMON] print_io_status (ob_io_manager.cpp:2639) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:1, hold_mem:8192 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4085, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:987761, total:987767) [2025-08-14 09:03:03.869022] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=500, ref_cnt=4, io_config={group_configs_cnt:3, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, param_config_:{memory_limit:536870912, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=930848, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:03:03.869069] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2826) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=42] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4085, [delay/us]: prepare:0, schedule:0, submit:600000000, device:987761, total:987767) [2025-08-14 09:03:03.869090] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:03:03.869105] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=1, sync_total=0, async_cnt= 1 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:03:03.869118] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:03:03.877323] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=20] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:03.887752] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:03.873, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:03.947322] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:03.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.982464] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:03.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.998006] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:03.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.016119] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:04.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.016153] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:04.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.021678] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:04.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.047396] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:03.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.096656] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:04.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.137385] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] EASY SLOW: thread: 7f9953efb300, lastrun: 1755162183.772427 cost: 0.364956 loop:26544, slowcnt: 3 [2025-08-14 09:03:04.147469] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:04.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.178655] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=2685059][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc65d8, timer:0x7f9970f8c420, task:0x564d6e8e3210, task_type:"N9oceanbase3omt19ObTenantTimezoneMgr18DeleteTenantTZTaskE", scheduled_time:1755161702763867, pushed_time:0, delay:5000000}, running_task_token={this:0x7f9970fc66b8, timer:0x7f9970f8c420, task:0x564d6e8e3228, task_type:"N9oceanbase3omt19ObTenantTimezoneMgr18UpdateTenantTZTaskE", scheduled_time:1755161702763810, pushed_time:1755161702764275, delay:5000000}, total_delay=481414787, threashold=1000000, tenant_id=500) [2025-08-14 09:03:04.182600] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:04.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.217400] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:04.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.237900] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=29] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:04.237926] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162184237896], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:04.237951] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162184237896], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:04.237980] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=25] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162184237896]}) [2025-08-14 09:03:04.237998] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162184237896]}) [2025-08-14 09:03:04.238018] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=17] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:04.238031] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162184238016], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:04.238047] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162184238016]}) [2025-08-14 09:03:04.238058] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162184238016]}) [2025-08-14 09:03:04.238256] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:03:04.238273] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:03:04.238284] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162184238252], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:04.238302] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162184238252]}) [2025-08-14 09:03:04.247533] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:04.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.249324] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:04.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.250018] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.248, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:04.259403] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:04.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.267654] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:04.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.267692] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:04.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.273377] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:04.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.303560] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:04.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.349266] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:04.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.383257] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:04.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.449319] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:04.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.490681] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:04.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.519031] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:04.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.519061] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:04.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.525449] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:04.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.549389] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:04.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.583386] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:04.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.599562] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:04.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.628802] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEAD-0-0] [lt=160818] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:03:04.649438] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:04.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.684266] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:03:04.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.713392] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:04.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.719081] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:04.726951] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:04.726987] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:04.733453] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=13] PNIO [ratelimit] time: 1755162184733450, bytes: 0, bw: 0.000000 MB/s, add_ts: 1005912, add_bytes: 0 [2025-08-14 09:03:04.733547] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:04.734331] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd110, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd3f0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:10.00s, expected_run_time:09:03:04.726, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.736262] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:04.736293] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:04.736288] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:04.738521] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:04.738543] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162184738516], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:04.738562] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162184738516], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:04.738841] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=275] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162184738516]}) [2025-08-14 09:03:04.738854] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:03:04.738865] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:03:04.738878] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=8] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:04.738893] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162184738877], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:04.738905] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162184738877]}) [2025-08-14 09:03:04.739102] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:03:04.739114] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:03:04.739128] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162184739097], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:04.739141] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162184739097]}) [2025-08-14 09:03:04.743088] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=22] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:04.743721] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:04.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.744613] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7270030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a3b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.735, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:04.744688] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=73][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ecb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.739, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:04.744722] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7353730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7396990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:04.744762] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:04.747213] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:04.748815] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:04.750493] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:04.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.750551] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:03.772436] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:04.752688] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=978770][errcode=0] epoll handler too much time :980260 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:03.758742] EDIAG [SQL.DAS] execute_all_task (ob_das_ref.cpp:385) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=18][errcode=-4038] LOGGER COST TOO MUCH TIME, cost: 1001018, owner: ObLog, click_count: 7, time dist:[FORMAT_END=1001006, ObMalloc#start=1, ObMalloc#end=6, ALLOC_END=1, APPEND_END=4, ALLOC_BLOCK#start=15, ALLOC_BLOCK#end=2], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x15ebc05e 0x9344205 0x9615b91 0x9613406 0x9613912 0x92df373 0x932ff51 0x932e8be 0x96ac524 0x96a672d 0x96a5033 0x97aa16f 0x124eac91 0x97a9944 0x261395f9 0xfa0e430 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:03.722506] EDIAG [STORAGE] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:123) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1044477, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1044472, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1c039e1e 0x97a8b06 0x97a7481 0x97a6a43 0x97a42c5 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:03.779533] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:03:04.753659] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=972649][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:03:04.753869] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:03:04.753885] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:03:03.835710] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=967259] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:03:04.756352] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:04.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.756534] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:04.756578] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:03.758742] WDIAG [SQL.DAS] execute_all_task (ob_das_ref.cpp:385) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=18][errcode=-4038] failed to execute aggregated das task(ret=-4038, agg_task={server_:"172.77.0.7:2882", start_status_:0, tasks_.get_size():0, failed_tasks_.get_size():1, success_tasks_.get_size():0, save_ret_:0}, async=true) [2025-08-14 09:03:04.764808] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:04.764839] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [384]OB_MALLOC COST TOO MUCH TIME, cost_time=986859, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=986850], seq:[0] [2025-08-14 09:03:04.766516] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:04.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.766611] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=92][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:04.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:03.722506] INFO [STORAGE] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:123) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] wead read timestamp is too old(timestamp={val:1755161749741193000, v:0}, gts_scn={val:1755161762721798002, v:0}, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1755161749741193000, v:0}}) [2025-08-14 09:03:03.815433] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=958954] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:03:04.747186] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:04.773417] WDIAG ev_invoke_pending (ev.c:2136) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26189][errcode=0] epoll handler too much time :26241 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:04.773509] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ea30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:04.773527] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919afbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:04.778432] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=12] gc stale ls task succ [2025-08-14 09:03:04.778678] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:04.783510] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:04.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.856610] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:04.880519] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c0b90, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571c0e70, time_wheel:0x7f9959fef990, thread_pool:0x7f9959fef170, is_running:1, time_interval:3.00s, expected_run_time:09:03:04.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.956668] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:04.983602] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:04.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.998357] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:04.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:05.056729] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:05.156760] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:05.183693] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:05.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:05.240337] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:05.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:05.250406] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:05.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:05.254062] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:03:05.254101] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:03:05.254112] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:03:05.254129] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:03:05.254138] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:03:05.254163] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:03:05.254179] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:03:05.254190] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:03:05.254199] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:03:05.254210] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:03:05.254409] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:03:05.254421] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:03:05.259254] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:05.359314] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:05.383833] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:05.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:05.459395] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:05.491926] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:05.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:05.559453] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:05.583959] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:05.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:05.659508] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:05.687667] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:03:05.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:05.718991] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=23] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:05.733309] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:05.733383] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:05.738978] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=59] PNIO [ratelimit] time: 1755162185738975, bytes: 0, bw: 0.000000 MB/s, add_ts: 1005525, add_bytes: 0 [2025-08-14 09:03:04.773535] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919854b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:05.740667] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:05.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.778612] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:05.740760] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=960693][errcode=0] epoll handler too much time :962156 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:04.778704] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:05.740780] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=960622][errcode=0] epoll handler too much time :962081 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:04.852253] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:04.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:05.012705] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:05.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:05.743142] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=23] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:05.748304] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:05.748355] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:05.754082] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [660]OB_MALLOC COST TOO MUCH TIME, cost_time=989004, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=989003], seq:[0] [2025-08-14 09:03:04.743064] EDIAG [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6822][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1025983, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1025981, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x99721a1 0x997a72e 0x2006ed17 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [567]OB_MALLOC COST TOO MUCH TIME, cost_time=998564, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [729]OB_MALLOC COST TOO MUCH TIME, cost_time=998761, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [2025-08-14 09:03:04.769756] EDIAG [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=1044477][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1003780, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1003778, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0xfebfd41 0x94edd90 0x94edb97 0x97a8fbd 0x97a8cc7 0x97a88d2 0x97a7481 0x97a6a43 0x97a42c5 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [659]OB_MALLOC COST TOO MUCH TIME, cost_time=1005830, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1005823], seq:[0] [659]OB_MALLOC COST TOO MUCH TIME, cost_time=1006096, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=53, WASH_BLOCK_END=186, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1005851], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:04.769776] EDIAG [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1421) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1006119, owner: ObLog, click_count: 5, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=1006104, ALLOC_END=2, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xfd2ed70 0xfd2c013 0xf9d16a8 0xf9d4150 0xfa09ca2 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:03.727178] EDIAG [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=1016241][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2047383, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2047382, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1bdd73e2 0x1bdd6e36 0x986bc6f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [553]OB_MALLOC COST TOO MUCH TIME, cost_time=1005327, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1005303, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [553]OB_MALLOC COST TOO MUCH TIME, cost_time=1005363, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1005342, WASH_BLOCK_END=21], seq:[0, 1, 2] [2025-08-14 09:03:03.815263] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=964720][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1960371, owner: ObLog, click_count: 5, time dist:[FORMAT_END=954975, ObMalloc#start=1, ObMalloc#end=1005371, ALLOC_END=24, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [392]OB_MALLOC COST TOO MUCH TIME, cost_time=1004369, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1004347, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=1004389, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1004378, WASH_BLOCK_END=11], seq:[0, 1, 2] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=1003887, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=1003869, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=1003904, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1003895, WASH_BLOCK_END=9], seq:[0, 1, 2] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=1001387, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1001366, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=1001404, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1001395, WASH_BLOCK_END=9], seq:[0, 1, 2] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=999177, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=999155, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=999195, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=999186, WASH_BLOCK_END=9], seq:[0, 1, 2] [2025-08-14 09:03:03.728870] EDIAG [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2047331, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2047331, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1bdd73e2 0x1bdd6e36 0x986bc6f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [477]OB_MALLOC COST TOO MUCH TIME, cost_time=993350, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=993327, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=993377, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=993360, WASH_BLOCK_END=17], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=900043, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=900022, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=900074, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=900051, WASH_BLOCK_END=22], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=739598, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=739574, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=739616, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=739606, WASH_BLOCK_END=10], seq:[0, 1, 2] [400]OB_MALLOC COST TOO MUCH TIME, cost_time=641108, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=641086, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [400]OB_MALLOC COST TOO MUCH TIME, cost_time=641132, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=641120, WASH_BLOCK_END=12], seq:[0, 1, 2] [381]OB_MALLOC COST TOO MUCH TIME, cost_time=539908, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=539884, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [381]OB_MALLOC COST TOO MUCH TIME, cost_time=539926, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=539917, WASH_BLOCK_END=9], seq:[0, 1, 2] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=1007763, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=986901], seq:[0, 1] [2025-08-14 09:03:04.771303] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1007784, owner: ObLog, click_count: 5, time dist:[FORMAT_END=7, ObMalloc#start=0, ObMalloc#end=1007776, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:05.755837] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:03:05.733281] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:05.754147] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:05.744, task_priority:1, with_handle_protected:1}) [660]OB_MALLOC COST TOO MUCH TIME, cost_time=1007860, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=989018], seq:[0, 1] [2025-08-14 09:03:04.766960] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=3937059] temporary file module use no blocks [2025-08-14 09:03:05.760815] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:04.769299] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=3951117] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:05.740748] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=18] temporary file module use no blocks [2025-08-14 09:03:05.740908] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=727100][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:05.127|207=728.20ms|210=0us|222=1us|total=748.79ms) [2025-08-14 09:03:05.740883] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=887284][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:04.852|207=888.63ms|210=0us|222=2us|total=909.46ms) [2025-08-14 09:03:05.761709] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=20793][errcode=-4389] timer task use too much time(end=1755162185761708, start=1755162184852249, delta=909459) [2025-08-14 09:03:05.737841] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=3955452] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:05.740361] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=965360][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:04.773|207=966.82ms|210=0us|222=2us|total=988.30ms) [2025-08-14 09:03:05.761837] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21442][errcode=-4389] timer task use too much time(end=1755162185761836, start=1755162184773534, delta=988302) [2025-08-14 09:03:05.761847] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1003112, expected_run_ts_us_=1755162184758733, ObClockGenerator::getRealClock()=1755162185761845, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:05.761867] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:05.761880] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1002910, expected_run_ts_us_=1755162184758970, ObClockGenerator::getRealClock()=1755162185761880, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:05.761891] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:05.761901] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1008993, expected_run_ts_us_=1755162184752908, ObClockGenerator::getRealClock()=1755162185761901, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d45b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:04.771243] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1007929, owner: ObLog, click_count: 5, time dist:[FORMAT_END=45, ObMalloc#start=1, ObMalloc#end=1007881, ALLOC_END=2, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:05.761911] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d45b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:05.741033] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=3938871] temporary file module use no blocks [2025-08-14 09:03:04.771936] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=3956630] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:03:05.768508] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:05.768537] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:04.743064] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6822] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:05.761922] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=995329, expected_run_ts_us_=1755162184766592, ObClockGenerator::getRealClock()=1755162185761921, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72747b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:05.774638] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12695][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72747b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:05.774662] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] task scheduled too delay(scheduled_delay=1008253, expected_run_ts_us_=1755162184766408, ObClockGenerator::getRealClock()=1755162185774661, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:05.774684] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:05.774695] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=999075, expected_run_ts_us_=1755162184775620, ObClockGenerator::getRealClock()=1755162185774695, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:04.769756] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=1044477] get_max_applied_scn(scn={val:1755161744727824001, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:49, palf_committed_end_lsn_:{lsn:7323686009}, palf_committed_end_scn_:{val:1755161747742105001, v:0}, last_check_scn_:{val:1755161749741193000, v:0}, max_applied_cb_scn_:{val:1755161744727824001, v:0}}) [2025-08-14 09:03:05.775090] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=1003780] get_min_unreplayed_log_info(lsn={lsn:7323685240}, scn={val:1755100946796590002, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873485, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:8946764, ret_code_:0}}, next_to_submit_lsn_:{lsn:7323685240}, next_to_submit_scn_:{val:1755100946796590002, v:0}, base_lsn_:{lsn:7180210176}, base_scn_:{val:1753840803166611000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:7323685240}, end_lsn:{lsn:7323685240}, read_buf:{buf_len_:3747840, buf_:0x7f98f7a05000}, block_size:67104768, log_storage_:0x7f9901131bf0, log_storage_:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:28, read_io_size:98508800, read_disk_cost_ts:645339, start_lsn:{lsn:7180210176}}}}}}) [2025-08-14 09:03:05.775926] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=92] get wrs ts(ls_id={id:1001}, delta=441041418, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:05.775946] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:05.776411] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] get wrs ts(ls_id={id:1}, delta=465994063, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:05.776424] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:05.776452] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119797572, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755162138745740) [2025-08-14 09:03:05.777033] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=465994063, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:05.777075] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=41] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:05.777343] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [726][T1001_ObXAHbWor][T1001][Y0-0000000000000000-0-0] [lt=12] XA scheduler heartbeat task statistics(avg_time=1) [2025-08-14 09:03:05.770638] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=1025983] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:05.777360] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6710] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:05.777372] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:05.777378] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:05.777389] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:04.769776] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1421) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=22] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-08-14 09:03:05.777578] WDIAG print_alloc_failed_msg (alloc_failed_reason.cpp:146) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=1006119][errcode=-4013] [OOPS]: alloc failed reason is that server memory has reached the upper limit(server_hold: 4293681152, server_limit: 4294967296, alloc_size: 2097152). detailed info: tenant_id=500, ctx_id=16, ctx_name=RPC_CTX_ID, ctx_hold=2981888, ctx_limit=9223372036854775807, tenant_hold=696590336, tenant_limit=9223372036854775807, backtrace=0x26a1806d 0xf98a0db 0xf989b06 0x9327c08 0x9315ac5 0x27aa2827 0x27a872f9 0xfcf0b15 0xfc721d0 0xfd2bbd0 0xf9d16a8 0xf9d4150 0xfa09ca2 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:05.777599] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_LOG_ELECTION_PREPARE_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-08-14 09:03:03.727178] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=1016241] finish update min start scn(max_decided_scn={val:1755161719775179002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1755161287778691002, v:0}, keep_alive_scn_:{val:1755161287778691002, v:0}, update_ts_:1755162183721257}) [2025-08-14 09:03:03.815263] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=964720] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:03:05.778666] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=11] [sql nio session](*s={this:0x7f98f733f030, session_id:3221589417, trace_id:YB42AC4D0007-00063C496359B268-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:252}, err:0, last_decode_time:1755161813752329, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1778, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:03:05.778703] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=37] [sql nio session](*s={this:0x7f98f733e030, session_id:3221589980, trace_id:YB42AC4D0007-00063C496359B261-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:251}, err:0, last_decode_time:1755161748778070, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1681, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:03:05.778714] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=11] [sql nio session](*s={this:0x7f98f7332030, session_id:3221590132, trace_id:YB42AC4D0007-00063C496359B265-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:266}, err:0, last_decode_time:1755161768783802, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:2125, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:03:05.779055] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:05.779078] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=10] gc stale ls task succ [2025-08-14 09:03:05.779119] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:03.728870] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=4] finish update min start scn(max_decided_scn={val:1755161547738516001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1755161493768447002, v:0}, keep_alive_scn_:{val:1755161493768447002, v:0}, update_ts_:1755162183721257}) [2025-08-14 09:03:04.878023] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=26] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:05.740921] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:213) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-08-14 09:03:05.779835] INFO [COMMON] handle (memory_dump.cpp:544) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=38853] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-08-14 09:03:05.741052] INFO [SERVER] run (obmp_disconnect.cpp:79) [1001][T1002_L0_G0][T1002][Y0-0000000000000000-0-0] [lt=51] free session successfully(sessid=3221590970) [2025-08-14 09:03:05.736259] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:04.771303] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:05.780631] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1007784][errcode=0] epoll handler too much time :1009335 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:05.780647] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:04.771243] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:04.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:05.780720] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1007929][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:04.771|207=1.01s|210=0us|222=1us|total=1.01s) [2025-08-14 09:03:05.780736] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4389] timer task use too much time(end=1755162185780735, start=1755162184771229, delta=1009506) [2025-08-14 09:03:05.780745] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=0] task scheduled too delay(scheduled_delay=988395, expected_run_ts_us_=1755162184792349, ObClockGenerator::getRealClock()=1755162185780745, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:04.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.776102] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=25][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1003224, owner: ObLog, click_count: 6, time dist:[FORMAT_END=22, ObMalloc#start=1, ObMalloc#end=1001410, ObMalloc#start=27, ObMalloc#end=51, FREE_END=1713], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:04.772859] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1006518, owner: ObLog, click_count: 6, time dist:[FORMAT_END=8, ObMalloc#start=1, ObMalloc#end=1004396, ObMalloc#start=21, ObMalloc#end=56, FREE_END=2036], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:04.773488] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1005918, owner: ObLog, click_count: 6, time dist:[FORMAT_END=5, ObMalloc#start=0, ObMalloc#end=1003921, ObMalloc#start=19, ObMalloc#end=55, FREE_END=1918], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:05.780821] WDIAG ev_invoke_pending (ev.c:2136) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=35517][errcode=0] epoll handler too much time :35592 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:05.774060] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get_max_applied_scn(scn={val:1755161547738516001, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:50, palf_committed_end_lsn_:{lsn:5282737655}, palf_committed_end_scn_:{val:1755161547738516001, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161547738516001, v:0}}) [2025-08-14 09:03:05.780871] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=1003224] gc stale ls task succ [2025-08-14 09:03:05.780945] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1005918][errcode=0] epoll handler too much time :1007463 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:05.780962] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:05.780984] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=540721][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162185239437], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:05.781003] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162185239437], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:05.781022] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162185239437]}) [2025-08-14 09:03:05.781036] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:03:05.781046] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:03:05.781068] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=8] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:04.778455] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1001326, owner: ObLog, click_count: 6, time dist:[FORMAT_END=7, ObMalloc#start=1, ObMalloc#end=999201, ObMalloc#start=25, ObMalloc#end=120, FREE_END=1972], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:05.781362] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6769] get wrs ts(ls_id={id:1}, delta=638038604, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:05.781383] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:05.781408] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162185774048}, version_delta=638042891) [2025-08-14 09:03:05.781990] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=638038604, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:05.782003] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:05.761498] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=20557][errcode=-4389] timer task use too much time(end=1755162185761497, start=1755162185012703, delta=748794) [2025-08-14 09:03:05.782224] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=20692][errcode=0] task scheduled too delay(scheduled_delay=773089, expected_run_ts_us_=1755162185009134, ObClockGenerator::getRealClock()=1755162185782224, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:05.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:05.774708] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:05.784084] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:05.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:05.860883] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:05.960951] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:05.984210] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:05.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:05.999402] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=5554][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:05.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:06.061021] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:06.161058] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:06.184305] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:06.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:06.261117] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:06.361178] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:06.384439] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:06.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:06.461237] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:06.561297] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:06.584562] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:06.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:06.661382] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:06.693598] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:03:06.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:06.719043] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=14] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:06.727261] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:05.780760] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:04.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:05.137942] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1589096, owner: ObLog, click_count: 7, time dist:[FORMAT_END=15, ObMalloc#start=0, ObMalloc#end=641138, ObMalloc#start=20, ObMalloc#end=53, ALLOC_END=947869, APPEND_END=1], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:05.039013] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1687915, owner: ObLog, click_count: 7, time dist:[FORMAT_END=31, ObMalloc#start=1, ObMalloc#end=739623, ObMalloc#start=19, ObMalloc#end=59, ALLOC_END=948181, APPEND_END=1], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [421]OB_MALLOC COST TOO MUCH TIME, cost_time=946871, tenant_id=500, label=IoControl, ctx_id=0, prio=0, size=65280, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=3], seq:[0, 1] [2025-08-14 09:03:06.728584] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=946386][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:05.780|207=947.84ms|210=0us|222=2us|total=947.87ms) [2025-08-14 09:03:06.728616] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=-4389] timer task use too much time(end=1755162186728616, start=1755162185780744, delta=947872) [2025-08-14 09:03:06.728649] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=718616, expected_run_ts_us_=1755162186010031, ObClockGenerator::getRealClock()=1755162186728648, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:06.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:06.728668] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:06.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:05.782258] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:05.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:06.728716] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=945025][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:05.782|207=946.49ms|210=0us|222=1us|total=946.50ms) [2025-08-14 09:03:06.728729] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task use too much time(end=1755162186728728, start=1755162185782223, delta=946505) [2025-08-14 09:03:06.728737] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=0] task scheduled too delay(scheduled_delay=1639420, expected_run_ts_us_=1755162185089316, ObClockGenerator::getRealClock()=1755162186728736, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:05.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:05.782278] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7555][errcode=0] task scheduled too delay(scheduled_delay=1006622, expected_run_ts_us_=1755162184775655, ObClockGenerator::getRealClock()=1755162185782278, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:05.780348] WDIAG ev_invoke_pending (ev.c:2136) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=44021][errcode=0] epoll handler too much time :44097 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:05.736311] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:06.729405] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=991593][errcode=0] epoll handler too much time :993099 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:05.736957] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:06.729426] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=990969][errcode=0] epoll handler too much time :992480 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:05.137942] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] EASY SLOW: thread: 7f995407d300, lastrun: 1755162184.772848 cost: 0.365093 loop:26557, slowcnt: 3 [2025-08-14 09:03:05.039013] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:04.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:06.155272] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=2702092][errcode=0] EASY SLOW: thread: 7f9954779300, lastrun: 1755162185.779808 cost: 0.375463 loop:26547, slowcnt: 3 [2025-08-14 09:03:06.729505] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1687915][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:05.390|207=1.69s|210=0us|222=2us|total=1.69s) [2025-08-14 09:03:06.729518] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task use too much time(end=1755162186729518, start=1755162185039011, delta=1690507) [2025-08-14 09:03:05.780917] WDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1006518][errcode=0] epoll handler too much time :1008068 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:06.729551] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=947190] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:05.781078] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162185781064], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:06.729573] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=947056] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162185781064]}) [2025-08-14 09:03:05.781313] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=1001326] gc stale ls task succ [2025-08-14 09:03:05.865720] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [571][T1_ArcSender][T1][YB42AC4D0007-00063C495BA99C16-0-0] [lt=1834520][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:03:05.866255] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=1835475][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:03:05.879684] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=900171] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:05.881347] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=39086] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [567]OB_MALLOC COST TOO MUCH TIME, cost_time=847225, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16448, [2025-08-14 09:03:06.729739] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=847063] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:03:06.729802] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:03:06.729848] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:03:06.729868] INFO [COMMON] print_io_status (ob_io_manager.cpp:2639) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:1, hold_mem:8192 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:0, norm_iops:0, bw:1431, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:988465, total:988471) [2025-08-14 09:03:06.729893] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=4, io_config={group_configs_cnt:3, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, param_config_:{memory_limit:536870912, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=930848, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:03:06.729944] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2826) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=46] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:1431, [delay/us]: prepare:0, schedule:0, submit:600000000, device:988465, total:988471) [2025-08-14 09:03:06.729967] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:03:06.729980] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=1, sync_total=0, async_cnt= 1 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:03:06.735274] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=947610] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:06.736257] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:06.736289] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [729]OB_MALLOC COST TOO MUCH TIME, cost_time=861125, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:03:06.244845] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:06.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:06.739360] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=493763][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:06.244|207=494.51ms|210=1us|222=1us|total=494.58ms) [2025-08-14 09:03:06.739428] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=67][errcode=-4389] timer task use too much time(end=1755162186739427, start=1755162186244844, delta=494583) [2025-08-14 09:03:06.739441] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=494807, expected_run_ts_us_=1755162186244633, ObClockGenerator::getRealClock()=1755162186739441, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:06.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:06.739459] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:06.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:06.512598] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fe030, caller:ob_table_lock_service.cpp:start:276, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE6$_1273, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe310, time_wheel:0x7f99535f3990, thread_pool:0x7f99535f3170, is_running:1, time_interval:10.00s, expected_run_time:09:03:06.426, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:06.739511] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=226567][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:06.512|207=226.91ms|210=0us|222=1us|total=226.93ms) [2025-08-14 09:03:06.739524] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] timer task use too much time(end=1755162186739524, start=1755162186512595, delta=226929) [2025-08-14 09:03:05.726964] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fb3b0, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fb690, time_wheel:0x7f99589fd990, thread_pool:0x7f9959fff170, is_running:1, time_interval:2.00s, expected_run_time:09:03:05.698, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:05.756717] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23399][errcode=0] epoll handler too much time :23452 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:06.739667] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=981450] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:05.873244] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a58b8, timer:0x7f9966ee3e30, task:0x7f994f1a3a78, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr15MediumCheckTaskE", scheduled_time:1755161764744054, pushed_time:0, delay:20000000}, running_task_token={this:0x7f994e1a57d8, timer:0x7f9966ee3e30, task:0x7f994f1a3a38, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr14MediumLoopTaskE", scheduled_time:1755161760733696, pushed_time:1755161760733763, delay:120000000}, total_delay=421129177, threashold=1000000, tenant_id=1) [2025-08-14 09:03:05.777610] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:57) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-08-14 09:03:06.728748] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:05.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:06.740454] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11682][errcode=0] task scheduled too delay(scheduled_delay=1536155, expected_run_ts_us_=1755162185204297, ObClockGenerator::getRealClock()=1755162186740453, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:05.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:06.728763] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=945053][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:05.779816] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:05.726964] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1011064, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1011058, ObMalloc#start=0, ObMalloc#end=3, ALLOC_END=0, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:06.741577] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=1011064][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:05.726|207=1.01s|210=0us|222=2us|total=1.01s) [2025-08-14 09:03:06.741602] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4389] timer task use too much time(end=1755162186741602, start=1755162185726962, delta=1014640) [2025-08-14 09:03:06.742828] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] update_check_range(min_check_version=2540, max_check_version=2540, global_version=2644) [2025-08-14 09:03:06.743368] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=33] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:06.744743] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6919] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:06.744772] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=29] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:06.744780] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:06.744788] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:06.744799] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:06.744813] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:06.746591] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=17] PNIO [ratelimit] time: 1755162186746589, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007614, add_bytes: 0 [2025-08-14 09:03:06.747064] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:06.747109] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:06.747143] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:06.750065] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:06.752134] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:06.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:05.777396] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:06.753509] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=974637] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:06.753525] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:06.740476] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:05.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:06.753575] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13068][errcode=0] task scheduled too delay(scheduled_delay=998087, expected_run_ts_us_=1755162185755487, ObClockGenerator::getRealClock()=1755162186753575, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:05.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:06.753599] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:05.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:06.740533] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11749][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:05.782|207=958.25ms|210=0us|222=2us|total=958.27ms) [2025-08-14 09:03:06.753652] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13099][errcode=-4389] timer task use too much time(end=1755162186753652, start=1755162185782277, delta=971375) [2025-08-14 09:03:06.753664] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1981763, expected_run_ts_us_=1755162184771901, ObClockGenerator::getRealClock()=1755162186753664, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:06.753699] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:06.729521] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [567]OB_MALLOC COST TOO MUCH TIME, cost_time=878792, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=847236], seq:[0, 1] [2025-08-14 09:03:06.744823] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:06.755580] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=10739] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:06.755590] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:06.762218] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:03:06.762254] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=35] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:03:06.762265] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:06.753713] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=1980741, expected_run_ts_us_=1755162184772971, ObClockGenerator::getRealClock()=1755162186753713, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:06.829591] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:06.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:06.862290] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:06.878888] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=2847186][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:03:06.929651] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:06.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:06.929855] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=848706] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:06.962332] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:06.992374] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:06.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:07.036342] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:06.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:07.065251] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:07.136411] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:07.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:07.165266] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:07.188443] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=2806060][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162186.777876 cost: 0.410565 loop:26530, slowcnt: 4 [2025-08-14 09:03:07.236485] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=53][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:07.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:07.246706] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:07.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:07.256769] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:07.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:07.265325] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:07.336564] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:07.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:07.365394] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:07.436625] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:07.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:07.465455] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:07.498149] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:07.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:07.536700] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:07.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:07.565514] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:07.636750] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:07.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:07.643948] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEAF-0-0] [lt=31] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:03:07.647496] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEB1-0-0] [lt=29] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:03:07.665548] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:07.694516] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571634a0, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957163780, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:5.00s, expected_run_time:09:03:07.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:07.694555] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:03:07.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:07.719118] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=24] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [857]OB_MALLOC COST TOO MUCH TIME, cost_time=978477, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=26389], seq:[0, 1] [2025-08-14 09:03:05.783536] EDIAG [STORAGE.TRANS] amend (ob_tenant_weak_read_server_version_mgr.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1933869, owner: ObLog, click_count: 5, time dist:[FORMAT_END=955368, ObMalloc#start=1, ObMalloc#end=978494, ALLOC_END=2, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x1c0234c0 0x97a449c 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [651]OB_MALLOC COST TOO MUCH TIME, cost_time=963162, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=963154], seq:[0] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=951196, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=951193], seq:[0] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=954171, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=954161], seq:[0] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=954180, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=954178], seq:[0, 1] [412]OB_MALLOC COST TOO MUCH TIME, cost_time=951308, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=951301], seq:[0] [412]OB_MALLOC COST TOO MUCH TIME, cost_time=951337, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=951335], seq:[0, 1] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=951328, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=951317, ALLOC_CHUNK_END=6], seq:[0, 1] [2025-08-14 09:03:05.773723] EDIAG [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=992010][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1943996, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1943990, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9826e08 0x98267df 0x982650b 0x98262e0 0x1d188ec1 0x1d17b8ec 0x1d17ad88 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [567]OB_MALLOC COST TOO MUCH TIME, cost_time=951543, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=951533, ALLOC_CHUNK_END=4], seq:[0, 1] [386]OB_MALLOC COST TOO MUCH TIME, cost_time=951313, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=951303, ALLOC_CHUNK_END=5], seq:[0, 1] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=947840, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=947832, ALLOC_CHUNK_END=4], seq:[0, 1] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=946290, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=946280, ALLOC_CHUNK_END=5], seq:[0, 1] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=942376, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=942368, ALLOC_CHUNK_END=4], seq:[0, 1] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=941775, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=941766, ALLOC_CHUNK_END=5], seq:[0, 1] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=941757, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=941748, ALLOC_CHUNK_END=5], seq:[0, 1] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=941772, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=941763, ALLOC_CHUNK_END=5], seq:[0, 1] [659]OB_MALLOC COST TOO MUCH TIME, cost_time=939245, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=939234, ALLOC_CHUNK_END=6], seq:[0, 1] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=938672, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=938663, ALLOC_CHUNK_END=4], seq:[0, 1] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=938567, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=938557, ALLOC_CHUNK_END=5], seq:[0, 1] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=940720, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=940711, ALLOC_CHUNK_END=5], seq:[0, 1] [729]OB_MALLOC COST TOO MUCH TIME, cost_time=990428, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=990426], seq:[0] [2025-08-14 09:03:05.783536] INFO [STORAGE.TRANS] amend (ob_tenant_weak_read_server_version_mgr.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [WRS] tenant has no valid partition on current server, generate weak read snapshot version based on current timestamp(tenant_id=1002, epoch_tstamp_=1755162179751719, total_part_count_=2, valid_inner_part_count_=0, valid_user_part_count_=0, version_={val:1755161758221798000, v:0}, base_version_when_no_valid_partition={val:1755161758221798000, v:0}) [2025-08-14 09:03:06.764833] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:07.720477] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=954199][errcode=0] epoll handler too much time :955652 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:06.767704] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [412][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=23] [sql nio session](*s={this:0x7f98f733a030, session_id:3221590255, trace_id:YB42AC4D0007-00063C49651A3232-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:257}, err:0, last_decode_time:1755161952736711, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1716, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:03:05.773723] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=992010] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:03:07.726931] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [1001]OB_MALLOC COST TOO MUCH TIME, cost_time=977448, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=977446], seq:[0] [1001]OB_MALLOC COST TOO MUCH TIME, cost_time=977465, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=977461], seq:[0, 1] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=976961, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=29561], seq:[0, 1] [2025-08-14 09:03:06.729993] EDIAG [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1004198, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=2, ALLOC_BLOCK#end=2, FORMAT_END=1004190, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x1f99e657 0x1f981d5d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [777]OB_MALLOC COST TOO MUCH TIME, cost_time=965017, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=947849, WASH_BLOCK_END=17168], seq:[0, 1, 2] [386]OB_MALLOC COST TOO MUCH TIME, cost_time=968566, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=951321, WASH_BLOCK_END=17241, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1, 2, 3, 4] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=958859, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=941765, WASH_BLOCK_END=17093], seq:[0, 1, 2] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=963570, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=946297, WASH_BLOCK_END=17272], seq:[0, 1, 2] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=959671, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=942386, WASH_BLOCK_END=17284], seq:[0, 1, 2] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=955508, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=938682, WASH_BLOCK_END=16826], seq:[0, 1, 2] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=959005, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=941779, WASH_BLOCK_END=17226], seq:[0, 1, 2] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=954655, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=938582, WASH_BLOCK_END=16073], seq:[0, 1, 2] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=959116, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=941783, WASH_BLOCK_END=17333], seq:[0, 1, 2] [659]OB_MALLOC COST TOO MUCH TIME, cost_time=956618, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=939252, WASH_BLOCK_END=17366], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=952486, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=940736, WASH_BLOCK_END=11750], seq:[0, 1, 2] [651]OB_MALLOC COST TOO MUCH TIME, cost_time=984884, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=963168], seq:[0, 1] [2025-08-14 09:03:06.753787] INFO [SERVER] run (obmp_disconnect.cpp:86) [1001][T1002_L0_G0][T1002][Y0-0000000000000000-0-0] [lt=38738] mark session id unused(sessid=3221590970) [2025-08-14 09:03:06.755643] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:07.734131] WDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=976989][errcode=0] epoll handler too much time :978476 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:07.721454] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:233) [827][T1002_RFLWorker][T1002][YB42AC4D0007-00063C495D099C16-0-0] [lt=25721] ObRemoteFetchWorker is running(thread_index=0) [2025-08-14 09:03:06.729993] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:03:06.773309] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:07.735737] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=1004198] [IO STATUS SERVER] net_in=0kB/s, net_out=0kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=1280000kB/s [2025-08-14 09:03:07.735742] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=960979][errcode=0] epoll handler too much time :962441 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:06.780936] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=8] gc stale ls task succ [2025-08-14 09:03:07.736252] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:07.736785] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:07.664, task_priority:1, with_handle_protected:1}) [729]OB_MALLOC COST TOO MUCH TIME, cost_time=1012865, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=79, WASH_BLOCK_END=156, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=990442], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:06.739807] EDIAG [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1012915, owner: ObLog, click_count: 5, time dist:[FORMAT_END=23, ObMalloc#start=1, ObMalloc#end=1012890, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0xfebfd41 0x94edd90 0x94edb97 0x97a8fbd 0x97a8cc7 0x97a88d2 0x97a7481 0x97a6a43 0x97a42c5 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:05.740900] EDIAG LIBalloc (ob_malloc_allocator.cpp:924) [471][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4013] LOGGER COST TOO MUCH TIME, cost: 1991947, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1991941, ALLOC_END=2, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0xf9b6d19 0xf9b6367 0xf9b6058 0xf989dfb 0x973c6f6 0x973c18d 0x7f9973af5dfa 0x7f9973af5efd 0x9796af8 0x1fb130a7 0x98164f4 0x9816209 0x981582f 0x1bdf8786 0x1fb1c7eb 0x1fb1ca60 0x1231e896 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:05.774586] EDIAG [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=20885][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1978940, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1943829, ObMalloc#start=0, ObMalloc#end=35088, ALLOC_END=22, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9826e08 0x98267df 0x982650b 0x98262e0 0x1d188ec1 0x1d17b8ec 0x1d17ad88 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:06.764401] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:07.738198] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=972326][errcode=0] epoll handler too much time :973815 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:06.777885] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:07.738247] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=958910][errcode=0] epoll handler too much time :960370 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:07.738272] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:07.738298] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:06.773223] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:07.738313] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=963631][errcode=0] epoll handler too much time :965100 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:06.782273] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=946881] gc stale ls task succ [2025-08-14 09:03:06.777816] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:06.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:04.773820] INFO [SHARE.LOCATION] dump_statistic (ob_tablet_location_refresh_service.cpp:240) [365][TbltRefreshSer][T0][YB42AC4D0007-00063C4960B99C42-0-0] [lt=14] [REFRESH_TABLET_LOCATION] dump statistic(tenant_id=1002, base_task_id={id:0}, tablet_list_cnt=0, total_task_cnt=0, unknown_task_cnt=0, doing_task_cnt=0, done_task_cnt=0) [2025-08-14 09:03:05.755897] EDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4016] LOGGER COST TOO MUCH TIME, cost: 2004440, owner: ObLog, click_count: 11, time dist:[ALLOC_BLOCK#start=2, ALLOC_BLOCK#end=74, ALLOC_BLOCK#start=13295, ALLOC_BLOCK#end=83, ALLOC_BLOCK#start=10183, ALLOC_BLOCK#end=116, ALLOC_BLOCK#start=11871, ALLOC_BLOCK#end=3, FORMAT_END=1968812, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x13b06ffb 0x958a39e 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:05.773880] EDIAG [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=20696][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1986591, owner: ObLog, click_count: 7, time dist:[FORMAT_END=1943882, ObMalloc#start=2, ObMalloc#end=17498, ObMalloc#start=30, ObMalloc#end=25178, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9826e08 0x98267df 0x982650b 0x98262e0 0x1d188ec1 0x1d17b8ec 0x1d17ad88 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:06.740575] EDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=959306][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1034186, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=2, FORMAT_END=1034179, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:06.753775] EDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24216][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1021071, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=1, FORMAT_END=1021066, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:06.767764] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1007218, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=951203, FORMAT_END=56011, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [705]OB_MALLOC COST TOO MUCH TIME, cost_time=1003542, tenant_id=500, label=CLIENT_REQ_CHUNK, ctx_id=17, prio=0, size=16480, owner: ALLOC_BLOCK, click_count: 19, time dist:[ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ObMalloc#start=946139, ObMalloc#end=57295, ObLog#start=12, ObLog#end=13, ALLOC_CHUNK_END=13], seq:[0, 1, 3, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18] [705]OB_MALLOC COST TOO MUCH TIME, cost_time=1003579, tenant_id=500, label=CLIENT_REQ_CHUNK, ctx_id=17, prio=0, size=16288, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1003571], seq:[0, 1] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=1007835, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=951554, WASH_BLOCK_END=17881, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=38400], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:06.762514] EDIAG [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1013230, owner: ObLog, click_count: 5, time dist:[FORMAT_END=5345, ObMalloc#start=0, ObMalloc#end=1007860, ALLOC_END=24, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1c03a517 0x97a8d73 0x97a88d2 0x97a7481 0x97a6a43 0x97a42c5 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:06.755741] EDIAG [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EECD-0-0] [lt=23][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1020049, owner: ObLog, click_count: 7, time dist:[FORMAT_END=16, ObMalloc#start=1, ObMalloc#end=984898, ObMalloc#start=35131, ObMalloc#end=2, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1dc20366 0x1dc1fa40 0x1dd7bdd2 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:05.861785] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1913289, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=2, FORMAT_END=1888559, ObMalloc#start=1, ObMalloc#end=24703, ALLOC_END=20, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [374]OB_MALLOC COST TOO MUCH TIME, cost_time=1008714, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=951337, WASH_BLOCK_END=34320, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=23056], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:06.767776] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14041][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1008764, owner: ObLog, click_count: 5, time dist:[FORMAT_END=22, ObMalloc#start=0, ObMalloc#end=1008723, ALLOC_END=19, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:07.747500] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:06.780526] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts statistics(tenant_id=1001, gts_rpc_cnt=0, get_gts_cache_cnt=10, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:03:07.748765] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:07.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:07.751392] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:07.738514] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=959243][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:06.777|207=960.70ms|210=0us|222=3us|total=960.72ms) [2025-08-14 09:03:07.753030] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=14490][errcode=-4389] timer task use too much time(end=1755162187753028, start=1755162186777814, delta=975214) [2025-08-14 09:03:07.753047] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=960696, expected_run_ts_us_=1755162186792349, ObClockGenerator::getRealClock()=1755162187753046, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:06.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:07.753069] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:06.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:07.730913] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=979264, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=979258], seq:[0] [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=979378, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=48, WASH_BLOCK_END=43, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=979283], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:06.741446] EDIAG [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=3935982][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2013380, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1033990, ObMalloc#start=1, ObMalloc#end=979387, ALLOC_END=2, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1f832f91 0x97c36fe 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:07.753253] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=971254] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162185781064]}) [2025-08-14 09:03:07.753270] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1001}) [2025-08-14 09:03:06.739807] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] get_max_applied_scn(scn={val:1755161719775179002, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:45, palf_committed_end_lsn_:{lsn:8461098908}, palf_committed_end_scn_:{val:1755161719775179002, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161719775179002, v:0}}) [2025-08-14 09:03:07.754286] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1012915] get_min_unreplayed_log_info(lsn={lsn:8461098908}, scn={val:1755100947600130001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161920778323, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:23084275, ret_code_:0}}, next_to_submit_lsn_:{lsn:8461098908}, next_to_submit_scn_:{val:1755100947600130001, v:0}, base_lsn_:{lsn:7515734016}, base_scn_:{val:1753840867171901045, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6e05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:8461098908}, end_lsn:{lsn:8461098908}, read_buf:{buf_len_:3747840, buf_:0x7f98f6e05000}, block_size:67104768, log_storage_:0x7f99297a3bf0, log_storage_:{log_tail:{lsn:8461098908}, readable_log_tail:{lsn:8461098908}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8455200768}, min_scn:{val:1755160384851871002, v:0}, curr_block_id:126, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:215, min_block_id:93, max_block_id:127, curr_writable_block_id:126}, logical_block_size_:67104768, curr_block_writable_size_:61206628, block_header_serialize_buf_:0x7f99297a48b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:198, read_io_size:699822080, read_disk_cost_ts:4178035, start_lsn:{lsn:7515734016}}}}}}) [2025-08-14 09:03:07.754353] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:07.754870] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=73] get wrs ts(ls_id={id:1}, delta=467978544, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:07.753278] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1001}) [2025-08-14 09:03:07.755312] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=1779][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:03:07.756315] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=0] get_fetch_log_dst failed or invalid(ret=0, palf_id=1001, self="172.77.0.7:2882", fetch_log_dst="172.77.0.7:2882") [2025-08-14 09:03:05.774586] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=20885] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:03:07.719394] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:03:07.754889] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:07.758934] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:07.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:07.762332] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:05.755897] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:03:07.763403] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=2004440][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:03:07.763419] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:03:07.763427] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:03:07.763441] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:03:07.763454] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:03:05.773880] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=20696] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:03:07.742156] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [628][T1_L0_G0][T1][YB42AC4D0007-00063C49607B92BE-0-0] [lt=42][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162182765011) [2025-08-14 09:03:07.772522] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:07.772550] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:07.774177] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:06.777907] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:07.775222] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=995808][errcode=0] epoll handler too much time :997321 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:07.775258] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:03:06.740575] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=959306][errcode=0] epoll handler too much time :960765 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:07.776361] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1034186] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:06.753775] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24216][errcode=0] epoll handler too much time :24261 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:07.776415] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1021071] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:07.719413] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=18] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:03:07.776438] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=56937] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:03:06.767764] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:06.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:07.776525] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1007218][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:06.767|207=1.01s|210=0us|222=2us|total=1.01s) [2025-08-14 09:03:07.776537] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task use too much time(end=1755162187776536, start=1755162186767763, delta=1008773) [2025-08-14 09:03:07.776544] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=0] task scheduled too delay(scheduled_delay=1021056, expected_run_ts_us_=1755162186755487, ObClockGenerator::getRealClock()=1755162187776544, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:06.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:07.776557] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:06.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:07.776579] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=0] task scheduled too delay(scheduled_delay=687262, expected_run_ts_us_=1755162187089316, ObClockGenerator::getRealClock()=1755162187776578, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:07.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:07.776588] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:07.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:07.776598] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=572300, expected_run_ts_us_=1755162187204297, ObClockGenerator::getRealClock()=1755162187776598, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:07.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:07.776606] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:07.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:07.776770] WDIAG print_alloc_failed_msg (alloc_failed_reason.cpp:146) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=58905726][errcode=-4013] [OOPS]: alloc failed reason is that server memory has reached the upper limit(server_hold: 4293795840, server_limit: 4294967296, alloc_size: 2097152). detailed info: tenant_id=500, ctx_id=0, ctx_name=DEFAULT_CTX_ID, ctx_hold=330993664, ctx_limit=9223372036854775807, tenant_hold=694607872, tenant_limit=9223372036854775807, backtrace=0x26a1d7e0 0x26a180ba 0xf98a0db 0xf989b06 0xf996126 0x93132cc 0x9329467 0x932778f 0x9315ac5 0x27a8a305 0x984531c 0x9844f4c 0xfcd220f 0xfb46d5a 0xfa8f822 0xfa63e1d 0xfa6935a 0x993f3fc 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 [2025-08-14 09:03:07.776841] WDIAG stat_cleanup (debug.c:35) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=0] PNIO [delay_warn] cost too much time: 1005007us, procedure: mod_alloc [2025-08-14 09:03:07.776919] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:07.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:06.779118] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=7] gc stale ls task succ [2025-08-14 09:03:06.762514] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=639017440, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:07.777298] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1013230] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:06.755741] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EECD-0-0] [lt=23] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-08-14 09:03:07.763526] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=37114] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:03:07.777440] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:07.763463] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:03:07.777459] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13970][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:03:07.777473] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:03:07.746783] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=18] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:06.784638] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=53][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:06.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:07.777644] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:07.731786] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:288) [860][T1002_ArcFetche][T1002][YB42AC4D0007-00063C495E399C16-0-0] [lt=979673] ObArchiveFetcher is running(thread_index=0) [2025-08-14 09:03:07.755327] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=14] get number(ret=-4038, service_type_=0, range=1, base_id=1755162187754917000, start_id=0, end_id=140295968341544) [2025-08-14 09:03:07.777850] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=22486][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:03:05.861785] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:05.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:07.777989] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1913289][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:05.861|207=1.92s|210=0us|222=1us|total=1.92s) [2025-08-14 09:03:07.778001] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task use too much time(end=1755162187778000, start=1755162185861783, delta=1916217) [2025-08-14 09:03:06.767776] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14041][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:07.778082] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1008764][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:06.753|207=1.02s|210=1us|222=1us|total=1.02s) [2025-08-14 09:03:07.735126] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B92C0-0-0] [lt=34][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162185881489) [2025-08-14 09:03:07.762897] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:07.778268] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15347][errcode=0] epoll handler too much time :15379 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:07.782319] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=953408] gc stale ls task succ [2025-08-14 09:03:07.782394] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=954700] gc stale ls task succ [2025-08-14 09:03:07.800353] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:07.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:07.804721] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fc8b0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcb90, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:1, time_interval:3.00s, expected_run_time:09:03:07.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:07.835900] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:03:07.835957] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:03:07.836004] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:03:07.836049] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:03:07.875668] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:07.892061] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c0b90, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571c0e70, time_wheel:0x7f9959fef990, thread_pool:0x7f9959fef170, is_running:1, time_interval:3.00s, expected_run_time:09:03:07.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:07.975708] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:07.997730] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:07.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:08.075771] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:08.175855] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:08.275932] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:08.375995] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:08.476070] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:08.577282] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:08.677359] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:08.696322] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:03:08.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:08.719267] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=19] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:08.728336] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:08.731435] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:07.756391] WARN [LIB] alloc (ob_malloc_allocator.cpp:924) [471][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=1991947][errcode=-4013] [OOPS]: alloc failed reason is that server memory has reached the upper limit(server_hold: 4293926912, server_limit: 4294967296, alloc_size: 2097152) [2025-08-14 09:03:07.777627] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:03:08.733004] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=953922][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:03:07.747528] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:08.733088] WDIAG ev_invoke_pending (ev.c:2136) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=984070][errcode=0] epoll handler too much time :985566 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:07.747571] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:07.778091] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] timer task use too much time(end=1755162187778091, start=1755162186753712, delta=1024379) [2025-08-14 09:03:08.733444] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=953904][errcode=0] task scheduled too delay(scheduled_delay=3947582, expected_run_ts_us_=1755162184785859, ObClockGenerator::getRealClock()=1755162188733442, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:08.733466] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:07.780300] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=996342] gc stale ls task succ [2025-08-14 09:03:07.777593] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=991450][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:06.784|207=992.95ms|210=1us|222=1us|total=992.97ms) [2025-08-14 09:03:08.733667] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=954628][errcode=-4389] timer task use too much time(end=1755162188733665, start=1755162186784637, delta=1949028) [2025-08-14 09:03:07.751848] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11126] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:07.878075] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:07.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:07.932033] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=14] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:08.021788] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:08.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:08.733810] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=710944][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:08.217|207=712.02ms|210=1us|222=1us|total=712.03ms) [2025-08-14 09:03:08.733821] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task use too much time(end=1755162188733821, start=1755162188021786, delta=712035) [2025-08-14 09:03:07.785295] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc5a78, timer:0x7f9970f8c420, task:0x564d6e8e31f8, task_type:"N9oceanbase3omt19ObTenantTimezoneMgr15AddTenantTZTaskE", scheduled_time:1755161702763912, pushed_time:0, delay:5000000}, running_task_token={this:0x7f9970fc66b8, timer:0x7f9970f8c420, task:0x564d6e8e3228, task_type:"N9oceanbase3omt19ObTenantTimezoneMgr18UpdateTenantTZTaskE", scheduled_time:1755161702763810, pushed_time:1755161702764275, delay:5000000}, total_delay=485021382, threashold=1000000, tenant_id=500) [2025-08-14 09:03:08.733833] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=0] task scheduled too delay(scheduled_delay=441483, expected_run_ts_us_=1755162188292349, ObClockGenerator::getRealClock()=1755162188733833, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:08.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:07.806085] INFO handle (memory_dump.cpp:671) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=20] statistics: tenant_cnt: 6, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 168 270 0 1 2 110 1 0 1 3 7 0 0 1 4 1 1 0 1 5 86 6 0 1 6 13 0 0 1 7 22 2 0 1 8 53 1 0 1 12 17 3 0 1 13 10 1 0 1 14 1 1 0 1 15 1 1 0 1 16 7 8 0 1 18 4 3 0 1 20 2 1 0 1 22 1 1 0 1 24 1 0 0 1 25 6 0 0 500 0 556 276 0 500 7 192 5 0 500 8 41 1 0 500 9 2 1 0 500 10 10 2 0 500 16 55 4 0 500 17 57 7 0 500 22 38 64 0 500 23 27 10 0 508 0 2 7 0 508 8 5 1 0 509 0 2 7 0 509 8 5 1 0 1001 0 147 253 0 1001 2 26 1 0 1001 3 5 0 0 1001 4 1 1 0 1001 5 85 6 0 1001 6 11 0 0 1001 7 30 2 0 1001 8 36 1 0 1001 12 17 3 0 1001 13 3 1 0 1001 14 1 1 0 1001 16 7 3 0 1001 18 4 3 0 1001 20 4 1 0 1001 25 8 0 0 1002 0 261 278 0 1002 2 198 1 0 1002 3 4 0 0 1002 4 1 1 0 1002 5 101 6 0 1002 6 14 0 0 1002 7 6 2 0 1002 8 73 1 0 1002 12 18 3 0 1002 13 7 1 0 1002 14 1 2 0 1002 16 9 8 0 1002 18 9 3 0 1002 20 2 1 0 1002 21 1 0 0 1002 24 1 0 0 1002 25 7 0 0 cost_time: 1078246 [2025-08-14 09:03:08.237036] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=573364][errcode=0] EASY SLOW: thread: 7f995427d300, lastrun: 1755162187.747520 cost: 0.489515 loop:26555, slowcnt: 4 [2025-08-14 09:03:07.836065] INFO [COMMON] print_io_status (ob_io_manager.cpp:2639) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:2, norm_iops:0, bw:7405, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:12479, total:12485) [2025-08-14 09:03:07.837265] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:07.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:08.017901] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:08.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:08.733846] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:08.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:08.733874] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=926366] [MEMORY] tenant: 1, limit: 2147483648 hold: 1013174272 cache_hold: 56623104 cache_used: 56623104 cache_item_count: 27 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 444993536 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 230686720 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 1163264 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 483328 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 53714944 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 1458176 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 14204928 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 111149056 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35651584 limit= 429496720 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 8003584 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= PS_CACHE_CTX_ID hold_bytes= 65536 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 12730368 limit= 9223372036854775807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 3481600 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3964928 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 212992 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 884736 limit= 9223372036854775807 [2025-08-14 09:03:08.732183] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EECD-0-0] [lt=1020049][errcode=0] cost too much time:mds_tenant_service.cpp:void oceanbase::storage::mds::ObTenantMdsTimer::dump_special_mds_table_status_task(), (this=|threshold=1.00s|start at 09:03:05.740|total=2.99s) [2025-08-14 09:03:08.734509] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EECD-0-0] [lt=2321][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571fcc90, caller:mds_tenant_service.cpp:mtl_start:117, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_145, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcf70, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:0, time_interval:15.00s, expected_run_time:09:03:19.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:08.734526] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EECD-0-0] [lt=15][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:03:08.734|total=3.93s) [2025-08-14 09:03:08.734540] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EECD-0-0] [lt=8][errcode=-4016] task executed too delay(delay_time=3932638, *p_task_={this:0x7f99571fc8b0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcb90, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:0, time_interval:3.00s, expected_run_time:09:03:10.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:08.733749] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=980416] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:08.736369] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2615] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:08.736384] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:08.736391] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:08.736398] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:08.736404] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:08.736411] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:08.736414] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:08.736418] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:08.733774] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=854402][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:07.878|207=855.70ms|210=0us|222=2us|total=855.71ms) [2025-08-14 09:03:08.736440] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=2661][errcode=-4389] timer task use too much time(end=1755162188736439, start=1755162187878073, delta=858366) [2025-08-14 09:03:08.736497] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:08.736585] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:08.733478] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=3948317, expected_run_ts_us_=1755162184785160, ObClockGenerator::getRealClock()=1755162188733477, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:08.737492] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=4006][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:08.737515] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] task scheduled too delay(scheduled_delay=3933436, expected_run_ts_us_=1755162184804079, ObClockGenerator::getRealClock()=1755162188737515, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919836b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:08.733687] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] task scheduled too delay(scheduled_delay=878417, expected_run_ts_us_=1755162187855269, ObClockGenerator::getRealClock()=1755162188733687, ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:07.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:08.733903] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=896465] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, param_config_:{memory_limit:536870912, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=865568, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:03:08.733941] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=714955][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:08.178|207=716.04ms|210=0us|222=1us|total=716.05ms) [2025-08-14 09:03:08.733919] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=895297][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:07.837|207=896.65ms|210=0us|222=2us|total=896.67ms) [2025-08-14 09:03:08.738603] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=4675][errcode=-4389] timer task use too much time(end=1755162188738602, start=1755162187837263, delta=901339) [2025-08-14 09:03:08.249464] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:08.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:08.738553] INFO [PALF] register_parent_ (log_config_mgr.cpp:2707) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=7] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.77.0.7:2882", reason=1, leader="172.77.0.7:2882", register_time_us=1755162183747293, last_submit_register_req_time_us=1755162183747293, register_parent_reason=1) [2025-08-14 09:03:08.738591] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=4643][errcode=-4389] timer task use too much time(end=1755162188738591, start=1755162188017899, delta=720692) [2025-08-14 09:03:08.738896] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=302][errcode=0] task scheduled too delay(scheduled_delay=733408, expected_run_ts_us_=1755162188005487, ObClockGenerator::getRealClock()=1755162188738896, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:08.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:07.757803] EDIAG [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1835][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1004082, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1004081, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x9c6174d 0xfd347d6 0xfd344b7 0xfd34254 0x1c029398 0x97a1ad0 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:07.756474] EDIAG [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=1933869][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1018900, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1017394, ObMalloc#start=7, ObMalloc#end=32, ObMalloc#start=18, ObMalloc#end=30, FREE_END=1419], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x9c6174d 0xfd347d6 0xfd344b7 0xfd34254 0x1c029398 0x97a1ad0 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:07.752683] EDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=970367][errcode=-4013] LOGGER COST TOO MUCH TIME, cost: 1023534, owner: ObLog, click_count: 3, time dist:[FORMAT_END=984953, ALLOC_END=38580, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0xfd2ef08 0xfd2bd1d 0xf9d16a8 0xf9d4150 0xfa09ca2 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:08.737526] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919836b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:08.739055] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1523][errcode=0] task scheduled too delay(scheduled_delay=3908655, expected_run_ts_us_=1755162184830400, ObClockGenerator::getRealClock()=1755162188739055, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:08.739066] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:08.739075] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=3908050, expected_run_ts_us_=1755162184831025, ObClockGenerator::getRealClock()=1755162188739075, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.831, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:08.737540] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=3845][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:07.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:08.736426] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:03:08.757098] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=30390] PNIO [ratelimit] time: 1755162188757095, bytes: 0, bw: 0.000000 MB/s, add_ts: 1009432, add_bytes: 0 [2025-08-14 09:03:08.738912] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:08.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:06.741446] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=3935982] current monitor number(seq_=-1) [2025-08-14 09:03:08.757982] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:08.742485] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6048] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:03:07.757803] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1835] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=467982621, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=252320, succ_cluster_heartbeat_count=1905, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161003593738002, v:0}, local_cluster_delta=1184164062, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:03:08.763478] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1004082] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119797572, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755162138745740) [2025-08-14 09:03:08.763996] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=468988601, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:08.764010] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:08.764532] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=30] get wrs ts(ls_id={id:1}, delta=468988601, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:08.764542] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:07.779324] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=640040806, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=253174, succ_cluster_heartbeat_count=2211, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161000606550000, v:0}, local_cluster_delta=1187172772, force_self_check=true, weak_read_refresh_interval=100000) [2025-08-14 09:03:08.764638] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=983796] get_min_unreplayed_log_info(lsn={lsn:5282737655}, scn={val:1755100947825033001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161624848121, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:13063726, ret_code_:0}}, next_to_submit_lsn_:{lsn:5282737655}, next_to_submit_scn_:{val:1755100947825033001, v:0}, base_lsn_:{lsn:4831543296}, base_scn_:{val:1755063445733337001, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7605000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:5282737655}, end_lsn:{lsn:5282737655}, read_buf:{buf_len_:3747840, buf_:0x7f98f7605000}, block_size:67104768, log_storage_:0x7f993e529bf0, log_storage_:{log_tail:{lsn:5282737655}, readable_log_tail:{lsn:5282737655}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:5234171904}, min_scn:{val:1755152244802535004, v:0}, curr_block_id:78, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:206, min_block_id:42, max_block_id:79, curr_writable_block_id:78}, logical_block_size_:67104768, curr_block_writable_size_:18539017, block_header_serialize_buf_:0x7f993e52a8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:85, read_io_size:301367296, read_disk_cost_ts:2364604, start_lsn:{lsn:4831543296}}}}}}) [2025-08-14 09:03:08.765175] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=80] get wrs ts(ls_id={id:1}, delta=641026339, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:08.765185] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:08.765656] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1}, delta=641026339, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:08.765664] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:08.766132] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] get wrs ts(ls_id={id:1}, delta=641026339, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:08.766139] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:08.771771] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=398][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:08.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:08.757719] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=18771][errcode=0] task scheduled too delay(scheduled_delay=668400, expected_run_ts_us_=1755162188089316, ObClockGenerator::getRealClock()=1755162188757718, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:08.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:08.771910] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14168][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:08.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:08.771925] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=567627, expected_run_ts_us_=1755162188204297, ObClockGenerator::getRealClock()=1755162188771924, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:08.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:08.771934] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:08.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:08.771953] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc190, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc470, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:10.00s, expected_run_time:09:03:08.720, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:08.771964] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:08.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:08.772000] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:08.734181] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=1593] finish to resize log external storage handler(current_ts=1755162187773948, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-08-14 09:03:08.752959] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:08.775658] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:08.757562] INFO [PALF] check_parent_health (log_config_mgr.cpp:2884) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=336] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.77.0.7:2882") [2025-08-14 09:03:08.739084] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.831, task_priority:1, with_handle_protected:0}) [857]OB_MALLOC COST TOO MUCH TIME, cost_time=959428, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [2025-08-14 09:03:07.763534] EDIAG [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=1978940][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1978196, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1978195, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0xfd250b6 0xfd247d3 0xfd244ea 0xfd242a2 0x1d19629b 0x1d17baa3 0x1d17ad94 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [357]OB_MALLOC COST TOO MUCH TIME, cost_time=968126, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=6052], seq:[0, 1] [2025-08-14 09:03:08.733991] EDIAG [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=20753][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1009275, owner: ObLog, click_count: 5, time dist:[FORMAT_END=41138, ObMalloc#start=1, ObMalloc#end=968135, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x18bdb084 0x18bda6b1 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:08.732248] EDIAG [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=5604][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1011047, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1002392, ALLOC_END=8654, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1fce4e91 0x1fce455d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:07.756742] EDIAG [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=1943996][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1985918, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1985917, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0xfd250b6 0xfd247d3 0xfd244ea 0xfd242a2 0x1d19629b 0x1d17baa3 0x1d17ad94 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [397]OB_MALLOC COST TOO MUCH TIME, cost_time=980141, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=980132], seq:[0] [2025-08-14 09:03:08.776852] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=37706][errcode=0] task scheduled too delay(scheduled_delay=3914718, expected_run_ts_us_=1755162184862133, ObClockGenerator::getRealClock()=1755162188776851, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.862, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:08.776872] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:04.862, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:07.752683] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=970367][errcode=-4013] LogRpc post_request failed(ret=-4013, palf_id_=1, req={this:0x7f9930c55e20, BASE:{msg_type:"Prepare Request", id:1, sender:"172.77.0.7:2882", receiver:"172.77.0.7:2882", restart_counter:1, ballot_number:65, debug_ts:{src_construct_ts:"03:03.745942", src_serialize_ts:"00:00.0", dest_deserialize_ts:"00:00.0", dest_process_ts:"00:00.0", process_delay:-1755162183745942}, biggest_min_cluster_version_ever_seen:4.3.5.2}, role:"Follower", is_buffer_valid:true, inner_priority_seed:4096, membership_version:{proposal_id:45, config_seq:42}}, server="172.77.0.7:2882") [2025-08-14 09:03:08.777813] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:08.777835] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=30670] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:08.740223] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EECF-0-0] [lt=10] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-08-14 09:03:08.734222] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B92C0-0-0] [lt=43012][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162185881489}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162188765674}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162188765674}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:03:08.753975] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:08.779349] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=624] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 444993536 used= 389633888 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 70 washed_blocks= 287 washed_size= 70221824 [MEMORY] request_cached_chunk_cnt= 4 [MEMORY] hold= 389633888 used= 379387395 count= 32872 avg_used= 11541 mod=SUMMARY [2025-08-14 09:03:08.779377] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 230686720 used= 228884480 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 block_cnt= 110 chunk_cnt= 110 mod=Memstore [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:03:08.779395] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1 ctx_id= EXECUTE_CTX_ID hold= 1163264 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 15 washed_size= 13516800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:03:08.779407] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:03:08.779420] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 53714944 used= 47487424 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 84 washed_blocks= 334 washed_size= 126640128 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 47487424 used= 43098388 count= 3994 avg_used= 10790 mod=SUMMARY [2025-08-14 09:03:08.776894] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=3038836, expected_run_ts_us_=1755162185738058, ObClockGenerator::getRealClock()=1755162188776894, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:08.779684] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=2784][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:08.779715] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] task scheduled too delay(scheduled_delay=3048278, expected_run_ts_us_=1755162185731436, ObClockGenerator::getRealClock()=1755162188779715, ret=0, ret="OB_SUCCESS", *this={this:0x7f98918114b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:08.779734] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918114b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:08.737632] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2826) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3717] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4080, [delay/us]: prepare:0, schedule:0, submit:600000000, device:12479, total:12485) [2025-08-14 09:03:08.779796] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=42084] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:03:08.779818] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:03:08.779827] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:03:08.778793] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:08.778774] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:08.780376] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=951757] gc stale ls task succ [2025-08-14 09:03:08.779274] WDIAG ev_invoke_pending (ev.c:2136) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25260][errcode=0] epoll handler too much time :25308 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:08.734060] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [628][T1_L0_G0][T1][YB42AC4D0007-00063C49607B92BE-0-0] [lt=21334][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162182765011}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162188765674}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162188765674}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:03:08.779431] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1 ctx_id= WORK_AREA hold= 1458176 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 13 washed_blocks= 66 washed_size= 25804800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:03:08.782789] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=3341] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 14204928 used= 10970224 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 277 washed_size= 31932416 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10970224 used= 6078924 count= 18573 avg_used= 327 mod=SUMMARY [2025-08-14 09:03:08.782804] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 111149056 used= 108896256 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 block_cnt= 211 chunk_cnt= 53 mod=CoStack [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 mod=SUMMARY [2025-08-14 09:03:08.782820] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 56623104 used= 56623104 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 56623104 used= 0 count= 27 avg_used= 0 mod=SUMMARY [2025-08-14 09:03:08.782832] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 34365376 limit= 429496720 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 34365376 used= 31086608 count= 475 avg_used= 65445 mod=SUMMARY [2025-08-14 09:03:08.779744] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=3045292, expected_run_ts_us_=1755162185734452, ObClockGenerator::getRealClock()=1755162188779744, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728aa90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:08.782896] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=3146][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728aa90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:08.782923] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] task scheduled too delay(scheduled_delay=3052106, expected_run_ts_us_=1755162185730817, ObClockGenerator::getRealClock()=1755162188782923, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:08.782935] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:08.782946] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=3050544, expected_run_ts_us_=1755162185732401, ObClockGenerator::getRealClock()=1755162188782945, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:08.801960] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:08.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:08.811986] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40b7f0, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40bad0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:10.00s, expected_run_time:09:03:08.802, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:08.833745] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1553][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:08.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:08.839281] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] [sql nio session](*s={this:0x7f98f72d0030, session_id:3221589415, trace_id:YB42AC4D0007-00063C4965FA43DC-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:245}, err:0, last_decode_time:1755161751714366, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1662, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:03:08.839309] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=27] [sql nio session](*s={this:0x7f98f72d5030, session_id:3221590200, trace_id:YB42AC4D0007-00063C4965FA43E0-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:248}, err:0, last_decode_time:1755161776777167, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1593, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:03:08.877893] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:08.933819] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:08.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:08.977956] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:09.033880] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:08.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:09.078018] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:09.133953] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:09.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:09.233140] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:03:09.719841] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=16] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:09.729227] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:08.880031] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:03:09.736225] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=854850] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:03:09.736283] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:03:09.736337] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:03:09.736374] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:03:08.934306] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=800553] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:09.736565] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=72] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:09.736590] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:08.782843] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 8003584 used= 4947392 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 10 washed_blocks= 23 washed_size= 12967936 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4947392 used= 4793344 count= 604 avg_used= 7936 mod=SUMMARY [2025-08-14 09:03:09.738062] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=953756] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:03:09.738083] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=14] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 65536 used= 31024 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 2031616 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 31024 used= 27058 count= 20 avg_used= 1352 mod=SUMMARY [2025-08-14 09:03:09.738103] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1 ctx_id= RPC_CTX_ID hold= 12730368 used= 4538368 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4538368 used= 2990336 count= 190 avg_used= 15738 mod=SUMMARY [2025-08-14 09:03:08.838688] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:08.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:09.233174] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:03:09.738562] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=504620][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:03:09.233984] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:09.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:09.738599] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=503849][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:09.233|207=504.61ms|210=0us|222=1us|total=504.62ms) [2025-08-14 09:03:09.738610] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task use too much time(end=1755162189738609, start=1755162189233983, delta=504626) [2025-08-14 09:03:08.782956] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.738822] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=954416][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:08.782|207=955.87ms|210=1us|222=1us|total=955.90ms) [2025-08-14 09:03:09.738848] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4389] timer task use too much time(end=1755162189738847, start=1755162188782945, delta=955902) [2025-08-14 09:03:09.738859] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=4005421, expected_run_ts_us_=1755162185733437, ObClockGenerator::getRealClock()=1755162189738859, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:08.836517] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:08.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:09.014659] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:09.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:09.023217] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:09.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:09.178251] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:09.739178] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=560075][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:09.245320] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=1589096][errcode=0] EASY SLOW: thread: 7f9953e7d300, lastrun: 1755162187.747563 cost: 1.497755 loop:26556, slowcnt: 14 [2025-08-14 09:03:09.699475] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:03:09.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:09.735954] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:09.727258] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:09.739317] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12041][errcode=0] epoll handler too much time :12070 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:07.763534] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=1978940] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:03:08.733991] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=20753] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:03:08.732248] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=5604] blacklist_loop exec finished(cost_time=30, is_enabled=true, send_cnt=0) [2025-08-14 09:03:09.738521] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=898466][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:08.838|207=899.83ms|210=0us|222=2us|total=899.86ms) [2025-08-14 09:03:09.744875] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=6342][errcode=-4389] timer task use too much time(end=1755162189744874, start=1755162188838687, delta=906187) [2025-08-14 09:03:07.756742] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=1943996] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:03:09.738117] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 3481600 used= 2391232 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 10 washed_size= 4907008 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2391232 used= 2367040 count= 36 avg_used= 65751 mod=SUMMARY [2025-08-14 09:03:09.745751] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7603] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 3964928 used= 3080192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 3 washed_size= 229376 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3080192 used= 2983936 count= 376 avg_used= 7936 mod=SUMMARY [2025-08-14 09:03:09.745772] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1 ctx_id= SCHEMA_SERVICE hold= 212992 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1884160 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:03:09.745787] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:03:09.745800] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= MERGE_NORMAL_CTX_ID hold= 884736 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 6 washed_blocks= 9 washed_size= 11698176 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:03:09.736391] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=1, sync_total=0, async_cnt= 1 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:03:09.746213] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9800] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:03:09.746366] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2172] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:09.747780] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2750] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:09.738584] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:03:09.738642] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=40957] snapshot_traffic_info once [2025-08-14 09:03:09.738878] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.754046] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15139][errcode=0] task scheduled too delay(scheduled_delay=4012417, expected_run_ts_us_=1755162185741627, ObClockGenerator::getRealClock()=1755162189754045, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.754071] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.754085] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=4010533, expected_run_ts_us_=1755162185743552, ObClockGenerator::getRealClock()=1755162189754085, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:08.820652] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=954838] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:03:08.820655] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=1960371] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:03:09.739107] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=901223][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:08.836|207=902.59ms|210=0us|222=1us|total=902.60ms) [2025-08-14 09:03:09.739139] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=723382][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:09.146|207=724.48ms|210=0us|222=2us|total=724.49ms) [2025-08-14 09:03:09.739158] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=714856][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:09.232|207=715.94ms|210=0us|222=1us|total=715.95ms) [2025-08-14 09:03:09.754173] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=14993][errcode=-4389] timer task use too much time(end=1755162189754173, start=1755162189023215, delta=730958) [2025-08-14 09:03:09.754194] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=461844, expected_run_ts_us_=1755162189292349, ObClockGenerator::getRealClock()=1755162189754194, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:09.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:09.754206] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:09.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:09.755087] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:09.755673] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=789] get wrs ts(ls_id={id:1001}, delta=445027714, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:09.755696] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:09.755738] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=35673][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fb3b0, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fb690, time_wheel:0x7f99589fd990, thread_pool:0x7f9959fff170, is_running:1, time_interval:2.00s, expected_run_time:09:03:09.698, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:09.759422] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=2013380] current monitor number(seq_=-1) [2025-08-14 09:03:09.761247] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=17] PNIO [ratelimit] time: 1755162189761245, bytes: 0, bw: 0.000000 MB/s, add_ts: 1004150, add_bytes: 0 [2025-08-14 09:03:09.763487] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:09.750456] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3615] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:09.764890] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14410][errcode=0] epoll handler too much time :14443 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:09.754097] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765273] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11155][errcode=0] task scheduled too delay(scheduled_delay=4022117, expected_run_ts_us_=1755162185743155, ObClockGenerator::getRealClock()=1755162189765273, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765300] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765315] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=4008416, expected_run_ts_us_=1755162185756899, ObClockGenerator::getRealClock()=1755162189765315, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fc830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765329] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fc830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765344] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] task scheduled too delay(scheduled_delay=4009156, expected_run_ts_us_=1755162185756188, ObClockGenerator::getRealClock()=1755162189765344, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765358] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765371] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=4009051, expected_run_ts_us_=1755162185756319, ObClockGenerator::getRealClock()=1755162189765371, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a54b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765390] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a54b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765403] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=4004856, expected_run_ts_us_=1755162185760547, ObClockGenerator::getRealClock()=1755162189765403, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72edc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765419] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72edc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765432] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=4000214, expected_run_ts_us_=1755162185765218, ObClockGenerator::getRealClock()=1755162189765432, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72edb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765444] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72edb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765456] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=3999334, expected_run_ts_us_=1755162185766122, ObClockGenerator::getRealClock()=1755162189765456, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765470] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765483] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=3996849, expected_run_ts_us_=1755162185768633, ObClockGenerator::getRealClock()=1755162189765482, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765497] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765509] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=4004511, expected_run_ts_us_=1755162185760998, ObClockGenerator::getRealClock()=1755162189765509, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765521] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765536] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=3994701, expected_run_ts_us_=1755162185770834, ObClockGenerator::getRealClock()=1755162189765536, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765548] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765560] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=3986221, expected_run_ts_us_=1755162185779339, ObClockGenerator::getRealClock()=1755162189765560, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728ac10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765572] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728ac10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765585] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=3962680, expected_run_ts_us_=1755162185802905, ObClockGenerator::getRealClock()=1755162189765585, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.802, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765599] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.802, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765612] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=3964137, expected_run_ts_us_=1755162185801474, ObClockGenerator::getRealClock()=1755162189765612, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765624] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765637] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=3961326, expected_run_ts_us_=1755162185804310, ObClockGenerator::getRealClock()=1755162189765637, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728f9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765649] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728f9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765664] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=3947123, expected_run_ts_us_=1755162185818540, ObClockGenerator::getRealClock()=1755162189765663, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eae10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765676] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eae10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765688] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=3942781, expected_run_ts_us_=1755162185822907, ObClockGenerator::getRealClock()=1755162189765688, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765700] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765723] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] task scheduled too delay(scheduled_delay=3900365, expected_run_ts_us_=1755162185865358, ObClockGenerator::getRealClock()=1755162189765723, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765737] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765749] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=3898976, expected_run_ts_us_=1755162185866773, ObClockGenerator::getRealClock()=1755162189765749, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.866, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765762] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:05.866, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765780] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=3436680, expected_run_ts_us_=1755162186329099, ObClockGenerator::getRealClock()=1755162189765780, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765792] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765849] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=48][errcode=0] task scheduled too delay(scheduled_delay=3040008, expected_run_ts_us_=1755162186725840, ObClockGenerator::getRealClock()=1755162189765849, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72756b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765861] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72756b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765871] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=3041737, expected_run_ts_us_=1755162186724133, ObClockGenerator::getRealClock()=1755162189765870, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.724, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765882] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.724, task_priority:1, with_handle_protected:0}) [729]OB_MALLOC COST TOO MUCH TIME, cost_time=901028, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=900614, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:03:06.771696] EDIAG [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=2047383][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2993984, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2993982, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b51df0f 0x986bcde 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:09.765893] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=3036520, expected_run_ts_us_=1755162186729372, ObClockGenerator::getRealClock()=1755162189765893, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a36b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765905] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a36b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765928] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] task scheduled too delay(scheduled_delay=3027679, expected_run_ts_us_=1755162186738249, ObClockGenerator::getRealClock()=1755162189765928, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919bfbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aa690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:07.763499] EDIAG [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=1986591][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2004320, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1988155, ALLOC_END=16164, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0xfd250b6 0xfd247d3 0xfd244ea 0xfd242a2 0x1d19629b 0x1d17baa3 0x1d17ad94 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:09.765939] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919bfbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aa690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765948] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=3027041, expected_run_ts_us_=1755162186738906, ObClockGenerator::getRealClock()=1755162189765947, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919911b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196cc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765956] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919911b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196cc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765965] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=3027672, expected_run_ts_us_=1755162186738292, ObClockGenerator::getRealClock()=1755162189765965, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d40b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765973] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d40b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765981] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=3031573, expected_run_ts_us_=1755162186734408, ObClockGenerator::getRealClock()=1755162189765981, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765989] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.765998] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=3027868, expected_run_ts_us_=1755162186738130, ObClockGenerator::getRealClock()=1755162189765998, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.766006] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.766015] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=3027728, expected_run_ts_us_=1755162186738286, ObClockGenerator::getRealClock()=1755162189766014, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.766023] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.766031] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=3027512, expected_run_ts_us_=1755162186738519, ObClockGenerator::getRealClock()=1755162189766031, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.766039] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.766049] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=3021524, expected_run_ts_us_=1755162186744524, ObClockGenerator::getRealClock()=1755162189766049, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.744, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.766057] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.744, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.766065] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=3023977, expected_run_ts_us_=1755162186742088, ObClockGenerator::getRealClock()=1755162189766065, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aaf90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.766073] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aaf90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.766082] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=3025780, expected_run_ts_us_=1755162186740301, ObClockGenerator::getRealClock()=1755162189766082, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aba10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.766096] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aba10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.766104] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=3020644, expected_run_ts_us_=1755162186745459, ObClockGenerator::getRealClock()=1755162189766104, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.766112] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.766121] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=3018078, expected_run_ts_us_=1755162186748042, ObClockGenerator::getRealClock()=1755162189766120, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196cb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.766131] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196cb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.766140] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=3022968, expected_run_ts_us_=1755162186743171, ObClockGenerator::getRealClock()=1755162189766140, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.766148] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.766156] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=3014209, expected_run_ts_us_=1755162186751947, ObClockGenerator::getRealClock()=1755162189766156, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.751, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.766166] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.751, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.766177] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=3002635, expected_run_ts_us_=1755162186763541, ObClockGenerator::getRealClock()=1755162189766177, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.766187] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.766196] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=2987080, expected_run_ts_us_=1755162186779116, ObClockGenerator::getRealClock()=1755162189766196, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919acf10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:06.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.766205] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:09.754140] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=15009][errcode=-4389] timer task use too much time(end=1755162189754139, start=1755162188836515, delta=917624) [2025-08-14 09:03:09.767052] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1}, delta=469987020, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:09.767092] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=37] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:09.767112] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [WRS] update tenant weak read server version(tenant_id=1001, server_version={version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162189766552}, version_delta=469991932) [2025-08-14 09:03:09.767129] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119797572, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755162138745740) [2025-08-14 09:03:09.767669] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=469987020, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:09.767679] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:09.768262] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] get wrs ts(ls_id={id:1}, delta=642023683, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:09.768275] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:09.768823] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=642023683, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:09.768832] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:09.754162] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15000][errcode=-4389] timer task use too much time(end=1755162189754162, start=1755162189014657, delta=739505) [2025-08-14 09:03:09.768857] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14671][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:06.771696] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=2047383] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136829134139001, v:0}, clog_base_lsn:{lsn:8186781696}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1753840867372367034, v:0}, all_id_meta:{id_meta:[{limited_id:1755161729775179000, latest_log_ts:{val:1755161719775179001, v:0}}, {limited_id:18000001, latest_log_ts:{val:1755157440846556001, v:0}}, {limited_id:60000001, latest_log_ts:{val:1755135657266496001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:46, palf_env_:0x7f99321fa030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [397]OB_MALLOC COST TOO MUCH TIME, cost_time=1000773, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=47, WASH_BLOCK_END=72, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=980155], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:08.772456] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1000860, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=27, FORMAT_END=36, ObMalloc#start=1, ObMalloc#end=1000789, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:06.771367] EDIAG [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=2047331][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3004686, owner: ObLog, click_count: 6, time dist:[FORMAT_END=3004486, ObMalloc#start=1, ObMalloc#end=70, ObMalloc#start=22, ObMalloc#end=36, FREE_END=71], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b51df0f 0x986bcde 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [660]OB_MALLOC COST TOO MUCH TIME, cost_time=981273, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=981271], seq:[0] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=986900, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=986894], seq:[0] [2025-08-14 09:03:09.754624] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 500, limit: 9223372036854775807 hold: 696401920 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 330993664 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 34881536 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 85983232 limit= 9223372036854775807 [MEMORY] ctx_id= LIBEASY hold_bytes= 3145728 limit= 9223372036854775807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 23068672 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 2981888 limit= 9223372036854775807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 37437440 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 78925824 limit= 9223372036854775807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 98983936 limit= 9223372036854775807 [2025-08-14 09:03:09.770755] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=16077] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 330993664 used= 301374768 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 495 washed_blocks= 2221 washed_size= 914604032 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 301374768 used= 293948813 count= 12682 avg_used= 23178 mod=SUMMARY [2025-08-14 09:03:09.770779] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= GLIBC hold= 35241984 used= 21959152 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 183 washed_blocks= 811 washed_size= 361119744 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 21959152 used= 10362901 count= 131698 avg_used= 78 mod=SUMMARY [2025-08-14 09:03:09.770792] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 500 ctx_id= CO_STACK hold= 85983232 used= 83091456 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 83091456 used= 82936896 count= 161 avg_used= 515136 block_cnt= 161 chunk_cnt= 41 mod=CoStack [MEMORY] hold= 83091456 used= 82936896 count= 161 avg_used= 515136 mod=SUMMARY [2025-08-14 09:03:09.770803] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 500 ctx_id= LIBEASY hold= 3145728 used= 3070816 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1048576 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3070816 used= 2859784 count= 223 avg_used= 12824 mod=SUMMARY [2025-08-14 09:03:09.770813] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 500 ctx_id= LOGGER_CTX_ID hold= 16777216 used= 18726912 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 18726912 used= 18717696 count= 9 avg_used= 2079744 mod=SUMMARY [2025-08-14 09:03:07.763499] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=1986591] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:03:08.772456] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [416]OB_MALLOC COST TOO MUCH TIME, cost_time=691337, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 16, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=691245, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=997158, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=96, WASH_BLOCK_END=95, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=12570], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:09.720407] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [663][T1001_StandbySc][T1001][YB42AC4D0007-00063C495C7A032C-0-0] [lt=1015][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1044325, , BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [567]OB_MALLOC COST TOO MUCH TIME, cost_time=1857591, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [729]OB_MALLOC COST TOO MUCH TIME, cost_time=1852802, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [857]OB_MALLOC COST TOO MUCH TIME, cost_time=1872846, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1], seq:[0, 1] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=1960047, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=986909], seq:[0, 1] [2025-08-14 09:03:09.770952] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1960070, owner: ObLog, click_count: 5, time dist:[FORMAT_END=7, ObMalloc#start=0, ObMalloc#end=1960062, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [660]OB_MALLOC COST TOO MUCH TIME, cost_time=1954679, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=981292], seq:[0, 1] [2025-08-14 09:03:09.776375] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1954725, owner: ObLog, click_count: 5, time dist:[FORMAT_END=34, ObMalloc#start=1, ObMalloc#end=1954686, ALLOC_END=3, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [386]OB_MALLOC COST TOO MUCH TIME, cost_time=955676, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=955666, ALLOC_CHUNK_END=2], seq:[0, 1] [386]OB_MALLOC COST TOO MUCH TIME, cost_time=955696, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=955695], seq:[0, 1] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=1954982, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1954974], seq:[0] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=1955003, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1954998], seq:[0, 1] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=954246, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=954217, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [2025-08-14 09:03:09.774853] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1000860][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:09.774870] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:09.770831] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 500 ctx_id= RPC_CTX_ID hold= 2981888 used= 1433600 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 55 washed_blocks= 68 washed_size= 112361472 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1433600 used= 946816 count= 59 avg_used= 16047 mod=SUMMARY [2025-08-14 09:03:09.777429] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6576] [MEMORY] tenant_id= 500 ctx_id= PKT_NIO hold= 37437440 used= 35915264 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 47 washed_blocks= 68 washed_size= 92872704 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 35915264 used= 34952832 count= 121 avg_used= 288866 mod=SUMMARY [2025-08-14 09:03:09.777454] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= SCHEMA_SERVICE hold= 78925824 used= 68920960 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 22 washed_size= 5767168 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 68920960 used= 67658879 count= 4858 avg_used= 13927 mod=SUMMARY [2025-08-14 09:03:09.777466] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 500 ctx_id= UNEXPECTED_IN_500 hold= 98983936 used= 97174864 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 97174864 used= 86783232 count= 3014 avg_used= 28793 mod=SUMMARY [2025-08-14 09:03:09.777479] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant: 508, limit: 1073741824 hold: 15613952 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 5128192 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10485760 limit= 9223372036854775807 [2025-08-14 09:03:09.777489] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 508 ctx_id= DEFAULT_CTX_ID hold= 5128192 used= 3765328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3765328 used= 3723912 count= 84 avg_used= 44332 mod=SUMMARY [2025-08-14 09:03:09.777512] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 508 ctx_id= CO_STACK hold= 10485760 used= 10321920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10321920 used= 10302720 count= 20 avg_used= 515136 mod=SUMMARY [2025-08-14 09:03:09.780291] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=4627] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:09.785558] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=220] gc stale ls task succ [2025-08-14 09:03:09.792520] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=173] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:09.792531] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162189792507], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:09.792543] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162189792507], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:09.792555] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162189792507]}) [2025-08-14 09:03:09.792563] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:03:09.792570] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:03:09.792577] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=5] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:09.777551] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant: 509, limit: 1073741824 hold: 15613952 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 5128192 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10485760 limit= 9223372036854775807 [2025-08-14 09:03:10.720212] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=941219] [MEMORY] tenant_id= 509 ctx_id= DEFAULT_CTX_ID hold= 5128192 used= 3765328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3765328 used= 3723912 count= 84 avg_used= 44332 mod=SUMMARY [2025-08-14 09:03:10.720258] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=27] [MEMORY] tenant_id= 509 ctx_id= CO_STACK hold= 10485760 used= 10321920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10321920 used= 10302720 count= 20 avg_used= 515136 mod=SUMMARY [2025-08-14 09:03:10.728310] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1925] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:10.750704] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=50] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:09.792582] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162189792576], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:10.754814] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=960771] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162189792576]}) [2025-08-14 09:03:10.754833] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162189792576]}) [2025-08-14 09:03:10.755018] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:03:10.755027] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:03:10.755037] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162190755011], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:10.755050] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162190755011], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:10.755059] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162190755011]}) [2025-08-14 09:03:09.936573] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=800885] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:10.720315] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 1001, limit: 1073741824 hold: 608747520 cache_hold: 44040192 cache_used: 44040192 cache_item_count: 21 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 302309376 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 54525952 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 3080192 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 483328 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 45514752 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 5521408 limit= 157286400 [MEMORY] ctx_id= GLIBC hold_bytes= 15499264 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 73424896 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35651584 limit= 214748360 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 3252224 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 14680064 limit= 1073741800 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 5570560 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3866624 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 1179648 limit= 9223372036854775807 [2025-08-14 09:03:10.759248] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=38831] [MEMORY] tenant_id= 1001 ctx_id= DEFAULT_CTX_ID hold= 302309376 used= 268994592 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 81 washed_blocks= 406 washed_size= 113410048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 268994592 used= 262877846 count= 3788 avg_used= 69397 mod=SUMMARY [2025-08-14 09:03:10.759264] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1001 ctx_id= MEMSTORE_CTX_ID hold= 54525952 used= 54099968 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 54099968 used= 54075216 count= 26 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:03:10.759272] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= EXECUTE_CTX_ID hold= 3080192 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 8 washed_size= 7405568 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:03:10.759281] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1001 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:03:10.759414] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [719][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571f9d70, caller:ob_table_lock_service.cpp:start:276, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE6$_1273, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa090, time_wheel:0x7f99375f7d10, thread_pool:0x7f99375f74f0, is_running:0, time_interval:10.00s, expected_run_time:09:03:12.332, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:10.759465] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [719][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=50][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:03:10.759|total=8.28s) [2025-08-14 09:03:10.736520] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:10.736501] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [321]OB_MALLOC COST TOO MUCH TIME, cost_time=954379, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=954263, WASH_BLOCK_END=112, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=2], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:10.736266] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3347] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:10.066720] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=287][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:09.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:10.759800] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=836][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739be30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:10.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:10.759821] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:10.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:10.759833] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7edc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:10.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:10.759844] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ebe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:10.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:10.759855] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:10.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:10.745311] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:10.763014] WDIAG ev_invoke_pending (ev.c:2136) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17675][errcode=0] epoll handler too much time :17718 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:10.764467] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=274][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:10.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:10.764497] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:10.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:10.759322] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= PLAN_CACHE_CTX_ID hold= 45514752 used= 36449984 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 77 washed_blocks= 201 washed_size= 132743168 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 36449984 used= 32916258 count= 3312 avg_used= 9938 mod=SUMMARY [2025-08-14 09:03:10.767580] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8234] [MEMORY] tenant_id= 1001 ctx_id= WORK_AREA hold= 5521408 used= 0 limit= 157286400 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 9 washed_blocks= 40 washed_size= 17547264 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:03:10.767593] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1001 ctx_id= GLIBC hold= 15499264 used= 10889728 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 25 washed_blocks= 164 washed_size= 47415296 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10889728 used= 6020141 count= 18313 avg_used= 328 mod=SUMMARY [2025-08-14 09:03:10.767604] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1001 ctx_id= CO_STACK hold= 73424896 used= 71221248 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 32 washed_blocks= 34 washed_size= 2072576 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 71221248 used= 71088768 count= 138 avg_used= 515136 block_cnt= 138 chunk_cnt= 36 mod=CoStack [MEMORY] hold= 71221248 used= 71088768 count= 138 avg_used= 515136 mod=SUMMARY [2025-08-14 09:03:10.767617] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= KVSTORE_CACHE_ID hold= 44040192 used= 44040192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 44040192 used= 0 count= 21 avg_used= 0 mod=SUMMARY [2025-08-14 09:03:10.768188] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:10.750, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:10.768213] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:10.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:10.768224] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728d1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7291110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:10.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:10.778256] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aa990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:10.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:10.778282] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:10.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:10.778294] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919afe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:10.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:10.778315] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d62b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:10.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:10.778329] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:10.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:10.779616] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:10.780750] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=301] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:10.759501] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22964][errcode=0] epoll handler too much time :23007 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:10.759489] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22933][errcode=0] epoll handler too much time :22974 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:10.759511] WDIAG ev_invoke_pending (ev.c:2136) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23210][errcode=0] epoll handler too much time :23254 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:10.788325] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:10.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:10.788351] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:10.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:10.788363] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:10.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:10.788383] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7397a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:10.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:10.802952] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:10.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:10.802976] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:10.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:10.802986] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:10.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:10.802997] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:10.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:10.822359] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:10.802, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:10.822384] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:10.814, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:10.822418] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:10.817, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:10.822429] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:10.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:10.822439] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:10.819, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:10.822449] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:10.813, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:10.833349] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919831b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:10.828, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:10.844340] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:10.835, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:10.855333] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:10.847, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:10.855352] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:10.844, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:10.855362] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:10.844, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:11.010774] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:11.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:11.010804] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:11.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:11.091322] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:11.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:11.212103] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:11.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:11.262441] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:11.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:11.262475] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:11.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:11.514221] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:11.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:11.514259] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:11.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:11.594777] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:11.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:11.629853] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ebbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:11.615, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:11.715601] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:11.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:11.727329] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:10.829741] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EED1-0-0] [lt=37714] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-08-14 09:03:11.732022] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25] get wrs ts(ls_id={id:1}, delta=441989443, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:11.732043] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:11.732068] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get_max_applied_scn(scn={val:1755161744727824001, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:49, palf_committed_end_lsn_:{lsn:7323686009}, palf_committed_end_scn_:{val:1755161747742105001, v:0}, last_check_scn_:{val:1755161749741193000, v:0}, max_applied_cb_scn_:{val:1755161744727824001, v:0}}) [2025-08-14 09:03:11.732083] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get_min_unreplayed_log_info(lsn={lsn:7323685240}, scn={val:1755100946796590002, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873485, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:8946764, ret_code_:0}}, next_to_submit_lsn_:{lsn:7323685240}, next_to_submit_scn_:{val:1755100946796590002, v:0}, base_lsn_:{lsn:7180210176}, base_scn_:{val:1753840803166611000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:7323685240}, end_lsn:{lsn:7323685240}, read_buf:{buf_len_:3747840, buf_:0x7f98f7a05000}, block_size:67104768, log_storage_:0x7f9901131bf0, log_storage_:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:28, read_io_size:98508800, read_disk_cost_ts:645339, start_lsn:{lsn:7180210176}}}}}}) [2025-08-14 09:03:11.732676] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=64] get wrs ts(ls_id={id:1001}, delta=447004357, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:11.732688] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:11.732713] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119812272, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-08-14 09:03:11.732743] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=441990988, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:11.732751] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:11.733331] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1001}, delta=447004357, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:11.733343] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:09.770952] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=32] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:10.787384] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=9] gc stale ls task succ [2025-08-14 09:03:10.955419] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=817432] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:09.776375] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:11.734078] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1954725][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:09.776|207=1.96s|210=0us|222=2us|total=1.96s) [2025-08-14 09:03:11.734096] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] timer task use too much time(end=1755162191734096, start=1755162189776373, delta=1957723) [2025-08-14 09:03:11.734106] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1941755, expected_run_ts_us_=1755162189792349, ObClockGenerator::getRealClock()=1755162191734105, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:09.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:11.734121] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:09.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:10.767625] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 33906624 limit= 214748360 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33906624 used= 30629648 count= 468 avg_used= 65447 mod=SUMMARY [2025-08-14 09:03:11.734198] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=965098] [MEMORY] tenant_id= 1001 ctx_id= TX_CALLBACK_CTX_ID hold= 3252224 used= 2342592 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 3 washed_blocks= 8 washed_size= 3039232 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2342592 used= 2269696 count= 286 avg_used= 7936 mod=SUMMARY [2025-08-14 09:03:11.734210] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:03:11.734220] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= RPC_CTX_ID hold= 14680064 used= 2457600 limit= 1073741800 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2457600 used= 1625600 count= 100 avg_used= 16256 mod=SUMMARY [2025-08-14 09:03:11.255256] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=6] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:11.735055] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=479071][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162191255251], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:11.735075] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162191255251], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:11.735089] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162191255251]}) [2025-08-14 09:03:11.735100] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:03:11.735117] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:03:11.735135] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=6] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:11.735142] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162191735131], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:11.735152] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162191735131]}) [2025-08-14 09:03:11.735160] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7] gts statistics(tenant_id=1001, gts_rpc_cnt=0, get_gts_cache_cnt=10, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:03:11.735614] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919abd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:11.721, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:11.735636] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919abb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:11.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:11.735645] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:11.726, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:11.735655] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fdc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:11.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:11.735664] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:11.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:10.780681] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=37] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:08.738629] EDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=488421][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2996181, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ed35c 0x97aead0 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:10.338487] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=496072][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1398839, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1398838, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:08.733106] EDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=984046][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3001825, , BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:08.781416] EDIAG [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [723][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2953866, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2949505, ALLOC_END=4360, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x973b640 0x973b132 0x973a32d 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:09.780005] EDIAG do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1955025, owner: ObLog, click_count: 5, time dist:[FORMAT_END=13, ObMalloc#start=1, ObMalloc#end=1955009, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x202c530b 0x202c6888 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:09.785426] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=389][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1960291, , BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:08.775519] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2968596, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:09.738716] EDIAG [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2029300, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2008548, ALLOC_END=20751, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x260b4cc1 0x260b9960 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [466]OB_MALLOC COST TOO MUCH TIME, cost_time=1922808, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1922782], seq:[0, 1, 2, 3, 4] [466]OB_MALLOC COST TOO MUCH TIME, cost_time=1922842, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1922836], seq:[0, 1] [2025-08-14 09:03:10.803401] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1922880, owner: ObLog, click_count: 5, time dist:[FORMAT_END=25, ObMalloc#start=2, ObMalloc#end=1922848, ALLOC_END=4, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:10.780802] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=161][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1946006, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1946005, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:10.780771] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=540][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1946097, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1946096, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:10.754440] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=23950][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1973674, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1973674, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:10.759331] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=990640][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1968842, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1968841, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 08:59:14.759149] EDIAG [SERVER] detect_master_rs_ls (ob_service.cpp:2601) [715][T1_L0_G9][T1][YB42AC4D0007-00063C495B49B11F-0-0] [lt=53][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1988173, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1988166, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x13b5e532 0x13b5cb6b 0x1254a2a5 0x984d055 0x9280839 0x926c154 0x12c5a0c9 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [857]OB_MALLOC COST TOO MUCH TIME, cost_time=899170, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=1890473, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1890432, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=1890527, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1890489, WASH_BLOCK_END=38], seq:[0, 1, 2] [2025-08-14 09:03:10.841255] EDIAG [RS] get_valid_sts_after (ob_tenant_info_loader.cpp:454) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=6074][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1890590, owner: ObLog, click_count: 5, time dist:[FORMAT_END=28, ObMalloc#start=1, ObMalloc#end=1890534, ALLOC_END=27, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1001d942 0x996d843 0x996d15a 0x996cfd8 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:10.759562] EDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=691792][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1972401, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=86, FORMAT_END=1972312, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ed35c 0x97aead0 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [472]OB_MALLOC COST TOO MUCH TIME, cost_time=1881938, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=1881898, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=1881968, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1881947, WASH_BLOCK_END=21], seq:[0, 1, 2] [2025-08-14 09:03:10.850255] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=346][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1882016, owner: ObLog, click_count: 5, time dist:[FORMAT_END=20, ObMalloc#start=2, ObMalloc#end=1881974, ALLOC_END=20, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:10.759334] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=988527][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1972854, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1972853, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [770]OB_MALLOC COST TOO MUCH TIME, cost_time=1777040, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1777012, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=1777061, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1777048, WASH_BLOCK_END=13], seq:[0, 1, 2] [2025-08-14 09:03:10.955536] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=2895][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1777110, owner: ObLog, click_count: 5, time dist:[FORMAT_END=20, ObMalloc#start=1, ObMalloc#end=1777067, ALLOC_END=22, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [400]OB_MALLOC COST TOO MUCH TIME, cost_time=1389039, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=2, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=2, ALLOC_CHUNK_END=1389009, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [400]OB_MALLOC COST TOO MUCH TIME, cost_time=1389057, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1389048, WASH_BLOCK_END=9], seq:[0, 1, 2] [2025-08-14 09:03:11.344245] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=493127][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1389098, owner: ObLog, click_count: 5, time dist:[FORMAT_END=16, ObMalloc#start=1, ObMalloc#end=1389062, ALLOC_END=18, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [768]OB_MALLOC COST TOO MUCH TIME, cost_time=1131003, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=18, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=1130957, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [768]OB_MALLOC COST TOO MUCH TIME, cost_time=1131019, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1131011, WASH_BLOCK_END=8], seq:[0, 1, 2] [2025-08-14 09:03:11.602732] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=280][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1131073, owner: ObLog, click_count: 5, time dist:[FORMAT_END=29, ObMalloc#start=1, ObMalloc#end=1131024, ALLOC_END=18, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [401]OB_MALLOC COST TOO MUCH TIME, cost_time=1014767, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=17, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1014726, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=1014784, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1014776, WASH_BLOCK_END=8], seq:[0, 1, 2] [2025-08-14 09:03:11.719253] EDIAG [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=165][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1014823, owner: ObLog, click_count: 5, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=1014789, ALLOC_END=20, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a1f867 0x27a135c3 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [393]OB_MALLOC COST TOO MUCH TIME, cost_time=1006865, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=1006817, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=1006881, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1006873, WASH_BLOCK_END=8], seq:[0, 1, 2] [2025-08-14 09:03:11.727262] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6058][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1006918, owner: ObLog, click_count: 5, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=1006886, ALLOC_END=19, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [394]OB_MALLOC COST TOO MUCH TIME, cost_time=1006762, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1006728, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=1006778, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1006770, WASH_BLOCK_END=8], seq:[0, 1, 2] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=1005575, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1005550, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=1005593, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1005584, WASH_BLOCK_END=8], seq:[0, 1, 2] [2025-08-14 09:03:09.753350] EDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14740][errcode=-4016] LOGGER COST TOO MUCH TIME, cost: 2978019, owner: ObLog, click_count: 9, time dist:[ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=83, ALLOC_BLOCK#start=11018, ALLOC_BLOCK#end=3, FORMAT_END=1961284, ObMalloc#start=0, ObMalloc#end=1005598, ALLOC_END=29, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x13b06ffb 0x958a39e 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=1005702, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1005670, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=1005720, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1005710, WASH_BLOCK_END=10], seq:[0, 1, 2] [2025-08-14 09:03:10.761970] EDIAG [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1971063, owner: ObLog, click_count: 5, time dist:[FORMAT_END=965319, ObMalloc#start=1, ObMalloc#end=1005726, ALLOC_END=17, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1f832f91 0x97c36fe 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [567]OB_MALLOC COST TOO MUCH TIME, cost_time=1005761, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=1005737, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=1005780, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1005769, WASH_BLOCK_END=11], seq:[0, 1, 2] [2025-08-14 09:03:11.728825] EDIAG [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1005834, owner: ObLog, click_count: 5, time dist:[FORMAT_END=27, ObMalloc#start=1, ObMalloc#end=1005786, ALLOC_END=20, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0xfebfd41 0x94edd90 0x94edb97 0x97a8fbd 0x97a8cc7 0x97a88d2 0x97a7481 0x97a6a43 0x97a42c5 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [729]OB_MALLOC COST TOO MUCH TIME, cost_time=1005734, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1005708, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [729]OB_MALLOC COST TOO MUCH TIME, cost_time=1005766, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1005753, WASH_BLOCK_END=13], seq:[0, 1, 2] [2025-08-14 09:03:11.729442] EDIAG [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1005814, owner: ObLog, click_count: 5, time dist:[FORMAT_END=22, ObMalloc#start=0, ObMalloc#end=1005772, ALLOC_END=19, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1c03a517 0x97a8d73 0x97a88d2 0x97a7481 0x97a6a43 0x97a42c5 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:11.737854] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=955725][errcode=0] epoll handler too much time :957182 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:11.737908] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22693] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:11.737937] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22720] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:11.737959] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22740] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:09.780005] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=10] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:10.782298] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=23] PNIO [ratelimit] time: 1755162190782295, bytes: 0, bw: 0.000000 MB/s, add_ts: 1021050, add_bytes: 0 [2025-08-14 09:03:10.767649] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:1131) [812][T1002_TransTime][T1002][Y0-0000000000000000-0-0] [lt=4960][errcode=0] [CACHE-SYNC-WASH] len: 3436 tenant sync wash failed, cache memblock info: [CACHE-SYNC-WASH] tenant_id= 1002 | cache_id= 3 | status= 1 | policy= 1 | kv_cnt= 218 | get_cnt= 435757 | score=86.549926 | [CACHE-SYNC-WASH] tenant_id= 1002 | cache_id= 21 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE-SYNC-WASH] tenant_id= 1002 | cache_id= 21 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1976 | score=1.800000 | [CACHE-SYNC-WASH] tenant_id= 1002 | cache_id= 18 | status= 1 | policy= 0 | kv_cnt= 59 | get_cnt= 183 | score=0.000000 | [CACHE-SYNC-WASH] tenant_id= 1002 | cache_id= 18 | status= 1 | policy= 1 | kv_cnt= 139 | get_cnt= 17504 | score=26.100000 | [CACHE-SYNC-WASH] tenant_id= 1002 | cache_id= 7 | status= 1 | policy= 0 | kv_cnt= 174 | get_cnt= 187 | score=0.000000 | [CACHE-SYNC-WASH] tenant_id= 1002 | cache_id= 5 | status= 1 | policy= 1 | kv_cnt= 236 | get_cnt= 12308 | score=0.000000 | [CACHE-SYNC-WASH] tenant_id= 1002 | cache_id= 20 | status= 1 | policy= 0 | kv_cnt= 33 | get_cnt= 78 | score=0.000000 | [CACHE-SYNC-WASH] tenant_id= 1002 | cache_id= 7 | status= 1 | policy= 1 | kv_cnt= 80 | get_cnt= 30967 | score=2.528982 | [CACHE-SYNC-WASH] tenant_id= 1002 | cache_id= 6 | status= 1 | policy= 0 | kv_cnt= 3 | get_cnt= 3 | score=0.000000 | [CACHE-SYNC-WASH] tenant_id= 1002 | cache_id= 12 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE-SYNC-WASH] tenant_id= 1002 | cache_id= 12 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 690 | score=0.000000 | [CACHE-SYNC-WASH] tenant_id= 1002 | cache_id= 6 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 494 | score=0.000000 | [CACHE-SYNC-WASH] tenant_id= 1002 | cache_id= 20 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 173 | score=0.000000 | [CACHE-SYNC-WASH] tenant_id= 1002 | cache_id= 3 | status= 1 | policy= 0 | kv_cnt= 711 | get_cnt= 1686 | score=0.000000 | [CACHE-SYNC-WASH] tenant_id= 1002 | cache_id= 19 | status= 1 | policy= 1 | kv_cnt= 189 | get_cnt= 3438 | score=34.200000 | [CACHE-SYNC-WASH] tenant_id= 1002 | cache_id= 8 | status= 1 | policy= 1 | kv_cnt= 396 | get_cnt= 25624 | score=1137.133397 | [CACHE-SYNC-WASH] tenant_id= 1002 | cache_id= 4 | status= 1 | policy= 1 | kv_cnt= 105 | get_cnt= 26345 | score=45.058803 | [CACHE-SYNC-WASH] tenant_id= 1002 | cache_id= 19 | status= 1 | policy= 0 | kv_cnt= 89 | get_cnt= 141 | score=0.000000 | [CACHE-SYNC-WASH] tenant_id= 1002 | cache_id= 15 | status= 1 | policy= 0 | kv_cnt= 13 | get_cnt= 13 | score=0.000000 | [CACHE-SYNC-WASH] tenant_id= 1002 | cache_id= 4 | status= 1 | policy= 0 | kv_cnt= 39 | get_cnt= 63 | score=1.620000 | [CACHE-SYNC-WASH] tenant_id= 1002 | cache_id= 5 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 43 | score=0.000000 | [CACHE-SYNC-WASH] tenant_id= 1002 | cache_id= 8 | status= 1 | policy= 0 | kv_cnt= 97 | get_cnt= 103 | score=527.064548 | [2025-08-14 09:03:08.738629] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=488421][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:08.249|207=489.16ms|210=0us|222=2us|total=3.49s) [2025-08-14 09:03:11.739369] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=2996181][errcode=-4389] timer task use too much time(end=1755162191739368, start=1755162188249462, delta=3489906) [2025-08-14 09:03:11.739380] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=3494746, expected_run_ts_us_=1755162188244633, ObClockGenerator::getRealClock()=1755162191739379, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:08.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:11.739396] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:08.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:10.338487] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=496072][errcode=0] EASY SLOW: thread: 7f995447d300, lastrun: 1755162189.770943 cost: 0.567542 loop:26564, slowcnt: 5 [2025-08-14 09:03:08.733106] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=984046][errcode=0] epoll handler too much time :985542 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:11.739498] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3001825] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:11.739510] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:11.739631] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [316][pnio1][T1][YB42AC4D0007-00063C49607B92BE-0-0] [lt=13][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1, delta=14594870, last_succ_cluster_heartbeat_tstamp_=1755162177144759) [2025-08-14 09:03:08.781416] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [723][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=24] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=16, total_destructed_detector=16, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f99333562b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-08-14 09:03:11.739772] WDIAG [DETECT] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [723][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=2953866][errcode=0] cost too much time:ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t), (this=|threshold=100.00ms|start at 09:03:11.739|total=2.96s) [2025-08-14 09:03:11.734231] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= TX_DATA_TABLE hold= 5570560 used= 2342784 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 8 washed_size= 2818048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2342784 used= 2319424 count= 30 avg_used= 77314 mod=SUMMARY [2025-08-14 09:03:11.739826] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5573] [MEMORY] tenant_id= 1001 ctx_id= MDS_DATA_ID hold= 3866624 used= 2785280 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 5 washed_size= 4521984 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2785280 used= 2698240 count= 340 avg_used= 7936 mod=SUMMARY [2025-08-14 09:03:11.739843] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= MERGE_NORMAL_CTX_ID hold= 1179648 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 8 washed_blocks= 24 washed_size= 15597568 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:03:11.735354] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:03:11.739927] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=4564][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:03:11.739940] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162191735350], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:11.739957] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162191735350]}) [2025-08-14 09:03:11.746293] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:11.747653] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3827] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:11.748220] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728be10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:11.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:08.775519] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:09.785426] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=389] gc stale ls task succ [2025-08-14 09:03:11.748727] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=1960291] gc stale ls task succ [2025-08-14 09:03:10.780727] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:11.748749] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=966559][errcode=0] epoll handler too much time :968028 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:10.761920] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEB1-0-0] [lt=19][errcode=-4016] task executed too delay(delay_time=90487, *p_task_={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:0, time_interval:3.00s, expected_run_time:09:03:13.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:11.739861] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant: 1002, limit: 8589934592 hold: 1943941120 cache_hold: 48234496 cache_used: 48234496 cache_item_count: 23 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 1005416448 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 415236096 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 1114112 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 950272 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 211812352 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 1441792 limit= 429496725 [MEMORY] ctx_id= GLIBC hold_bytes= 12582912 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 149913600 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 37748736 limit= 1717986900 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 7766016 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 8077312 limit= 8589934500 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 5718016 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3047424 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 1032192 limit= 9223372036854775807 [2025-08-14 09:03:11.750044] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10134] [MEMORY] tenant_id= 1002 ctx_id= DEFAULT_CTX_ID hold= 1005416448 used= 915956976 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 15 [MEMORY] hold= 185860096 used= 185534336 count= 114 avg_used= 1627494 block_cnt= 114 chunk_cnt= 104 mod=MysqlRequesReco [MEMORY] hold= 167792640 used= 167772192 count= 1 avg_used= 167772192 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 83927040 used= 83886080 count= 2 avg_used= 41943040 block_cnt= 2 chunk_cnt= 2 mod=LogGroupBuffer [MEMORY] hold= 67190784 used= 67109344 count= 4 avg_used= 16777336 block_cnt= 4 chunk_cnt= 4 mod=MrgMemPoolChk [MEMORY] hold= 915956976 used= 898770921 count= 9455 avg_used= 95057 mod=SUMMARY [2025-08-14 09:03:11.750060] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1002 ctx_id= MEMSTORE_CTX_ID hold= 415236096 used= 411992064 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 411992064 used= 411803568 count= 198 avg_used= 2079816 block_cnt= 198 chunk_cnt= 198 mod=Memstore [MEMORY] hold= 411992064 used= 411803568 count= 198 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:03:11.750068] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= EXECUTE_CTX_ID hold= 1114112 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 7 washed_size= 7274496 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:03:11.739414] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=3009270, expected_run_ts_us_=1755162188730144, ObClockGenerator::getRealClock()=1755162191739414, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8030, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8310, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:10.00s, expected_run_time:09:03:08.730, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:11.750089] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=10656][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8030, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8310, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:10.00s, expected_run_time:09:03:08.730, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:11.750114] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] task scheduled too delay(scheduled_delay=2766583, expected_run_ts_us_=1755162188983531, ObClockGenerator::getRealClock()=1755162191750114, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:08.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:11.750128] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:08.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:11.750670] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:11.750668] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2968596][errcode=0] epoll handler too much time :2975159 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:11.750787] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=109] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:11.750796] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:11.758244] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72847b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:11.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:08.921610] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [882][T1002_ArcSender][T1002][YB42AC4D0007-00063C495EC99C16-0-0] [lt=10] ObArchiveSender is running(thread_index=6) [2025-08-14 09:03:11.763652] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:11.763726] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEB3-0-0] [lt=985402] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:03:11.750077] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1002 ctx_id= TRANS_CTX_MGR_ID hold= 950272 used= 933888 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1146880 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 933888 used= 921984 count= 2 avg_used= 460992 mod=SUMMARY [2025-08-14 09:03:11.765578] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=15463] [MEMORY] tenant_id= 1002 ctx_id= PLAN_CACHE_CTX_ID hold= 211812352 used= 201995008 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 176930880 used= 160228962 count= 13450 avg_used= 11912 block_cnt= 13450 chunk_cnt= 101 mod=SqlPhyPlan [MEMORY] hold= 201995008 used= 183048143 count= 17906 avg_used= 10222 mod=SUMMARY [2025-08-14 09:03:11.765591] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1002 ctx_id= WORK_AREA hold= 1441792 used= 0 limit= 429496725 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 14 washed_blocks= 54 washed_size= 27918336 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:03:11.765602] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= GLIBC hold= 12582912 used= 11425280 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 11425280 used= 6322556 count= 19216 avg_used= 329 mod=SUMMARY [2025-08-14 09:03:11.765612] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1002 ctx_id= CO_STACK hold= 149913600 used= 147087360 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 68 washed_blocks= 70 washed_size= 3178496 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 147087360 used= 146813760 count= 285 avg_used= 515136 block_cnt= 285 chunk_cnt= 73 mod=CoStack [MEMORY] hold= 147087360 used= 146813760 count= 285 avg_used= 515136 mod=SUMMARY [2025-08-14 09:03:11.767413] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fb3b0, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fb690, time_wheel:0x7f99589fd990, thread_pool:0x7f9959fff170, is_running:1, time_interval:2.00s, expected_run_time:09:03:11.698, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:11.750178] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:11.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:11.770642] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:11.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:11.770671] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:11.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:11.770939] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:11.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:11.770958] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891806f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:11.751, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:11.770968] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:11.750, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:11.770979] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73539b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7396b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:11.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:09.738716] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=16] log compressor cycles once. (ret=0, cost_time=1038405, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25898504192, fast_delete_log_mode=false) [2025-08-14 09:03:11.778465] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:11.778616] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:11.784673] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=954400] PNIO [ratelimit] time: 1755162191784671, bytes: 0, bw: 0.000000 MB/s, add_ts: 1002376, add_bytes: 0 [2025-08-14 09:03:11.801107] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:11.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:11.811167] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d22b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eaf90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:11.808, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:11.821239] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:11.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:11.821264] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:11.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:11.841315] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:11.831, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:11.841339] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:11.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:11.851351] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ac10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:11.849, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:11.861410] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:11.852, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:11.871472] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:11.861, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:11.901636] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919859b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:11.896, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:12.015509] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:12.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:12.015539] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:12.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:12.096051] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:12.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:12.216718] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:12.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:12.266988] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=57][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:12.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:12.267027] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:12.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:12.518383] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:12.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:12.518420] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:12.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:12.598865] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:12.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:11.763745] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEB3-0-0] [lt=14][errcode=-4389] run this task use too much time(*p_task_={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:0, time_interval:3.00s, expected_run_time:09:03:13.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:12.717172] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEB3-0-0] [lt=951980][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:03:12.717|total=1.96s) [2025-08-14 09:03:11.765625] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1002 ctx_id= KVSTORE_CACHE_ID hold= 48234496 used= 48234496 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 48234496 used= 0 count= 23 avg_used= 0 mod=SUMMARY [2025-08-14 09:03:12.717278] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=950197] [MEMORY] tenant_id= 1002 ctx_id= META_OBJ_CTX_ID hold= 37748736 used= 35741632 limit= 1717986900 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 35741632 used= 32457488 count= 496 avg_used= 65438 mod=SUMMARY [2025-08-14 09:03:12.717290] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1002 ctx_id= TX_CALLBACK_CTX_ID hold= 7766016 used= 6934400 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 17 washed_size= 6914048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 6934400 used= 6729728 count= 848 avg_used= 7936 mod=SUMMARY [2025-08-14 09:03:12.717300] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= LOB_CTX_ID hold= 147456 used= 16384 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 16384 used= 15872 count= 2 avg_used= 7936 mod=SUMMARY [2025-08-14 09:03:12.717310] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= RPC_CTX_ID hold= 8077312 used= 3981312 limit= 8589934500 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 6 washed_blocks= 6 washed_size= 10797056 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3981312 used= 2620416 count= 168 avg_used= 15597 mod=SUMMARY [2025-08-14 09:03:12.717321] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= TX_DATA_TABLE hold= 5718016 used= 4521856 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 9 washed_blocks= 16 washed_size= 13156352 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4521856 used= 4480128 count= 40 avg_used= 112003 mod=SUMMARY [2025-08-14 09:03:11.749270] WDIAG [RPC] handle_resp (ob_poc_rpc_proxy.cpp:192) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] async_cb handler cost too much time(total_time=1959281, decode_time=1959257, process_time=21, session_destroy_time=3, ret=0, pcode=1815) [2025-08-14 09:03:12.397474] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [402][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=688886][errcode=0] EASY SLOW: thread: 7f9953cfb300, lastrun: 1755162191.719247 cost: 0.678224 loop:26516, slowcnt: 6 [2025-08-14 09:03:11.774000] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:12.717758] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=942330][errcode=0] epoll handler too much time :943767 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:11.787954] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=945226] gc stale ls task succ [2025-08-14 09:03:12.240176] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:12.717797] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=476895][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162192240170], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:12.717817] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162192240170], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:12.717832] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162192240170]}) [2025-08-14 09:03:12.717842] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162192240170]}) [2025-08-14 09:03:11.999856] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=17722][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:11.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:12.717959] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=717014][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:11.999|207=718.10ms|210=0us|222=2us|total=718.13ms) [2025-08-14 09:03:12.717985] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4389] timer task use too much time(end=1755162192717984, start=1755162191999853, delta=718131) [2025-08-14 09:03:12.717997] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=473363, expected_run_ts_us_=1755162192244633, ObClockGenerator::getRealClock()=1755162192717996, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:12.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:12.718011] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:12.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:11.979048] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=21][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:03:12.727375] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:10.803401] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c0b90, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571c0e70, time_wheel:0x7f9959fef990, thread_pool:0x7f9959fef170, is_running:1, time_interval:3.00s, expected_run_time:09:03:10.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:12.729213] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=1922880][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:10.803|207=1.93s|210=0us|222=2us|total=1.93s) [2025-08-14 09:03:12.729242] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4389] timer task use too much time(end=1755162192729242, start=1755162190803398, delta=1925844) [2025-08-14 09:03:10.780802] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=161] gc stale ls task succ [2025-08-14 09:03:12.729785] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=1946006] gc stale ls task succ [2025-08-14 09:03:10.780771] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=540] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:12.729833] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1946097][errcode=0] epoll handler too much time :1949068 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:12.729850] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:11.734011] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1960070][errcode=0] epoll handler too much time :1963066 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:12.730663] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=995134] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:12.730675] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:10.754440] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=23950] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:10.759331] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=990640] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:12.733353] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=442987065, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:12.733371] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:12.733872] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1001}, delta=448000434, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:12.733880] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:12.733901] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=434512100, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=251748, succ_cluster_heartbeat_count=1502, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161006924036004, v:0}, local_cluster_delta=1185809862, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:03:12.733938] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26] get wrs ts(ls_id={id:1}, delta=442987065, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:12.733945] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:12.717392] WDIAG stat_cleanup (debug.c:35) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=966657][errcode=0] PNIO [delay_warn] cost too much time: 2942020us, procedure: pn_pktc_resp_cb [2025-08-14 09:03:12.734733] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1890590][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:12.734742] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:12.734802] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:12.724, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:12.034296] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=777457] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:10.759562] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=691792][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:10.667|207=692.84ms|210=0us|222=1us|total=2.67s) [2025-08-14 09:03:12.734963] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1972401][errcode=-4389] timer task use too much time(end=1755162192734963, start=1755162190066718, delta=2668245) [2025-08-14 09:03:12.734972] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=0] task scheduled too delay(scheduled_delay=1941037, expected_run_ts_us_=1755162190793934, ObClockGenerator::getRealClock()=1755162192734972, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:10.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:12.734988] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:10.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:10.850255] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=346][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:10.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:12.735133] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1882016][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:10.850|207=1.88s|210=1us|222=1us|total=1.88s) [2025-08-14 09:03:12.735142] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] timer task use too much time(end=1755162192735141, start=1755162190850253, delta=1884888) [2025-08-14 09:03:10.759334] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=988527] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:03:10.955536] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=2895][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:03:10.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:12.735361] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1777110][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:10.955|207=1.78s|210=0us|222=2us|total=1.78s) [2025-08-14 09:03:12.735370] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] timer task use too much time(end=1755162192735369, start=1755162190955534, delta=1779835) [2025-08-14 09:03:11.344245] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=493127][errcode=0] EASY SLOW: thread: 7f9953e7d300, lastrun: 1755162187.747563 cost: 3.596679 loop:26556, slowcnt: 30 [2025-08-14 09:03:11.602732] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=280][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163c60, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99319fc090, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:1.00s, expected_run_time:09:03:11.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:12.735529] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=1131073][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:11.602|207=1.13s|210=0us|222=1us|total=1.13s) [2025-08-14 09:03:12.735537] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] timer task use too much time(end=1755162192735537, start=1755162191602729, delta=1132808) [2025-08-14 09:03:11.719253] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=165] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:12.735626] WDIAG ev_invoke_pending (ev.c:2136) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=1014823][errcode=0] epoll handler too much time :1016377 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:12.735638] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:11.727262] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6058] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:09.753350] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14740][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:03:10.761970] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=23] current monitor number(seq_=-1) [2025-08-14 09:03:12.736090] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:12.736110] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:11.728825] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] get_max_applied_scn(scn={val:1755161547738516001, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:50, palf_committed_end_lsn_:{lsn:5282737655}, palf_committed_end_scn_:{val:1755161547738516001, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161547738516001, v:0}}) [2025-08-14 09:03:12.736664] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1005834] get wrs ts(ls_id={id:1}, delta=644989742, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:12.736676] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:11.729442] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=471952309, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:12.736811] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [416]OB_MALLOC COST TOO MUCH TIME, cost_time=986960, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=986925], seq:[0, 1, 2, 3, 4, 5, 6] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=986971, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=986970], seq:[0, 1] [2025-08-14 09:03:09.946493] EDIAG [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2786399, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1f9d7fcf 0x99b8be2 0x1f981833 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [833]OB_MALLOC COST TOO MUCH TIME, cost_time=987052, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=987019], seq:[0, 1, 2, 3, 4, 5, 6] [833]OB_MALLOC COST TOO MUCH TIME, cost_time=987062, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=987061], seq:[0, 1] [2025-08-14 09:03:08.838109] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=917038][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3893322, , BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:09.946493] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:03:12.737180] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2786399] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:03:12.737227] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:03:12.737276] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:03:08.838109] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=917038] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:03:12.717864] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=8] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:12.737447] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=19550][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162192717860], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:12.737465] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162192717860]}) [2025-08-14 09:03:12.737642] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:03:12.717332] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= MDS_DATA_ID hold= 3047424 used= 1998848 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 4 washed_size= 1146880 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1998848 used= 1936384 count= 244 avg_used= 7936 mod=SUMMARY [2025-08-14 09:03:12.737688] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=20305] [MEMORY] tenant_id= 1002 ctx_id= MDS_CTX_ID hold= 147456 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:03:12.737704] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1002 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:03:12.737715] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= MERGE_NORMAL_CTX_ID hold= 1032192 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 12 washed_size= 13647872 [MEMORY] request_cached_chunk_cnt= 0 [777]OB_MALLOC COST TOO MUCH TIME, cost_time=989790, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=989756], seq:[0, 1, 2, 3, 4, 5, 6] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=989812, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=989810], seq:[0, 1] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=984979, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=984935], seq:[0, 1, 2, 3, 4, 5, 6] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=984992, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=984992], seq:[0, 1] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=969676, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=969640], seq:[0, 1, 2, 3, 4, 5, 6] [2025-08-14 09:03:11.755255] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2658] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:12.741758] WDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=985012][errcode=0] epoll handler too much time :986513 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [660]OB_MALLOC COST TOO MUCH TIME, cost_time=969688, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=969686], seq:[0, 1] [2025-08-14 09:03:11.770618] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:11.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:12.741811] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=969721][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:03:11.770|207=971.19ms|210=0us|222=2us|total=971.21ms) [2025-08-14 09:03:12.741836] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=-4389] timer task use too much time(end=1755162192741836, start=1755162191770616, delta=971220) [2025-08-14 09:03:12.741849] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=949498, expected_run_ts_us_=1755162191792349, ObClockGenerator::getRealClock()=1755162192741848, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:11.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:12.741867] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:11.792, task_priority:1, with_handle_protected:1}) [571]OB_MALLOC COST TOO MUCH TIME, cost_time=910740, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=910705], seq:[0, 1, 2, 3, 4, 5, 6] [571]OB_MALLOC COST TOO MUCH TIME, cost_time=910752, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=910751], seq:[0, 1] [2025-08-14 09:03:11.829780] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [571][T1_ArcSender][T1][YB42AC4D0007-00063C495BA99C16-0-0] [lt=862633][errcode=-4018] try free send task failed(ret=-4018) [733]OB_MALLOC COST TOO MUCH TIME, cost_time=910719, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=910687], seq:[0, 1, 2, 3, 4, 5, 6] [733]OB_MALLOC COST TOO MUCH TIME, cost_time=910730, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=910729], seq:[0, 1] [2025-08-14 09:03:11.829834] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=862107][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:03:12.743164] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8] get wrs ts(ls_id={id:1001}, delta=448000434, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:12.743191] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:12.747190] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=157] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:12.748823] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=5] gc stale ls task succ [2025-08-14 09:03:10.759270] EDIAG [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=1978196][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1988930, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=6, ALLOC_BLOCK#end=97, ALLOC_BLOCK#start=11037, ALLOC_BLOCK#end=3, FORMAT_END=1977783, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1d17bb52 0x1d17ada0 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:10.777525] EDIAG [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [726][T1001_ObXAHbWor][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1971489, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=7, ALLOC_BLOCK#end=1, FORMAT_END=1971477, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b64d8ba 0x1b64d154 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:12.735718] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1006918][errcode=0] epoll handler too much time :1008469 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:12.749459] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13706] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:12.717226] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:826) [812][T1002_TransTime][T1002][Y0-0000000000000000-0-0] [lt=970162] can not find enough memory block to wash(ret=-4273, size_washed=0, size_to_wash=6291456) [2025-08-14 09:03:12.750876] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:845) [812][T1002_TransTime][T1002][Y0-0000000000000000-0-0] [lt=33594] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_to_wash=6291456) [2025-08-14 09:03:11.769384] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=13861] [COMPACTION DAG STATUS] (start_time=1755162181733778, end_time=1755162191760221, dag_status={}) [2025-08-14 09:03:10.759270] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=1978196] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623081220537, flush_io_finished_ret_:0, flush_io_finished_round_:26313, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:03:11.727454] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=248] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:12.748456] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:12.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:12.735545] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=4][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571634a0, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957163780, time_wheel:0x7f9922d27990, thread_pool:0x7f9922d27170, is_running:1, time_interval:5.00s, expected_run_time:09:03:12.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:12.751667] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:12.736792] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1005814] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:12.735898] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=2978019][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:03:12.751707] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15777][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:03:12.751734] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:03:12.734272] WDIAG eloop_delay_warn (debug.h:93) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=16842][errcode=0] PNIO [delay_warn] eloop handle events delay high: 2959020, malloc=0/0 write=0/0 read=0/2 server_process=0/0 client_cb=2942020/1 [2025-08-14 09:03:10.740641] EDIAG [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=2004320][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2011625, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2011624, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1d17bb52 0x1d17ada0 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:12.737654] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=12] get number(ret=-4038, service_type_=0, range=1, base_id=1755162192728258000, start_id=0, end_id=140295968341544) [2025-08-14 09:03:12.751805] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=14127][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:03:12.751822] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162192737636], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:12.751839] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162192737636]}) [2025-08-14 09:03:12.751851] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162192737636]}) [2025-08-14 09:03:12.751863] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1002}) [2025-08-14 09:03:12.751874] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1002}) [2025-08-14 09:03:10.777525] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [726][T1001_ObXAHbWor][T1001][Y0-0000000000000000-0-0] [lt=14] XA scheduler heartbeat task statistics(avg_time=1) [2025-08-14 09:03:11.728322] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [280][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=29913][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:195, tid:768}]) [2025-08-14 09:03:12.741627] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=1955025] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:12.741687] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:12.754650] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12941][errcode=0] epoll handler too much time :12970 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:12.747422] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:12.754697] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:12.754801] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:12.754937] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:12.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:12.754976] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:12.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:11.769398] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] [COMPACTION DAG STATUS] (start_time=1755162181733855, end_time=1755162191760242, dag_status={}) [2025-08-14 09:03:12.717340] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=17980] [COMPACTION DAG STATUS] (start_time=1755162181732719, end_time=1755162191766259, dag_status={}) [2025-08-14 09:03:10.740641] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=2004320] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623084214679, flush_io_finished_ret_:0, flush_io_finished_round_:26346, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:03:12.740279] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=644989742, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:12.755443] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15138] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:12.758521] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=3170][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:12.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:12.764096] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:12.766620] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:12.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:12.773410] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:11.727454] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=248][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1022614, owner: ObLog, click_count: 5, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=1006783, ALLOC_END=15815, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:08.775168] EDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3994080, owner: ObLog, click_count: 5, time dist:[FORMAT_END=3994071, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0xfc0e73d 0x993f560 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:08.776289] EDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=18692][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3994682, , BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0xfc0e73d 0x993f560 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:12.751748] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:03:12.774305] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22518][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:03:12.774331] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1022614][errcode=0] epoll handler too much time :1046888 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:03:12.774360] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:12.774492] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:03:12.774510] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:03:12.719508] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:03:12.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:12.774659] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=55063][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:12.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:12.774674] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:12.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:12.774776] WDIAG pktc_handle_req_queue (pktc_post.h:138) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=17475][errcode=0] PNIO [delay_warn] delay high: 1023007 [2025-08-14 09:03:12.774813] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:438) [316][pnio1][T1][YB42AC4D0007-00063C49607B92C0-0-0] [lt=12][errcode=-4016] tenant weak read service cluster heartbeat cost too much time(tenant_id_=1, delta=1035183, last_succ_cluster_heartbeat_tstamp_=1755162191739629) [2025-08-14 09:03:08.775168] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1001, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:8571658, max_lsn:7323687369, max_scn:{val:1755161749741193002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:7278610047}, reuse_lsn:{lsn:7323686131}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:7281744329}}, last_submit_lsn:{lsn:7323686131}, last_submit_end_lsn:{lsn:7323687369}, last_submit_log_id:8571658, last_submit_log_pid:49, max_flushed_lsn:{lsn:7323686009}, max_flushed_end_lsn:{lsn:7323686131}, max_flushed_log_pid:49, committed_end_lsn:{lsn:7323686009}, last_slide_log_id:8571656, last_slide_scn:{val:1755161747742105001, v:0}, last_slide_lsn:{lsn:7323685887}, last_slide_end_lsn:{lsn:7323686009}, last_slide_log_pid:49, last_slide_log_accum_checksum:2505458373, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f990112a530}, [StateMgr]={this:0x7f990112e9b0, self:"172.77.0.7:2882", palf_id:1001, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"PENDING", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:49}, leader:"0.0.0.0:0", leader_epoch:-1, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:7323687369}, scan_disk_log_finished:true, last_check_start_id:8571657, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161767744317, palf_role_change_cb:0x7f9901135470, allow_vote:true}, [ConfigMgr]={palf_id:1001, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:49, prev:{version:1, config:{config_version:{proposal_id:48, config_seq:47}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:49, config_seq:48}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_mode_pid:49}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_end_lsn:{lsn:7323685240}, prev_mode_pid:49}, reconfig_barrier:{prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_end_lsn:{lsn:7323685240}, prev_mode_pid:49}, persistent_config_version:{proposal_id:49, config_seq:48}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755161739755312, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755161739755312, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f990112bc70}, [ModeMgr]={palf_id:1001, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, accepted_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1001, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1753069671378640001, v:0}, base_lsn_for_block_gc:{lsn:7247314944}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:49}, log_config_meta:{version:3, proposal_id:49, prev:{version:1, config:{config_version:{proposal_id:48, config_seq:47}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:49, config_seq:48}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_mode_pid:49}, log_snapshot_meta:{version:2, base_lsn:{lsn:7247314944}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1753069671378640000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}}, log_meta_storage:{log_tail:{lsn:765952}, readable_log_tail:{lsn:765952}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7381524480}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/meta", dir_fd:232, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66338816, block_header_serialize_buf_:0x7f9901130bb8, flashback_version:0}, log_storage:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, palf_epoch:2, last_purge_throttling_ts:1755161769777679, this:0x7f990112f0f0}, [Reconfirm]={palf_id:1001, self:"172.77.0.7:2882", state:"WAITING_LOG_FLUSHED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:1755161769777679, this:0x7f990112eb30}) [2025-08-14 09:03:12.775338] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=3994080] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [2025-08-14 09:03:12.775354] INFO [COMMON] print_io_status (ob_io_manager.cpp:2639) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:2729, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:499338, total:499343) [2025-08-14 09:03:12.775361] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=19] current log_task status(palf_id=1001, self="172.77.0.7:2882", log_id=8571657, log_task={header:{begin_lsn:{lsn:7323686009}, end_lsn:{lsn:7323686131}, log_id:8571657, min_scn:{val:1755161749741193000, v:0}, max_scn:{val:1755161749741193000, v:0}, data_len:66, proposal_id:49, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:7323686009}, data_checksum:2222604892, accum_checksum:1531236791, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1755161749745909, freeze_ts:1755161749745914, submit_ts:1755161749745940, flushed_ts:1755161785758706, gen_to_freeze cost time:5, gen_to_submit cost time:31, submit_to_flush cost time:36012766}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [2025-08-14 09:03:12.776260] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14867] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119797572, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755162138745740) [2025-08-14 09:03:12.776289] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] get_max_applied_scn(scn={val:1755161719775179002, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:45, palf_committed_end_lsn_:{lsn:8461098908}, palf_committed_end_scn_:{val:1755161719775179002, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161719775179002, v:0}}) [2025-08-14 09:03:12.776311] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] get_min_unreplayed_log_info(lsn={lsn:8461098908}, scn={val:1755100947600130001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161920778323, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:23084275, ret_code_:0}}, next_to_submit_lsn_:{lsn:8461098908}, next_to_submit_scn_:{val:1755100947600130001, v:0}, base_lsn_:{lsn:7515734016}, base_scn_:{val:1753840867171901045, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6e05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:8461098908}, end_lsn:{lsn:8461098908}, read_buf:{buf_len_:3747840, buf_:0x7f98f6e05000}, block_size:67104768, log_storage_:0x7f99297a3bf0, log_storage_:{log_tail:{lsn:8461098908}, readable_log_tail:{lsn:8461098908}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8455200768}, min_scn:{val:1755160384851871002, v:0}, curr_block_id:126, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:215, min_block_id:93, max_block_id:127, curr_writable_block_id:126}, logical_block_size_:67104768, curr_block_writable_size_:61206628, block_header_serialize_buf_:0x7f99297a48b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:198, read_io_size:699822080, read_disk_cost_ts:4178035, start_lsn:{lsn:7515734016}}}}}}) [2025-08-14 09:03:12.776946] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=72] get wrs ts(ls_id={id:1}, delta=473000521, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:12.776964] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:08.776289] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=18692] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:12632581, max_lsn:8461098908, max_scn:{val:1755161719775179002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:8214679957}, reuse_lsn:{lsn:8461098908}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:8419155868}}, last_submit_lsn:{lsn:8461098786}, last_submit_end_lsn:{lsn:8461098908}, last_submit_log_id:12632581, last_submit_log_pid:45, max_flushed_lsn:{lsn:8461098786}, max_flushed_end_lsn:{lsn:8461098908}, max_flushed_log_pid:45, committed_end_lsn:{lsn:8461098908}, last_slide_log_id:12632581, last_slide_scn:{val:1755161719775179002, v:0}, last_slide_lsn:{lsn:8461098786}, last_slide_end_lsn:{lsn:8461098908}, last_slide_log_pid:45, last_slide_log_accum_checksum:1091335677, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f992979c530}, [StateMgr]={this:0x7f99297a09b0, self:"172.77.0.7:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:46}, leader:"0.0.0.0:0", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:8461098908}, scan_disk_log_finished:true, last_check_start_id:12632582, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161899748977, palf_role_change_cb:0x7f99297a7470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:45, prev:{version:1, config:{config_version:{proposal_id:43, config_seq:41}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:45, config_seq:42}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:41, prev_lsn:{lsn:8461098218}, prev_mode_pid:45}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:41, prev_lsn:{lsn:8461098218}, prev_end_lsn:{lsn:8461098377}, prev_mode_pid:45}, reconfig_barrier:{prev_log_proposal_id:41, prev_lsn:{lsn:8461098218}, prev_end_lsn:{lsn:8461098377}, prev_mode_pid:45}, persistent_config_version:{proposal_id:45, config_seq:42}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755162183747293, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755162183747293, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f992979dc70}, [ModeMgr]={palf_id:1, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:45, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:46, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:46, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:1753719139394335001, v:0}, min_block_id:93, min_block_min_scn:{val:1753712284262753001, v:0}, base_lsn_for_block_gc:{lsn:8186781696}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:46}, log_config_meta:{version:3, proposal_id:45, prev:{version:1, config:{config_version:{proposal_id:43, config_seq:41}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:45, config_seq:42}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:41, prev_lsn:{lsn:8461098218}, prev_mode_pid:45}, log_snapshot_meta:{version:2, base_lsn:{lsn:8186781696}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:46, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:655360}, readable_log_tail:{lsn:655360}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8522305536}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/meta", dir_fd:214, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66449408, block_header_serialize_buf_:0x7f99297a2bb8, flashback_version:0}, log_storage:{log_tail:{lsn:8461098908}, readable_log_tail:{lsn:8461098908}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8455200768}, min_scn:{val:1755160384851871002, v:0}, curr_block_id:126, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:215, min_block_id:93, max_block_id:127, curr_writable_block_id:126}, logical_block_size_:67104768, curr_block_writable_size_:61206628, block_header_serialize_buf_:0x7f99297a48b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1755161900739661, this:0x7f99297a10f0}, [Reconfirm]={palf_id:1, self:"172.77.0.7:2882", state:"FETCH_MAX_LOG_LSN", new_proposal_id:46, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"172.77.0.7:2882", majority_max_accept_pid:45, majority_max_lsn:{lsn:8461098908}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:1755161954782466, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f99297a0b30}) [2025-08-14 09:03:12.777045] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=3994682] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161956777957, last_flushed_end_lsn:{lsn:8461098908}}]) [2025-08-14 09:03:12.777062] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=15] current log_task status(palf_id=1, self="172.77.0.7:2882", log_id=12632582, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161956777957, last_flushed_end_lsn:{lsn:8461098908}}]) [2025-08-14 09:03:12.777929] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [717]OB_MALLOC COST TOO MUCH TIME, cost_time=958306, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=958299], seq:[0] [2025-08-14 09:03:12.778480] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:12.778555] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:03:12.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:03:12.778564] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:03:12.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:12.790275] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=928419] gc stale ls task succ [2025-08-14 09:03:12.792288] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=13] PNIO [ratelimit] time: 1755162192792286, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007615, add_bytes: 0 [2025-08-14 09:03:12.808748] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:12.835212] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:13.252253] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:13.252267] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162193252249], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:13.252282] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162193252249], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:13.252296] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162193252249]}) [2025-08-14 09:03:13.252309] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:13.252315] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162193252307], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:13.252325] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162193252307]}) [2025-08-14 09:03:13.252489] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:03:13.252497] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:03:13.252507] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162193252486], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:13.252516] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162193252486]}) [2025-08-14 09:03:13.681166] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEB5-0-0] [lt=23] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:03:13.719442] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [856]OB_MALLOC COST TOO MUCH TIME, cost_time=904857, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=904851], seq:[0] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=900568, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=900566], seq:[0] [2025-08-14 09:03:13.726266] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:12.778583] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:13.726404] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] timer task use too much time(end=1755162193726404, start=1755162192778582, delta=947822) [2025-08-14 09:03:13.730265] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:66) [812][T1002_TransTime][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-08-14 09:03:13.730281] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:93) [812][T1002_TransTime][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-08-14 09:03:13.731539] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:13.736119] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=1971063] current monitor number(seq_=-1) [2025-08-14 09:03:12.835035] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:13.737276] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] timer task use too much time(end=1755162193737275, start=1755162192835033, delta=902242) [2025-08-14 09:03:13.016002] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:13.737323] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162193737322, start=1755162193016000, delta=721322) [2025-08-14 09:03:13.740800] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:12.774879] WDIAG [RPC] create (ob_poc_rpc_server.cpp:140) [316][pnio1][T0][YB42AC4D0007-00063C4960CB90F7-0-0] [lt=14][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1815, fly_ts=1045978, send_timestamp=1755162191728899, peer="172.77.0.7:49788", sz=265) [857]OB_MALLOC COST TOO MUCH TIME, cost_time=914193, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:03:13.749171] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:13.749214] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=1011047] blacklist_loop exec finished(cost_time=15, is_enabled=true, send_cnt=0) [2025-08-14 09:03:13.749612] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=444003240, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:13.749633] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:13.750166] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1001}, delta=449016609, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:13.750177] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:13.750202] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=444003240, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:13.750212] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:13.750696] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1001}, delta=449016609, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:13.750707] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [717]OB_MALLOC COST TOO MUCH TIME, cost_time=995097, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=45, WASH_BLOCK_END=22210, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=958422], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:10.759084] EDIAG [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=1985918][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2988363, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1993247, ObMalloc#start=0, ObMalloc#end=995110, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1d17bb52 0x1d17ada0 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:10.759084] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=1985918] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623081220090, flush_io_finished_ret_:0, flush_io_finished_round_:26335, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:03:13.752665] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=131] gc stale ls task succ [2025-08-14 09:03:13.752940] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7243] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:13.753012] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:13.753760] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=6] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:13.753786] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162193753755], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:13.753805] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162193753755], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:13.753821] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162193753755]}) [2025-08-14 09:03:13.753833] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:03:13.753844] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:03:13.753856] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=8] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:13.753865] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162193753854], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:13.753877] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162193753854]}) [2025-08-14 09:03:13.754056] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:03:13.754067] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:03:13.754083] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162193754051], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:13.754095] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162193754051]}) [770]OB_MALLOC COST TOO MUCH TIME, cost_time=915078, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=900577], seq:[0, 1] [2025-08-14 09:03:12.839270] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:13.755779] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162193755779, start=1755162192839267, delta=916512) [2025-08-14 09:03:13.756159] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=12990] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:13.756211] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:13.752706] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [856]OB_MALLOC COST TOO MUCH TIME, cost_time=926520, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=904872], seq:[0, 1] [2025-08-14 09:03:12.834805] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:13.762763] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=926553] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:03:13.762787] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:03:13.762800] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [403]OB_MALLOC COST TOO MUCH TIME, cost_time=984544, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=984534], seq:[0] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=919980, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=919975], seq:[0] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=919998, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=919996], seq:[0, 1] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=765051, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=765046], seq:[0] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=765073, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=765071], seq:[0, 1] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=984573, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=984565], seq:[0, 1] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=730305, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=730299], seq:[0] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=730316, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=730315], seq:[0, 1] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=491037, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=491032], seq:[0] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=491046, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=491043], seq:[0, 1] [2025-08-14 09:03:12.843278] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=645104759, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=253178, succ_cluster_heartbeat_count=2213, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161000606550000, v:0}, local_cluster_delta=1192236725, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:03:13.764787] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=920069] get_min_unreplayed_log_info(lsn={lsn:5282737655}, scn={val:1755100947825033001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161624848121, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:13063726, ret_code_:0}}, next_to_submit_lsn_:{lsn:5282737655}, next_to_submit_scn_:{val:1755100947825033001, v:0}, base_lsn_:{lsn:4831543296}, base_scn_:{val:1755063445733337001, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7605000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:5282737655}, end_lsn:{lsn:5282737655}, read_buf:{buf_len_:3747840, buf_:0x7f98f7605000}, block_size:67104768, log_storage_:0x7f993e529bf0, log_storage_:{log_tail:{lsn:5282737655}, readable_log_tail:{lsn:5282737655}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:5234171904}, min_scn:{val:1755152244802535004, v:0}, curr_block_id:78, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:206, min_block_id:42, max_block_id:79, curr_writable_block_id:78}, logical_block_size_:67104768, curr_block_writable_size_:18539017, block_header_serialize_buf_:0x7f993e52a8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:85, read_io_size:301367296, read_disk_cost_ts:2364604, start_lsn:{lsn:4831543296}}}}}}) [2025-08-14 09:03:13.765482] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=81] get wrs ts(ls_id={id:1}, delta=646025560, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:13.765512] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:13.766100] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=646025560, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:13.766117] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:12.999910] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:13.766191] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] timer task use too much time(end=1755162193766191, start=1755162192999907, delta=766284) [2025-08-14 09:03:12.778639] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:13.766278] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=986145][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:13.035041] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=699498] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:13.274692] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:03:13.766545] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=491087][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:03:13.766564] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:03:13.766583] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:03:13.766596] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:03:13.766618] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:03:13.766630] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:03:13.745, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:03:03.751], idx=659, next_allow_print_ts="09:03:43.745") [2025-08-14 09:03:13.766647] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:03:13.766662] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:03:13.766674] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:03:13.766687] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:03:13.766857] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:03:13.766873] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [404]OB_MALLOC COST TOO MUCH TIME, cost_time=363663, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=363654], seq:[0] [404]OB_MALLOC COST TOO MUCH TIME, cost_time=363683, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=363682], seq:[0, 1] [768]OB_MALLOC COST TOO MUCH TIME, cost_time=132101, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=132087], seq:[0] [768]OB_MALLOC COST TOO MUCH TIME, cost_time=132119, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=132118], seq:[0, 1] [2025-08-14 09:03:13.404133] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:13.636031] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [768][T1002_CoordTR][T1002][Y0-0000000000000000-0-0] [lt=16092][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:13.727303] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:13.768486] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=41118][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:13.729847] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=6] gc stale ls task succ [2025-08-14 09:03:13.730309] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:13.768546] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=38179][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:13.731215] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=1973674] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:13.731260] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=1968842] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:13.735305] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=1972854] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:03:13.735996] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:13.768660] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32613][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:13.736158] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:13.768692] WDIAG ev_invoke_pending (ev.c:2136) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32484][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:13.736464] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:13.768720] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=32207][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:13.768792] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:13.770718] WDIAG [COORDINATOR] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [471][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=975111][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:13.741204] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, param_config_:{memory_limit:536870912, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=865568, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:03:12.856249] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=473081069, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=252324, succ_cluster_heartbeat_count=1905, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161003593738002, v:0}, local_cluster_delta=1189262510, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:03:13.772658] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=914462] get wrs ts(ls_id={id:1}, delta=473996743, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:13.772677] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:13.773313] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=473997802, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:13.773327] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:13.777395] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:13.777817] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:13.779258] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:13.779291] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:13.780473] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:13.782180] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:13.790306] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=14] gc stale ls task succ [2025-08-14 09:03:13.800881] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=20] PNIO [ratelimit] time: 1755162193800879, bytes: 0, bw: 0.000000 MB/s, add_ts: 1008593, add_bytes: 0 [2025-08-14 09:03:13.825841] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=932055] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:03:13.825901] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=932067] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:03:13.832449] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:13.832948] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EED3-0-0] [lt=899312] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-08-14 09:03:13.837356] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:13.837568] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:13.842893] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=3893322] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:03:13.850295] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=444108764, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:13.850326] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:13.850895] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25] get wrs ts(ls_id={id:1001}, delta=449122133, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:13.850907] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:13.866225] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=646126698, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:13.866265] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:13.866295] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162193865621}, version_delta=646127779) [2025-08-14 09:03:13.873494] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=474098156, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:13.873524] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:13.950331] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=444208695, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:13.950355] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:13.950858] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1001}, delta=449222064, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:13.950870] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:13.958645] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:13.966331] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33] get wrs ts(ls_id={id:1}, delta=646226567, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:13.966363] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:13.973486] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=474197599, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:13.973512] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.018194] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:14.028793] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:14.050391] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=444308262, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:14.050414] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.050910] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1001}, delta=449322700, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:14.050921] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.066394] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33] get wrs ts(ls_id={id:1}, delta=646327860, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:14.066424] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.068431] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=730337] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:14.073535] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=34] get wrs ts(ls_id={id:1}, delta=474297621, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:14.073560] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.150477] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=444408997, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:14.150497] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.151027] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=449422366, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:14.151038] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.166341] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] get wrs ts(ls_id={id:1}, delta=646427599, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:14.166368] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.173620] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=474398392, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:14.173645] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.241703] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:14.250505] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119812272, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-08-14 09:03:14.250548] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=32] get wrs ts(ls_id={id:1}, delta=444508687, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:14.250561] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.251104] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1001}, delta=449522056, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:14.251115] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.254260] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:14.254284] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162194254254], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:14.254303] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162194254254], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:14.254321] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162194254254]}) [2025-08-14 09:03:14.254344] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=19] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:14.254351] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162194254342], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:14.254362] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162194254342]}) [2025-08-14 09:03:14.254370] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162194254342]}) [2025-08-14 09:03:14.254559] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:03:14.254579] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:03:14.254594] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162194254555], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:14.254605] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162194254555]}) [2025-08-14 09:03:14.254616] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] gts statistics(tenant_id=1002, gts_rpc_cnt=0, get_gts_cache_cnt=39, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:03:14.266460] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29] get wrs ts(ls_id={id:1}, delta=646527930, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:14.266491] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.267109] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:03:14.267147] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:03:14.267160] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:03:14.267181] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:03:14.267198] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:03:14.267217] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:03:14.267251] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:03:14.267270] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:03:14.267287] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:03:14.267313] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:03:14.267498] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:03:14.267517] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:03:14.273659] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=474497596, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:14.273691] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.350580] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=444608807, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:14.350609] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.351113] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1001}, delta=449623240, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:14.351124] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.366526] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36] get wrs ts(ls_id={id:1}, delta=646626333, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:14.366553] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.374798] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=474598855, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:14.374820] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.450623] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=444709212, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:14.450644] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.451164] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1001}, delta=449722581, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:14.451175] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.466601] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] get wrs ts(ls_id={id:1}, delta=646727785, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:14.466629] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.474864] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=42] get wrs ts(ls_id={id:1}, delta=474698543, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:14.474884] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.550684] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=444809152, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:14.550704] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.551209] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1001}, delta=449822521, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:14.551220] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.566644] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] get wrs ts(ls_id={id:1}, delta=646827709, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:14.566670] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.574929] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=474798396, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:14.574949] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.650778] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] get wrs ts(ls_id={id:1}, delta=444908955, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:14.650808] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.651327] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1001}, delta=449922324, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:14.651339] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.666796] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=646926815, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:14.666820] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.674978] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=474899140, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:14.675010] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=29] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.727598] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:14.727645] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:14.728270] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [471][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:0, time_interval:100.00ms, expected_run_time:09:03:14.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:14.729041] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=14] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:13.858997] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=865544][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a5c38, timer:0x7f994b3feee0, task:0x7f994b3fef20, task_type:"N9oceanbase8observer18ObTableLoadService15ObHeartBeatTaskE", scheduled_time:1755162132756830, pushed_time:0, delay:10000000}, running_task_token={this:0x7f994e1a5df8, timer:0x7f994b3feee0, task:0x7f994b3fef50, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162023758648, pushed_time:1755162122757495, delay:1000000}, total_delay=61102166, threashold=1000000, tenant_id=1) [2025-08-14 09:03:14.729530] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:14.729879] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=38620] gc stale ls task succ [2025-08-14 09:03:14.732882] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:14.735638] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:14.736524] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:14.736798] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:14.737208] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=9] current monitor number(seq_=-1) [2025-08-14 09:03:14.740614] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:13.771802] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=3004686] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:14.745952] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:14.749895] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:14.753317] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8851] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:14.753878] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=21] gc stale ls task succ [2025-08-14 09:03:14.755153] INFO print_info (ob_kvcache_hazard_domain.cpp:138) [812][T1002_TransTime][T1002][Y0-0000000000000000-0-0] [lt=7] [KVCACHE-HAZARD-POINTER] retired memory size: 0, blocks size: 94, free list size: 1003, memory_efficiency: 0.897691 [2025-08-14 09:03:14.755367] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:14.755407] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:14.756051] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=11] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:14.756072] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162194756047], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:14.756090] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162194756047], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:14.756104] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162194756047]}) [2025-08-14 09:03:14.756116] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=30, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:03:14.756132] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:03:14.756145] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:03:14.756157] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:14.756170] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162194756156], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:14.756178] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162194756156]}) [2025-08-14 09:03:14.756384] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:03:14.756396] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:03:14.756409] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162194756381], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:14.756421] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162194756381]}) [2025-08-14 09:03:14.756776] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=36] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:14.757043] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get_max_applied_scn(scn={val:1755161749741193000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:50, palf_committed_end_lsn_:{lsn:1604090090}, palf_committed_end_scn_:{val:1755161749741193000, v:0}, last_check_scn_:{val:1755161749741193001, v:0}, max_applied_cb_scn_:{val:1755161749741193000, v:0}}) [2025-08-14 09:03:14.757070] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] get_min_unreplayed_log_info(lsn={lsn:1604088821}, scn={val:1755100948296325001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873479, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:12492179, ret_code_:0}}, next_to_submit_lsn_:{lsn:1604088821}, next_to_submit_scn_:{val:1755100948296325001, v:0}, base_lsn_:{lsn:1409200128}, base_scn_:{val:1753840802540914000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:1604088821}, end_lsn:{lsn:1604088821}, read_buf:{buf_len_:3747840, buf_:0x7f98f6a05000}, block_size:67104768, log_storage_:0x7f990bfebbf0, log_storage_:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:42, read_io_size:147709952, read_disk_cost_ts:1061450, start_lsn:{lsn:1409200128}}}}}}) [2025-08-14 09:03:14.757162] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=89] get wrs ts(ls_id={id:1}, delta=445015896, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:14.757188] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.767726] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:03:14.767773] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:03:14.767786] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:03:14.767803] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:03:14.767815] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:03:14.767833] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:03:14.767851] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:03:14.767873] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:03:14.767884] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:03:14.767896] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:03:14.767909] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:03:14.768082] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:03:14.768106] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:03:14.769199] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:14.771772] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=647032508, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:14.771801] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.772471] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:14.775039] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=474998071, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:14.775069] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=27] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.778006] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:14.778300] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:14.778376] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:14.790870] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:14.794496] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:03:14.807533] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=26] PNIO [ratelimit] time: 1755162194807531, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006652, add_bytes: 0 [2025-08-14 09:03:14.837991] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:14.838580] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:14.870583] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=647131207, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:14.870613] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.878936] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=475102991, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:14.878963] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.964257] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:14.970730] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=647231485, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:14.970776] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=43] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:14.984881] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=28] get wrs ts(ls_id={id:1}, delta=475209597, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:14.984916] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=31] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:15.069540] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=17] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:15.070750] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=647331851, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:15.070775] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:15.084888] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=475308893, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:15.084913] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:15.098461] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:15.170883] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=647431019, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:15.170916] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:15.185032] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] get wrs ts(ls_id={id:1}, delta=475408569, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:15.185057] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:15.250646] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:15.256655] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=18] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:15.256679] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162195256651], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:15.256698] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162195256651], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:15.256714] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162195256651]}) [2025-08-14 09:03:15.256730] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=12] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:15.256738] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162195256728], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:15.256759] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=20] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162195256728]}) [2025-08-14 09:03:15.256950] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:03:15.256962] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:03:15.256984] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162195256945], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:15.256996] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162195256945]}) [2025-08-14 09:03:15.268259] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:03:15.268299] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:03:15.268317] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:03:15.268351] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:03:15.268367] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:03:15.268401] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:03:15.268415] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:03:15.268440] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:03:15.268454] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:03:15.268466] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:03:15.268475] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:03:15.268645] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:03:15.268663] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:03:15.270337] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161101770923, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755161572801635) [2025-08-14 09:03:15.270955] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] get wrs ts(ls_id={id:1}, delta=647531641, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:15.270970] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:15.278940] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:15.285054] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=475508771, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:15.285076] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:15.371024] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] get wrs ts(ls_id={id:1}, delta=647631802, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:15.371047] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:15.385117] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=475608951, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:15.385139] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:15.471117] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=647731413, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:15.471150] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:15.485156] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=475708833, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:15.485178] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:15.571136] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=647831633, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:15.571160] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:15.585197] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=475809755, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:15.585220] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:15.671217] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=647932164, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:15.671260] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=40] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:15.685249] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=34] get wrs ts(ls_id={id:1}, delta=475909275, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:15.685275] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:15.719045] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=23] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:15.720268] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=37296] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:13.769934] EDIAG [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=2993984][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1950415, owner: ObLog, click_count: 7, time dist:[FORMAT_END=1950403, ObMalloc#start=1, ObMalloc#end=4, ALLOC_END=0, APPEND_END=6, ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=2], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1bf2ec62 0x986ebd7 0x986bee6 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:13.769934] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=2993984] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:14.756719] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=1988930] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:03:15.727738] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:15.727781] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:15.728314] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:15.730694] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=53] gc stale ls task succ [2025-08-14 09:03:14.755050] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=2011625] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:03:15.737329] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:15.737361] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:15.737385] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:15.738869] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=37305] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:15.741851] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:15.743054] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:15.745882] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:15.747032] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:15.749287] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:15.752409] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:15.753944] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=18] gc stale ls task succ [2025-08-14 09:03:15.755304] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:15.760931] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=20] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:14.763044] EDIAG [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=2988363][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1000036, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1000025, ObMalloc#start=2, ObMalloc#end=3, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1d189030 0x1d18285e 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:14.763044] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=2988363] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:03:15.768926] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:03:15.768958] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:03:15.768969] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:03:15.768986] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:03:15.768998] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:03:15.769016] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:03:15.769032] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:03:15.769052] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:03:15.769063] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:03:15.769075] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:03:15.769088] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:03:15.769637] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:1131) [812][T1002_TransTime][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:15.769670] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:826) [812][T1002_TransTime][T1002][Y0-0000000000000000-0-0] [lt=23] can not find enough memory block to wash(ret=-4273, size_washed=0, size_to_wash=6291456) [2025-08-14 09:03:15.769874] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:03:15.769896] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:03:15.775221] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:15.775274] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:15.776086] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=648014818, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:15.776108] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:15.776678] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:15.776715] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162195776674], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:15.776739] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162195776674], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:15.776762] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162195776674]}) [2025-08-14 09:03:15.776778] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162195776674]}) [2025-08-14 09:03:15.776797] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=15] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:15.776806] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162195776795], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:15.776820] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162195776795]}) [2025-08-14 09:03:15.776999] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:03:15.777019] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:03:15.777031] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162195776994], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:15.777047] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162195776994]}) [2025-08-14 09:03:15.777054] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162195776994]}) [2025-08-14 09:03:15.778676] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:15.778709] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:15.778999] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:15.780786] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:15.785325] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=475978155, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:15.785352] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:15.794554] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=23] gc stale ls task succ [2025-08-14 09:03:15.815133] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=24] PNIO [ratelimit] time: 1755162195815131, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007600, add_bytes: 0 [2025-08-14 09:03:15.841853] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=868765][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a5d18, timer:0x7f994b3feee0, task:0x7f994b3fef38, task_type:"N9oceanbase8observer18ObTableLoadService8ObGCTaskE", scheduled_time:1755162074747216, pushed_time:0, delay:30000000}, running_task_token={this:0x7f994e1a5df8, timer:0x7f994b3feee0, task:0x7f994b3fef50, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162023758648, pushed_time:1755162122757495, delay:1000000}, total_delay=121094636, threashold=1000000, tenant_id=1) [2025-08-14 09:03:15.876143] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=648014818, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:15.876165] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:15.885427] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] get wrs ts(ls_id={id:1}, delta=475978155, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:15.885463] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=32] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:15.976246] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=648014818, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:15.976268] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:15.985445] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] get wrs ts(ls_id={id:1}, delta=475978155, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:15.985472] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:16.070699] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=30] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:16.076274] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=648014818, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:16.076295] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:16.085499] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] get wrs ts(ls_id={id:1}, delta=475978155, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:16.085525] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:16.102481] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a7058, timer:0x7f994b1dfb70, task:0x7f98ebbf7c30, task_type:"N9oceanbase8observer26ObTableLoadResourceManager21ObRefreshAndCheckTaskE", scheduled_time:1755161774744853, pushed_time:0, delay:30000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=421357626, threashold=1000000, tenant_id=1002) [2025-08-14 09:03:16.176347] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=648014818, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:16.176369] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:16.185545] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=31] get wrs ts(ls_id={id:1}, delta=475978155, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:16.185578] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=31] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:16.270037] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:03:16.270070] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:03:16.270083] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:03:16.270101] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:03:16.270111] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:03:16.270137] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:03:16.270153] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:03:16.270170] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:03:16.270182] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:03:16.270201] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:03:16.270213] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:03:16.270389] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:03:16.270403] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:03:16.276394] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=648014818, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:16.276414] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:16.277299] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=12] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:16.277326] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162196277295], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:16.277351] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=22] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162196277295]}) [2025-08-14 09:03:16.277369] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=14] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:16.277374] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162196277367], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:16.277399] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=23] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162196277367]}) [2025-08-14 09:03:16.277575] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:03:16.277591] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:03:16.277602] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162196277564], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:16.277619] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162196277564]}) [2025-08-14 09:03:16.285621] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=28] get wrs ts(ls_id={id:1}, delta=475978155, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:16.285646] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:16.376468] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=648014818, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:16.376491] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:16.385659] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] get wrs ts(ls_id={id:1}, delta=475978155, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:16.385685] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:16.476540] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=648014818, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:16.476562] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:16.485724] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=475978155, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:16.485748] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:16.576592] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] get wrs ts(ls_id={id:1}, delta=648014818, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:16.576619] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:16.585811] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=31] get wrs ts(ls_id={id:1}, delta=475978155, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:16.585837] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:16.676659] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=648014818, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:16.676684] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:16.685810] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=30] get wrs ts(ls_id={id:1}, delta=475978155, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:16.685846] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:16.691811] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEB7-0-0] [lt=24] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:03:16.718669] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=33283] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:03:16.726787] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:16.727477] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:16.727512] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:16.729211] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=969152] tx gc loop thread is running(MTL_ID()=1) [2025-08-14 09:03:16.729244] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=31] try gc retain ctx [2025-08-14 09:03:16.729253] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] try to retry start_working [2025-08-14 09:03:16.729263] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:125) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] try to adjust log cb pool [2025-08-14 09:03:16.730710] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=21] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:16.731128] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=16] gc stale ls task succ [2025-08-14 09:03:16.732322] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:15.769683] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:845) [812][T1002_TransTime][T1002][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_to_wash=6291456) [2025-08-14 09:03:15.511158] EDIAG [COMMON] generate_mod_stat_task (memory_dump.cpp:213) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1224455, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1224451, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf98b221 0xf98aea2 0x96b1d30 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:16.737370] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:16.737406] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:16.737433] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:15.511158] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:213) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=17] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-08-14 09:03:16.737505] INFO [COMMON] handle (memory_dump.cpp:544) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=1224455] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-08-14 09:03:16.740998] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:16.742511] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=37] current monitor number(seq_=-1) [2025-08-14 09:03:16.745872] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:16.746001] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:16.750256] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:16.754009] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:03:16.754284] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:16.755312] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:16.761022] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=23] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:16.763367] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=31] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:16.769320] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=1950415] tx gc loop thread is running(MTL_ID()=1001) [2025-08-14 09:03:16.769335] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=15] try gc retain ctx [2025-08-14 09:03:16.769344] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=8] try to retry start_working [2025-08-14 09:03:16.770538] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:03:16.770570] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:03:16.770582] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:03:16.770599] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:03:16.770617] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:03:16.770634] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:03:16.770650] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:03:16.770669] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:03:16.770680] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:03:16.770691] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:03:16.770704] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:03:16.770882] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:03:16.770896] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:03:16.773281] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:16.777866] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=14] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:16.777896] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162196777862], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:16.777920] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162196777862]}) [2025-08-14 09:03:16.777948] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=17] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:16.777956] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162196777939], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:16.777970] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162196777939]}) [2025-08-14 09:03:16.777981] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts statistics(tenant_id=1001, gts_rpc_cnt=0, get_gts_cache_cnt=67, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:03:16.778179] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:03:16.778196] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:03:16.778208] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162196778175], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:16.778225] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162196778175]}) [2025-08-14 09:03:16.778689] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:16.778719] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:16.794576] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=22] gc stale ls task succ [2025-08-14 09:03:16.822769] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=26] PNIO [ratelimit] time: 1755162196822767, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007636, add_bytes: 0 [2025-08-14 09:03:16.842737] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=43][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:16.848534] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EED5-0-0] [lt=32] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-08-14 09:03:16.948265] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=1389098][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:17.074473] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=18] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:17.264128] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=947108][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc41f8, timer:0x7f9970f40f80, task:0x564d6d07c3b0, task_type:"N9oceanbase5share22ObDeviceCredentialTaskE", scheduled_time:1755162016246491, pushed_time:0, delay:1200000000}, running_task_token={this:0x7f9970fc5fb8, timer:0x7f9970f40f80, task:0x564d727182f0, task_type:"N9oceanbase5share16ObAshRefreshTaskE", scheduled_time:1755161611717613, pushed_time:1755161614757742, delay:120000000}, total_delay=181017636, threashold=1000000, tenant_id=500) [2025-08-14 09:03:17.271066] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:03:17.271104] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:03:17.271120] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:03:17.271149] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:03:17.271165] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:03:17.271183] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:03:17.271204] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:03:17.271220] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:03:17.271242] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:03:17.271257] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:03:17.271276] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:03:17.271455] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:03:17.271473] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:03:17.278469] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162197278463], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:17.278503] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=31] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162197278463]}) [2025-08-14 09:03:17.278529] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162197278527], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:17.278543] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162197278527]}) [2025-08-14 09:03:17.278555] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162197278527]}) [2025-08-14 09:03:17.278748] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:03:17.278770] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:03:17.278785] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162197278744], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:17.278799] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162197278744]}) [2025-08-14 09:03:17.719737] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=21] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:16.725580] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2826) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=29793] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:2030, [delay/us]: prepare:0, schedule:0, submit:600000000, device:499338, total:499343) [2025-08-14 09:03:17.726021] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=998915] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:03:17.727281] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:17.727325] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:17.729284] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:17.731189] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:17.731256] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=18] gc stale ls task succ [2025-08-14 09:03:17.732262] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:17.732313] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=26] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [729]OB_MALLOC COST TOO MUCH TIME, cost_time=949485, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:03:16.742137] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4015] clock out of order(cur_ts=1755162195777499, cur_ts_=1755162195753334, delta=24165) [2025-08-14 09:03:16.580253] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1186654, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1186653, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:17.736014] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:66) [812][T1002_TransTime][T1002][Y0-0000000000000000-0-0] [lt=963812][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-08-14 09:03:17.736042] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:93) [812][T1002_TransTime][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-08-14 09:03:17.737016] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=475978155, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:17.737044] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:17.737766] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] get wrs ts(ls_id={id:1}, delta=475978155, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:17.737786] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:17.743795] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=44] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:17.751161] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:17.751190] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:17.761110] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=22] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:17.764442] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:17.764517] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:17.765573] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc5618, timer:0x564d6efbde60, task:0x564d6efbde20, task_type:"N9oceanbase12blocksstable14ObBlockManager13MarkBlockTaskE", scheduled_time:1755162196749035, pushed_time:0, delay:30000000}, running_task_token={this:0x7f9970fc5538, timer:0x564d6efbde60, task:0x564d6efbde38, task_type:"N9oceanbase12blocksstable14ObBlockManager19InspectBadBlockTaskE", scheduled_time:1755162049755883, pushed_time:1755162166774569, delay:1000000}, total_delay=1016536, threashold=1000000, tenant_id=500) [2025-08-14 09:03:16.580253] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:17.768735] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=1186654][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:17.768749] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task use too much time(end=1755162197768749, start=1755162196580251, delta=1188498) [2025-08-14 09:03:17.778782] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:17.778807] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [567]OB_MALLOC COST TOO MUCH TIME, cost_time=1938391, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [812]OB_MALLOC COST TOO MUCH TIME, cost_time=12948589, tenant_id=1002, label=OB_TX_COMMIT, ctx_id=16, prio=0, size=16448, owner: ALLOC_BLOCK, click_count: 27, time dist:[ObMalloc#start=4961658, ObMalloc#end=17462, ObLog#start=25747, ObLog#end=4960, ObLog#start=980823, ObLog#end=970162, ObLog#start=976471, ObLog#end=33594, ObLog#start=977895, ObMalloc#start=989302, ObLog#start=33990, ObLog#start=1012929, ObLog#end=963812, ObLog#start=999550, ObLog#end=28, ObLog#end=47], seq:[1, 2, 3, 4, 5, 6, 7, 8, 11, 17, 19, 21, 23, 24, 25, 26] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=944818, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 16, time dist:[ALLOC_CHUNK_END=15, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=64, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=78, ALLOC_CHUNK_END=22, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=944583, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] [276]OB_MALLOC COST TOO MUCH TIME, cost_time=654424, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 14, time dist:[ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=654335, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13] [729]OB_MALLOC COST TOO MUCH TIME, cost_time=920355, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16448, [2025-08-14 09:03:14.770266] EDIAG [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=984386][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4936525, owner: ObLog, click_count: 3, time dist:[FORMAT_END=4936523, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x99721a1 0x997a72e 0x2006ed17 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [471]OB_MALLOC COST TOO MUCH TIME, cost_time=1976792, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1976785], seq:[0] [471]OB_MALLOC COST TOO MUCH TIME, cost_time=1977129, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=71, WASH_BLOCK_END=237, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1976811], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:13.862865] EDIAG [RS] get_valid_sts_after (ob_tenant_info_loader.cpp:454) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5851369, owner: ObLog, click_count: 3, time dist:[FORMAT_END=5851368, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1001d942 0x996d843 0x996d15a 0x996cfd8 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [399]OB_MALLOC COST TOO MUCH TIME, cost_time=1977246, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=47061], seq:[0, 1] [2025-08-14 09:03:17.743176] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1977436, owner: ObLog, click_count: 5, time dist:[FORMAT_END=7, ObMalloc#start=1, ObMalloc#end=1977279, ALLOC_END=148, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [812]OB_MALLOC COST TOO MUCH TIME, cost_time=14933318, tenant_id=1002, label=OB_TX_COMMIT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=13929030], seq:[0, 1] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=1977866, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=47864], seq:[0, 1] [2025-08-14 09:03:17.743084] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1977895, owner: ObLog, click_count: 5, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=1977877, ALLOC_END=4, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [383]OB_MALLOC COST TOO MUCH TIME, cost_time=1977843, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=47770], seq:[0, 1] [2025-08-14 09:03:17.743196] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1977860, owner: ObLog, click_count: 5, time dist:[FORMAT_END=6, ObMalloc#start=0, ObMalloc#end=1977852, ALLOC_END=2, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:10.738619] EDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=901688][errcode=0] LOGGER COST TOO MUCH TIME, cost: 68884323, owner: ObLog, click_count: 5, time dist:[FORMAT_END=68884309, ObMalloc#start=2, ObMalloc#end=11, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ed35c 0x97aead0 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:17.743658] EDIAG [STORAGE] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:123) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1980699, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1980696, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1c039e1e 0x97a8b06 0x97a7481 0x97a6a43 0x97a42c5 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [705]OB_MALLOC COST TOO MUCH TIME, cost_time=1982352, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1982340, ALLOC_CHUNK_END=5], seq:[0, 1] [822]OB_MALLOC COST TOO MUCH TIME, cost_time=1982641, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1982630, ALLOC_CHUNK_END=6], seq:[0, 1] [357]OB_MALLOC COST TOO MUCH TIME, cost_time=1982653, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1982643, ALLOC_CHUNK_END=5], seq:[0, 1] [389]OB_MALLOC COST TOO MUCH TIME, cost_time=1979881, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1979871, ALLOC_CHUNK_END=5], seq:[0, 1] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=1974169, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1974159, ALLOC_CHUNK_END=5], seq:[0, 1] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=1974034, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1974024, ALLOC_CHUNK_END=6], seq:[0, 1] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=1972472, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1972462, ALLOC_CHUNK_END=5], seq:[0, 1] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=1971227, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1971218, ALLOC_CHUNK_END=5], seq:[0, 1] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=1967478, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1967468, ALLOC_CHUNK_END=6], seq:[0, 1] [386]OB_MALLOC COST TOO MUCH TIME, cost_time=1963126, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1963115, ALLOC_CHUNK_END=6], seq:[0, 1] [303]OB_MALLOC COST TOO MUCH TIME, cost_time=1961168, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1961157, ALLOC_CHUNK_END=4], seq:[0, 1] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=1948034, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1948024, ALLOC_CHUNK_END=5], seq:[0, 1] [2025-08-14 09:03:17.795418] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:17.795433] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162197795415], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:17.795448] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162197795415], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:17.795461] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162197795415]}) [2025-08-14 09:03:17.795470] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:03:17.795477] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:03:17.795486] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=5] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:17.795491] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162197795484], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:17.795500] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162197795484]}) [2025-08-14 09:03:17.795663] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:03:17.795748] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=44][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:17.796199] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=23] gc stale ls task succ [2025-08-14 09:03:17.830367] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=23] PNIO [ratelimit] time: 1755162197830365, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007598, add_bytes: 0 [2025-08-14 09:03:17.903720] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6b18, timer:0x7f994b1dfb70, task:0x7f9920110718, task_type:"N9oceanbase10rootserver25ObMViewPushRefreshScnTaskE", scheduled_time:1755161774754765, pushed_time:0, delay:30000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=423148953, threashold=1000000, tenant_id=1002) [2025-08-14 09:03:17.903751] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a73d8, timer:0x7f994b1dfb70, task:0x7f9920110748, task_type:"N9oceanbase10rootserver22ObReplicaSafeCheckTaskE", scheduled_time:1755161774754766, pushed_time:0, delay:0}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=423148984, threashold=1000000, tenant_id=1002) [2025-08-14 09:03:17.903776] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6cd8, timer:0x7f994b1dfb70, task:0x7f99201107e0, task_type:"N9oceanbase10rootserver24ObCollectMvMergeInfoTaskE", scheduled_time:1755161774755618, pushed_time:0, delay:30000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=423148158, threashold=1000000, tenant_id=1002) [2025-08-14 09:03:18.718669] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=48] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:18.729165] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:18.729804] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:18.732668] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=13] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:03:17.778545] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:17.752090] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [726][T1001_ObXAHbWor][T1001][Y0-0000000000000000-0-0] [lt=1971489] XA scheduler heartbeat task statistics(avg_time=1) [2025-08-14 09:03:18.745325] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=31] current monitor number(seq_=-1) [2025-08-14 09:03:18.800350] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=8] gc stale ls task succ [2025-08-14 09:03:18.829747] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=26] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:03:18.837983] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=16] PNIO [ratelimit] time: 1755162198837981, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007616, add_bytes: 0 [2025-08-14 09:03:19.700863] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEB9-0-0] [lt=24] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:03:19.706596] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:14.770266] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=984386] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:19.719510] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=14] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:19.720630] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [556][T1_OBJLockGC][T1][Y0-0000000000000000-0-0] [lt=302][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:17.742295] WDIAG [COORDINATOR] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [471][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:17.743176] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:19.723632] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1977436][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:17.743084] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:19.724000] WDIAG ev_invoke_pending (ev.c:2136) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1977895][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:19.724020] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:17.743196] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:19.724069] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1977860][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:19.724085] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:10.738619] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=901688][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:02:09.835|207=903.05ms|210=0us|222=1us|total=69.89s) [2025-08-14 09:03:19.727157] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=68884323][errcode=-4389] timer task use too much time(end=1755162199727156, start=1755162129835562, delta=69891594) [2025-08-14 09:03:19.727171] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:17.743658] INFO [STORAGE] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:123) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25] wead read timestamp is too old(timestamp={val:1755161749741193000, v:0}, gts_scn={val:1755161762721798002, v:0}, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1755161749741193000, v:0}}) [2025-08-14 09:03:19.729150] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:19.729644] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=1980699] get_max_applied_scn(scn={val:1755161744727824001, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:49, palf_committed_end_lsn_:{lsn:7323686009}, palf_committed_end_scn_:{val:1755161747742105001, v:0}, last_check_scn_:{val:1755161749741193000, v:0}, max_applied_cb_scn_:{val:1755161744727824001, v:0}}) [2025-08-14 09:03:19.729671] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] get_min_unreplayed_log_info(lsn={lsn:7323685240}, scn={val:1755100946796590002, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873485, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:8946764, ret_code_:0}}, next_to_submit_lsn_:{lsn:7323685240}, next_to_submit_scn_:{val:1755100946796590002, v:0}, base_lsn_:{lsn:7180210176}, base_scn_:{val:1753840803166611000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:7323685240}, end_lsn:{lsn:7323685240}, read_buf:{buf_len_:3747840, buf_:0x7f98f7a05000}, block_size:67104768, log_storage_:0x7f9901131bf0, log_storage_:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:28, read_io_size:98508800, read_disk_cost_ts:645339, start_lsn:{lsn:7180210176}}}}}}) [400]OB_MALLOC COST TOO MUCH TIME, cost_time=1778196, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1778151], seq:[0, 1, 2, 3, 4, 5, 6] [400]OB_MALLOC COST TOO MUCH TIME, cost_time=1778211, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1778209], seq:[0, 1] [2025-08-14 09:03:17.949091] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1778262, owner: ObLog, click_count: 5, time dist:[FORMAT_END=25, ObMalloc#start=1, ObMalloc#end=1778231, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [419]OB_MALLOC COST TOO MUCH TIME, cost_time=1651962, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1651915], seq:[0, 1, 2, 3, 4, 5, 6] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=1651996, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1651996], seq:[0, 1] [2025-08-14 09:03:18.075603] EDIAG try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1652018, owner: ObLog, click_count: 5, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=1652002, ALLOC_END=0, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x26499505 0x97074a3 0x26498fa9 0x7f9973e081cf 0x7f9973a39dd3 [567]OB_MALLOC COST TOO MUCH TIME, cost_time=1010978, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=18, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1010930], seq:[0, 1, 2, 3, 4, 5, 6] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=1010987, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1010986], seq:[0, 1] [2025-08-14 09:03:18.717601] EDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1011039, owner: ObLog, click_count: 5, time dist:[FORMAT_END=25, ObMalloc#start=18, ObMalloc#end=1010992, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0xfa06846 0xfa05f62 0xfa05c84 0xfa05a50 0x1c020b34 0x97a4183 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:17.949091] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:18.075603] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=21] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:18.717601] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161101770923, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755161572801635) [2025-08-14 09:03:19.730200] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1011039] get_max_applied_scn(scn={val:1755161547738516001, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:50, palf_committed_end_lsn_:{lsn:5282737655}, palf_committed_end_scn_:{val:1755161547738516001, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161547738516001, v:0}}) [2025-08-14 09:03:19.730222] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] get_min_unreplayed_log_info(lsn={lsn:5282737655}, scn={val:1755100947825033001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161624848121, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:13063726, ret_code_:0}}, next_to_submit_lsn_:{lsn:5282737655}, next_to_submit_scn_:{val:1755100947825033001, v:0}, base_lsn_:{lsn:4831543296}, base_scn_:{val:1755063445733337001, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7605000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:5282737655}, end_lsn:{lsn:5282737655}, read_buf:{buf_len_:3747840, buf_:0x7f98f7605000}, block_size:67104768, log_storage_:0x7f993e529bf0, log_storage_:{log_tail:{lsn:5282737655}, readable_log_tail:{lsn:5282737655}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:5234171904}, min_scn:{val:1755152244802535004, v:0}, curr_block_id:78, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:206, min_block_id:42, max_block_id:79, curr_writable_block_id:78}, logical_block_size_:67104768, curr_block_writable_size_:18539017, block_header_serialize_buf_:0x7f993e52a8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:85, read_io_size:301367296, read_disk_cost_ts:2364604, start_lsn:{lsn:4831543296}}}}}}) [2025-08-14 09:03:19.730340] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=88] get wrs ts(ls_id={id:1001}, delta=454993415, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:19.730364] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:19.731045] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=75] get wrs ts(ls_id={id:1}, delta=651982723, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:19.731066] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:19.731099] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=651992579, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=253208, succ_cluster_heartbeat_count=2213, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161000606550000, v:0}, local_cluster_delta=1199124545, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:03:19.732901] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=8] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:03:17.795274] EDIAG [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=980080][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1937012, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1937001, ObMalloc#start=1, ObMalloc#end=4, ALLOC_END=0, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x99721a1 0x997a72e 0x2006ed17 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:17.795671] EDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1936702, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1936695, ObMalloc#start=0, ObMalloc#end=3, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x1b511e3c 0x95df1fc 0x95ded4c 0x1b4f8508 0x1b4fbf53 0x1b5014e0 0x1b5009d1 0x1b50387c 0x1b8905de 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [736]OB_MALLOC COST TOO MUCH TIME, cost_time=1012807, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1012758], seq:[0, 1, 2, 3, 4, 5, 6] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=1012819, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1012817], seq:[0, 1] [2025-08-14 09:03:18.721032] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1012851, owner: ObLog, click_count: 5, time dist:[FORMAT_END=22, ObMalloc#start=1, ObMalloc#end=1012825, ALLOC_END=0, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [858]OB_MALLOC COST TOO MUCH TIME, cost_time=1002637, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1002603], seq:[0, 1, 2, 3, 4, 5, 6] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=1002648, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1002647], seq:[0, 1] [2025-08-14 09:03:18.731319] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=18][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1002672, owner: ObLog, click_count: 5, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=1002656, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:17.795274] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=980080] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:19.714305] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=4936525] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:17.795671] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=8] get number(ret=-4038, service_type_=0, range=1, base_id=1755162197794857000, start_id=0, end_id=140295968341544) [2025-08-14 09:03:19.735330] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=1936702][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:03:19.735359] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:18.721032] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=30] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:18.731319] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=18] gc stale ls task succ [2025-08-14 09:03:19.735527] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=1002672] gc stale ls task succ [2025-08-14 09:03:19.735550] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:19.735574] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [574]OB_MALLOC COST TOO MUCH TIME, cost_time=1001582, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1001548], seq:[0, 1, 2, 3, 4, 5, 6] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=997516, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=17, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=997468], seq:[0, 1, 2, 3, 4, 5, 6] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=1001605, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1001603], seq:[0, 1] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=997527, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=997527], seq:[0, 1] [2025-08-14 09:03:18.732504] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1001633, owner: ObLog, click_count: 5, time dist:[FORMAT_END=14, ObMalloc#start=0, ObMalloc#end=1001612, ALLOC_END=2, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [384]OB_MALLOC COST TOO MUCH TIME, cost_time=1001624, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1001589], seq:[0, 1, 2, 3, 4, 5, 6] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=1001635, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1001634], seq:[0, 1] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=993404, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=993372], seq:[0, 1, 2, 3, 4, 5, 6] [2025-08-14 09:03:18.732566] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1001653, owner: ObLog, click_count: 5, time dist:[FORMAT_END=8, ObMalloc#start=0, ObMalloc#end=1001640, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [395]OB_MALLOC COST TOO MUCH TIME, cost_time=984242, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=984208], seq:[0, 1, 2, 3, 4, 5, 6] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=984253, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=984253], seq:[0, 1] [388]OB_MALLOC COST TOO MUCH TIME, cost_time=988337, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=988299], seq:[0, 1, 2, 3, 4, 5, 6] [388]OB_MALLOC COST TOO MUCH TIME, cost_time=988347, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=988347], seq:[0, 1] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=973575, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=973532, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=975399, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=19, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=975347, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=973211, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=973174, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=964881, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=964840, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=958085, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=958050, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3, 4, 5, 6, 7] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=956817, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=956768, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=956025, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=17, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=955976, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=955860, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=955825, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [376]OB_MALLOC COST TOO MUCH TIME, cost_time=939444, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=939403, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [553]OB_MALLOC COST TOO MUCH TIME, cost_time=905437, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=905397, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [833]OB_MALLOC COST TOO MUCH TIME, cost_time=890362, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=890325, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=975394, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=18, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=975345], seq:[0, 1, 2, 3, 4, 5, 6] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=975948, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=975945], seq:[0, 1] [2025-08-14 09:03:17.849285] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1884280, owner: ObLog, click_count: 5, time dist:[FORMAT_END=908317, ObMalloc#start=1, ObMalloc#end=975955, ALLOC_END=3, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [396]OB_MALLOC COST TOO MUCH TIME, cost_time=993416, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=993415], seq:[0, 1] [2025-08-14 09:03:18.732504] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=30] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:18.732566] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:19.735801] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1001653][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:18.750056] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:19.735820] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:19.735821] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=984275][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:18.745989] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:19.735878] WDIAG ev_invoke_pending (ev.c:2136) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=988394][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:17.849285] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:19.736437] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1884280][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:19.736466] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4389] timer task use too much time(end=1755162199736465, start=1755162197849283, delta=1887182) [2025-08-14 09:03:18.740840] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:19.736504] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=994158][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=766084, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=766057], seq:[0, 1, 2, 3, 4, 5, 6] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=766100, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=766099], seq:[0, 1] [2025-08-14 09:03:18.969637] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=1398839][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:19.739661] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:17.842090] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=910758][errcode=-4018] LOGGER COST TOO MUCH TIME, cost: 1899769, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1899760, ObMalloc#start=2, ObMalloc#end=3, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3b4b 0x113d34aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:00:35.773948] EDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [664][T1001_COMMONLSS][T1001][YB42AC4D0007-00063C496439FDEB-0-0] [lt=16][errcode=-4283] LOGGER COST TOO MUCH TIME, cost: 163723302, owner: ObLog, click_count: 5, time dist:[FORMAT_END=163723294, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x124fe4b5 0x972619d 0x124dfd77 0x96a5a4d 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x99ba990 0x1ee9bc6a 0x10052c69 0x9b44909 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:19.723658] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:19.742836] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=0] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:19.735257] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=1937012] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:17.842090] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=910758][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:03:18.758866] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:288) [570][T1_ArcFetcher][T1][YB42AC4D0007-00063C495B999C16-0-0] [lt=7582] ObArchiveFetcher is running(thread_index=0) [2025-08-14 09:03:18.944724] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [880][T1002_ArcSender][T1002][YB42AC4D0007-00063C495EB99C16-0-0] [lt=49512] ObArchiveSender is running(thread_index=5) [2025-08-14 09:00:35.773948] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [664][T1001_COMMONLSS][T1001][YB42AC4D0007-00063C496439FDEB-0-0] [lt=16][errcode=-4283] result set close failed(ret=-4283) [2025-08-14 09:03:17.819096] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=738760][errcode=-4018] LOGGER COST TOO MUCH TIME, cost: 1923401, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1923393, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3b4b 0x113d34aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:17.819096] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=738760][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:03:19.745410] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=12] current monitor number(seq_=-1) [2025-08-14 09:03:19.750034] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:16.778990] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2969081, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=3, FORMAT_END=2969069, ObMalloc#start=0, ObMalloc#end=2, ALLOC_END=0, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [474]OB_MALLOC COST TOO MUCH TIME, cost_time=1999915, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1974046, WASH_BLOCK_END=25868], seq:[0, 1, 2] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=1993293, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1967484, WASH_BLOCK_END=25808], seq:[0, 1, 2] [386]OB_MALLOC COST TOO MUCH TIME, cost_time=1988926, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1963134, WASH_BLOCK_END=25792], seq:[0, 1, 2] [729]OB_MALLOC COST TOO MUCH TIME, cost_time=1915400, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=920370], seq:[0, 1] [2025-08-14 09:03:19.735278] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=20941] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:19.752390] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=17084] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:19.752425] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=34] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:19.752457] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=31] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:19.752467] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:19.752478] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:19.752486] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:16.778990] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:19.752586] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2969081][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:19.752615] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:19.752639] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:19.753749] WDIAG stat_cleanup (debug.c:35) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=964990][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:19.755593] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] get_max_applied_scn(scn={val:1755161719775179002, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:45, palf_committed_end_lsn_:{lsn:8461098908}, palf_committed_end_scn_:{val:1755161719775179002, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161719775179002, v:0}}) [2025-08-14 09:03:19.755619] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] get_min_unreplayed_log_info(lsn={lsn:8461098908}, scn={val:1755100947600130001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161920778323, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:23084275, ret_code_:0}}, next_to_submit_lsn_:{lsn:8461098908}, next_to_submit_scn_:{val:1755100947600130001, v:0}, base_lsn_:{lsn:7515734016}, base_scn_:{val:1753840867171901045, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6e05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:8461098908}, end_lsn:{lsn:8461098908}, read_buf:{buf_len_:3747840, buf_:0x7f98f6e05000}, block_size:67104768, log_storage_:0x7f99297a3bf0, log_storage_:{log_tail:{lsn:8461098908}, readable_log_tail:{lsn:8461098908}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8455200768}, min_scn:{val:1755160384851871002, v:0}, curr_block_id:126, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:215, min_block_id:93, max_block_id:127, curr_writable_block_id:126}, logical_block_size_:67104768, curr_block_writable_size_:61206628, block_header_serialize_buf_:0x7f99297a48b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:198, read_io_size:699822080, read_disk_cost_ts:4178035, start_lsn:{lsn:7515734016}}}}}}) [705]OB_MALLOC COST TOO MUCH TIME, cost_time=2012727, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1982364, WASH_BLOCK_END=30363], seq:[0, 1, 2] [357]OB_MALLOC COST TOO MUCH TIME, cost_time=2012701, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1982661, WASH_BLOCK_END=30040], seq:[0, 1, 2] [389]OB_MALLOC COST TOO MUCH TIME, cost_time=2009908, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1979888, WASH_BLOCK_END=30020], seq:[0, 1, 2] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=2002393, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1972480, WASH_BLOCK_END=29913], seq:[0, 1, 2] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=1009374, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=973223, WASH_BLOCK_END=36150], seq:[0, 1, 2] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=2016886, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1974177, WASH_BLOCK_END=42708], seq:[0, 1, 2] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=2013846, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1971234, WASH_BLOCK_END=42612], seq:[0, 1, 2] [303]OB_MALLOC COST TOO MUCH TIME, cost_time=2003609, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1961187, WASH_BLOCK_END=42421], seq:[0, 1, 2] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=1990398, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1948076, WASH_BLOCK_END=42322], seq:[0, 1, 2] [376]OB_MALLOC COST TOO MUCH TIME, cost_time=975462, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=939499, WASH_BLOCK_END=35963], seq:[0, 1, 2] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=993054, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=956827, WASH_BLOCK_END=36227], seq:[0, 1, 2] [833]OB_MALLOC COST TOO MUCH TIME, cost_time=926296, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=890371, WASH_BLOCK_END=35925], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=1001206, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=964896, WASH_BLOCK_END=36310], seq:[0, 1, 2] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=994411, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=958095, WASH_BLOCK_END=36316], seq:[0, 1, 2] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=992057, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=955880, WASH_BLOCK_END=36177], seq:[0, 1, 2] [553]OB_MALLOC COST TOO MUCH TIME, cost_time=941485, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=905454, WASH_BLOCK_END=36031], seq:[0, 1, 2] [2025-08-14 09:03:19.723111] EDIAG [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=5851369][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1016038, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1016038, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1b52e1b7 0x996d441 0x996cfd8 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [857]OB_MALLOC COST TOO MUCH TIME, cost_time=973648, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=973640], seq:[0] [822]OB_MALLOC COST TOO MUCH TIME, cost_time=989475, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=989473], seq:[0] [857]OB_MALLOC COST TOO MUCH TIME, cost_time=1005091, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=973714], seq:[0, 1] [2025-08-14 09:03:19.730395] EDIAG [STORAGE.TRANS] amend (ob_tenant_weak_read_server_version_mgr.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1041770, owner: ObLog, click_count: 5, time dist:[FORMAT_END=36658, ObMalloc#start=1, ObMalloc#end=1005109, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x1c0234c0 0x97a449c 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:19.756545] EDIAG [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=965573][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1016033, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1016032, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x1d0f5b0a 0x1d0f57c6 0x1d17ad7c 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [729]OB_MALLOC COST TOO MUCH TIME, cost_time=908284, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [571]OB_MALLOC COST TOO MUCH TIME, cost_time=1959705, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1959698], seq:[0] [998]OB_MALLOC COST TOO MUCH TIME, cost_time=943710, tenant_id=500, label=SeArray, ctx_id=0, prio=0, size=7936, [2025-08-14 09:03:19.743208] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7938] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:19.759290] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=16057] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:19.759298] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:19.759304] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:19.759309] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:19.759315] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:19.759320] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:19.764374] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=61] get wrs ts(ls_id={id:1}, delta=479989185, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:19.764406] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=28] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:19.764436] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=479989254, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=252355, succ_cluster_heartbeat_count=1905, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161003593738002, v:0}, local_cluster_delta=1196170695, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:03:19.764474] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=32] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119797572, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755162138745740) [2025-08-14 09:03:19.765152] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=479989185, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:19.765166] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:19.846259] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:20.727635] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:19.771529] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=103] get wrs ts(ls_id={id:1}, delta=652030113, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:20.731697] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=958703] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:20.733166] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=21] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:03:20.735740] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=8] gc stale ls task succ [2025-08-14 09:03:20.740555] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:19.723111] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=5851369] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:03:19.772496] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=13] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:03:19.772613] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:03:20.740762] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1016038] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:03:20.763674] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22874][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:19.730395] INFO [STORAGE.TRANS] amend (ob_tenant_weak_read_server_version_mgr.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [WRS] tenant has no valid partition on current server, generate weak read snapshot version based on current timestamp(tenant_id=1002, epoch_tstamp_=1755162194757031, total_part_count_=2, valid_inner_part_count_=0, valid_user_part_count_=0, version_={val:1755161758221798000, v:0}, base_version_when_no_valid_partition={val:1755161758221798000, v:0}) [2025-08-14 09:03:20.773773] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=1041770] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=442551972, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=251759, succ_cluster_heartbeat_count=1502, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161006924036004, v:0}, local_cluster_delta=1193849734, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:03:19.756545] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=965573] temporary file module use no blocks [2025-08-14 09:03:20.774760] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=480998658, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [822]OB_MALLOC COST TOO MUCH TIME, cost_time=4005919, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1982653, WASH_BLOCK_END=42825, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=989490], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:18.795326] EDIAG [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2955975, owner: ObLog, click_count: 5, time dist:[FORMAT_END=16, ObMalloc#start=1, ObMalloc#end=975469, ALLOC_END=1980489, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1fce4e91 0x1fce455d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:17.755315] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4011491, owner: ObLog, click_count: 5, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=2013856, ALLOC_END=1997624, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:18.761241] EDIAG do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3007166, owner: ObLog, click_count: 5, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=1009391, ALLOC_END=1997762, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x202c530b 0x202c6888 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:17.778834] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3988141, owner: ObLog, click_count: 5, time dist:[FORMAT_END=20, ObMalloc#start=1, ObMalloc#end=1990406, ALLOC_END=1997713, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:17.768682] EDIAG [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4004817, owner: ObLog, click_count: 3, time dist:[FORMAT_END=4004816, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1bdd73e2 0x1bdd6e36 0x986bc6f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:17.763500] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4010075, owner: ObLog, click_count: 5, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=1988935, ALLOC_END=2021129, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:17.752399] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4021239, owner: ObLog, click_count: 5, time dist:[FORMAT_END=36, ObMalloc#start=1, ObMalloc#end=1999941, ALLOC_END=2021260, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:17.759100] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [776][T1002_StandbySc][T1002][YB42AC4D0007-00063C495DFA033D-0-0] [lt=22][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4014772, owner: ObLog, click_count: 5, time dist:[FORMAT_END=15, ObMalloc#start=1, ObMalloc#end=1993306, ALLOC_END=2021450, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [770]OB_MALLOC COST TOO MUCH TIME, cost_time=2006275, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=2006273], seq:[0] [2025-08-14 09:03:20.733401] EDIAG [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1046523, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1046522, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1c02313f 0x97a45d8 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [857]OB_MALLOC COST TOO MUCH TIME, cost_time=1950358, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1], seq:[0, 1] [571]OB_MALLOC COST TOO MUCH TIME, cost_time=2969682, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1959731], seq:[0, 1] [2025-08-14 09:03:17.842040] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [571][T1_ArcSender][T1][YB42AC4D0007-00063C495BA99C16-0-0] [lt=910779][errcode=-4018] LOGGER COST TOO MUCH TIME, cost: 4878792, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1909087, ObMalloc#start=1, ObMalloc#end=2969698, ALLOC_END=5, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3b4b 0x113d34aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:17.743708] EDIAG [STORAGE.TRANS] refresh_elr_tenant_config_ (ob_tx_elr_util.cpp:42) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4977808, owner: ObLog, click_count: 3, time dist:[FORMAT_END=4977807, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b878a47 0x1b8787a8 0x1b878197 0x986bac9 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:19.744963] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2980646, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=1, FORMAT_END=2980641, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:17.760443] EDIAG [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=37891][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4014306, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1950933, ObMalloc#start=1, ObMalloc#end=57767, ALLOC_END=2005604, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x99721a1 0x997a72e 0x2006ed17 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [770]OB_MALLOC COST TOO MUCH TIME, cost_time=3963597, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=973585, WASH_BLOCK_END=36195, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=2006289], seq:[0, 1, 2, 3, 4] [729]OB_MALLOC COST TOO MUCH TIME, cost_time=970478, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=2958754, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=2958745], seq:[0] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=2970511, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=2970503], seq:[0] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=3982302, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=975416, WASH_BLOCK_END=36356, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2970528], seq:[0, 1, 2, 3, 4] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=2939612, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=2939602, ALLOC_CHUNK_END=6], seq:[0, 1] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=2912864, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=2912855, ALLOC_CHUNK_END=5], seq:[0, 1] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=2897541, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=2897531, ALLOC_CHUNK_END=5], seq:[0, 1] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=2024976, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=2024967, ALLOC_CHUNK_END=4], seq:[0, 1] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=2011456, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=2011446, ALLOC_CHUNK_END=4], seq:[0, 1] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=2016944, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=2016932, ALLOC_CHUNK_END=4], seq:[0, 1] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=2009013, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=2009003, ALLOC_CHUNK_END=5], seq:[0, 1] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=2009717, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=2009705, ALLOC_CHUNK_END=4], seq:[0, 1] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=2009710, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=2009697, ALLOC_CHUNK_END=6], seq:[0, 1] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=2009064, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=2009054, ALLOC_CHUNK_END=4], seq:[0, 1] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=2009097, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=2009087, ALLOC_CHUNK_END=4], seq:[0, 1] [383]OB_MALLOC COST TOO MUCH TIME, cost_time=2008506, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=2008494, ALLOC_CHUNK_END=5], seq:[0, 1] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=1995585, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1995563, ALLOC_CHUNK_END=5], seq:[0, 1] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=1991254, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1991243, ALLOC_CHUNK_END=5], seq:[0, 1] [723]OB_MALLOC COST TOO MUCH TIME, cost_time=1974186, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=18, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=1974153, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [528]OB_MALLOC COST TOO MUCH TIME, cost_time=1973081, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1973052, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=1908367, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1908346, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [400]OB_MALLOC COST TOO MUCH TIME, cost_time=1712675, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1712656, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [552]OB_MALLOC COST TOO MUCH TIME, cost_time=1031167, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=1031144, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [717]OB_MALLOC COST TOO MUCH TIME, cost_time=1031197, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=1031171, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [2025-08-14 09:03:20.774786] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:20.774825] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=34] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119797572, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755162138745740) [2025-08-14 09:03:20.775700] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=480999717, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:21.727680] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:21.733897] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:03:17.743643] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=34] LogLoopThread round_cost_time(us)(round_cost_time=19893940) [2025-08-14 09:03:18.795326] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=15] blacklist_loop exec finished(cost_time=16, is_enabled=true, send_cnt=0) [2025-08-14 09:03:20.763737] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=966730] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:03:20.775715] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=1001088, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1001060], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:17.755315] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:21.772910] WDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4011491][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:21.772930] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=1001100, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1001099], seq:[0, 1] [2025-08-14 09:03:21.772942] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:18.761241] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=12] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:17.778834] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:21.746309] EDIAG [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1001132, owner: ObLog, click_count: 5, time dist:[FORMAT_END=16, ObMalloc#start=1, ObMalloc#end=1001113, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1f832f91 0x97c36fe 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:21.773029] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=3988141][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:21.773056] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=-4389] timer task use too much time(end=1755162201773055, start=1755162197778832, delta=3994223) [2025-08-14 09:03:20.763705] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=966818] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:03:17.768682] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] finish update min start scn(max_decided_scn={val:1755161547738516001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1755161493768447002, v:0}, keep_alive_scn_:{val:1755161493768447002, v:0}, update_ts_:1755162195753334}) [2025-08-14 09:03:17.763500] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:21.779673] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4010075][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:21.779692] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:21.779706] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:17.752399] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:21.779743] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=4021239][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:21.779762] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4389] timer task use too much time(end=1755162201779761, start=1755162197752397, delta=4027364) [2025-08-14 09:03:21.737908] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=9] gc stale ls task succ [2025-08-14 09:03:17.760443] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=37891] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:21.780889] WDIAG pktc_handle_req_queue (pktc_post.h:138) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:21.748866] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:21.781395] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32477][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:20.733401] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162199770954}, version_delta=652994884) [2025-08-14 09:03:17.842040] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [571][T1_ArcSender][T1][YB42AC4D0007-00063C495BA99C16-0-0] [lt=910779][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:03:22.728456] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:17.743708] INFO [STORAGE.TRANS] refresh_elr_tenant_config_ (ob_tx_elr_util.cpp:42) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=7] refresh tenant config success(tenant_id=1001, *this={last_refresh_ts:1755162195753334, can_tenant_elr:true}) [2025-08-14 09:03:19.744963] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:22.730147] WDIAG ev_invoke_pending (ev.c:2136) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2980646][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:17.849308] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:21.746309] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=21] current monitor number(seq_=-1) [772]OB_MALLOC COST TOO MUCH TIME, cost_time=991829, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=991798, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [833]OB_MALLOC COST TOO MUCH TIME, cost_time=991836, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=991813], seq:[0, 1, 2, 3, 4] [833]OB_MALLOC COST TOO MUCH TIME, cost_time=991847, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=991846], seq:[0, 1] [2025-08-14 09:03:18.844616] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=54] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:03:18.844616] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=54][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3898568, owner: ObLog, click_count: 7, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=926303, ObMalloc#start=1980392, ObMalloc#end=991854, ALLOC_END=1, APPEND_END=6], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [553]OB_MALLOC COST TOO MUCH TIME, cost_time=991872, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=991848], seq:[0, 1, 2, 3, 4] [553]OB_MALLOC COST TOO MUCH TIME, cost_time=991880, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=991880], seq:[0, 1] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=991805, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=991781], seq:[0, 1, 2, 3, 4] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=991815, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=991814], seq:[0, 1] [2025-08-14 09:03:18.829470] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3913748, owner: ObLog, click_count: 7, time dist:[FORMAT_END=14, ObMalloc#start=1, ObMalloc#end=941493, ObMalloc#start=1980348, ObMalloc#end=991887, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:18.829470] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=15] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:03:18.776424] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:18.776424] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3966727, owner: ObLog, click_count: 7, time dist:[FORMAT_END=28, ObMalloc#start=0, ObMalloc#end=994418, ObMalloc#start=1980456, ObMalloc#end=991821, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:22.749173] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=3966727][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:22.749215] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4389] timer task use too much time(end=1755162202749214, start=1755162198776422, delta=3972792) [382]OB_MALLOC COST TOO MUCH TIME, cost_time=975275, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=975246], seq:[0, 1, 2, 3, 4] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=975253, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=975230], seq:[0, 1, 2, 3, 4] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=975296, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=975295], seq:[0, 1] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=975271, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=975270], seq:[0, 1] [2025-08-14 09:03:18.778815] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3964785, owner: ObLog, click_count: 7, time dist:[FORMAT_END=9, ObMalloc#start=0, ObMalloc#end=992063, ObMalloc#start=1997407, ObMalloc#end=975302, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:17.752255] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4989802, owner: ObLog, click_count: 7, time dist:[FORMAT_END=8, ObMalloc#start=1, ObMalloc#end=2016895, ObMalloc#start=1997617, ObMalloc#end=975277, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [403]OB_MALLOC COST TOO MUCH TIME, cost_time=975222, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=975200], seq:[0, 1, 2, 3, 4] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=975232, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=975231], seq:[0, 1] [2025-08-14 09:03:18.777741] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3965958, owner: ObLog, click_count: 7, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=993062, ObMalloc#start=1997643, ObMalloc#end=975238, ALLOC_END=0, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [303]OB_MALLOC COST TOO MUCH TIME, cost_time=975149, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=975125], seq:[0, 1, 2, 3, 4] [303]OB_MALLOC COST TOO MUCH TIME, cost_time=975158, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=975157], seq:[0, 1] [2025-08-14 09:03:18.778815] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:17.752255] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:22.749620] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3964785][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:22.749623] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4989802][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:22.749640] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:22.749640] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:22.749656] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:22.749656] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:18.777741] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:22.749715] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=3965958][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:22.749731] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:22.749745] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:17.765594] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] update_check_range(min_check_version=2540, max_check_version=2540, global_version=2645) [497]OB_MALLOC COST TOO MUCH TIME, cost_time=974616, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=974589], seq:[0, 1, 2, 3, 4] [497]OB_MALLOC COST TOO MUCH TIME, cost_time=974628, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=974626], seq:[0, 1] [357]OB_MALLOC COST TOO MUCH TIME, cost_time=968903, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=968875], seq:[0, 1, 2, 3, 4] [357]OB_MALLOC COST TOO MUCH TIME, cost_time=968912, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=968912], seq:[0, 1] [705]OB_MALLOC COST TOO MUCH TIME, cost_time=968679, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=968654], seq:[0, 1, 2, 3, 4] [705]OB_MALLOC COST TOO MUCH TIME, cost_time=968686, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=968686], seq:[0, 1] [2025-08-14 09:03:17.743624] EDIAG [PALF] log_loop_ (log_loop_thread.cpp:154) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4998837, owner: ObLog, click_count: 7, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=2012746, ObMalloc#start=2017384, ObMalloc#end=968693, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xfabb534 0x993e263 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [389]OB_MALLOC COST TOO MUCH TIME, cost_time=968697, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=968672], seq:[0, 1, 2, 3, 4] [389]OB_MALLOC COST TOO MUCH TIME, cost_time=968705, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=968705], seq:[0, 1] [2025-08-14 09:03:17.746490] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4996022, owner: ObLog, click_count: 7, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=2009916, ObMalloc#start=2017380, ObMalloc#end=968711, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [730]OB_MALLOC COST TOO MUCH TIME, cost_time=968760, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=968735], seq:[0, 1, 2, 3, 4] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=968769, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=968768], seq:[0, 1] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=973221, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=973196], seq:[0, 1, 2, 3, 4] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=973437, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=973436], seq:[0, 1] [2025-08-14 09:03:21.713325] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:15.755161] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=1009275] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:03:17.743624] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=29] LogLoopThread round_cost_time(us)(round_cost_time=73973284) [2025-08-14 09:03:17.746490] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:22.750090] WDIAG ev_invoke_pending (ev.c:2136) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4996022][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:22.750107] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:22.750122] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:17.754030] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=21] gc stale ls task succ [381]OB_MALLOC COST TOO MUCH TIME, cost_time=968137, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=968107], seq:[0, 1, 2, 3, 4] [381]OB_MALLOC COST TOO MUCH TIME, cost_time=968152, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=968150], seq:[0, 1] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=700484, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=700435], seq:[0, 1, 2, 3, 4, 5, 6] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=700508, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=700507], seq:[0, 1] [721]OB_MALLOC COST TOO MUCH TIME, cost_time=321912, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=321876], seq:[0, 1, 2, 3, 4, 5, 6] [721]OB_MALLOC COST TOO MUCH TIME, cost_time=321932, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=321931], seq:[0, 1] [2025-08-14 09:03:19.759112] INFO [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:171) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=11] handle gts request statistics(total_rt=1959446, total_cnt=14, avg_rt=1.306297333333333372e+05, avg_cnt=1.399999999999999911e+00) [2025-08-14 09:03:22.049081] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=766126][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:22.428256] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [721][T1001_OBJLockGC][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:22.750728] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [721][T1001_OBJLockGC][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162202750728, start=1755162202428253, delta=322475) [2025-08-14 09:03:22.727215] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=41] get_max_applied_scn(scn={val:1755161749741193000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:50, palf_committed_end_lsn_:{lsn:1604090090}, palf_committed_end_scn_:{val:1755161749741193000, v:0}, last_check_scn_:{val:1755161749741193001, v:0}, max_applied_cb_scn_:{val:1755161749741193000, v:0}}) [2025-08-14 09:03:22.750779] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23524] get_min_unreplayed_log_info(lsn={lsn:1604088821}, scn={val:1755100948296325001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873479, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:12492179, ret_code_:0}}, next_to_submit_lsn_:{lsn:1604088821}, next_to_submit_scn_:{val:1755100948296325001, v:0}, base_lsn_:{lsn:1409200128}, base_scn_:{val:1753840802540914000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:1604088821}, end_lsn:{lsn:1604088821}, read_buf:{buf_len_:3747840, buf_:0x7f98f6a05000}, block_size:67104768, log_storage_:0x7f990bfebbf0, log_storage_:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:42, read_io_size:147709952, read_disk_cost_ts:1061450, start_lsn:{lsn:1409200128}}}}}}) [2025-08-14 09:03:22.750892] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=110] get wrs ts(ls_id={id:1}, delta=451987393, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [567]OB_MALLOC COST TOO MUCH TIME, cost_time=969965, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [2025-08-14 09:03:17.775131] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:22.730205] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4014306] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:22.753768] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=23525] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:22.753785] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=17] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:22.753805] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=19] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:22.730165] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:22.753843] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:17.849308] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4890672, owner: ObLog, click_count: 7, time dist:[FORMAT_END=908356, ObMalloc#start=0, ObMalloc#end=3982313, ALLOC_END=2, APPEND_END=0, ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=2], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [736]OB_MALLOC COST TOO MUCH TIME, cost_time=2987395, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2009735, WASH_BLOCK_END=977660], seq:[0, 1, 2] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=2986787, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2009076, WASH_BLOCK_END=977711], seq:[0, 1, 2] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=2987492, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2009725, WASH_BLOCK_END=977767], seq:[0, 1, 2] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=2973375, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1995593, WASH_BLOCK_END=977782], seq:[0, 1, 2] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=2969042, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1991262, WASH_BLOCK_END=977780], seq:[0, 1, 2] [552]OB_MALLOC COST TOO MUCH TIME, cost_time=2009195, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1031177, WASH_BLOCK_END=978018], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=1969756, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=991840, WASH_BLOCK_END=977916], seq:[0, 1, 2] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=2988301, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2009021, WASH_BLOCK_END=979246, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=33], seq:[0, 1, 2, 3, 4] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=3882649, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2897549, WASH_BLOCK_END=985100], seq:[0, 1, 2] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=2993363, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2009105, WASH_BLOCK_END=984258], seq:[0, 1, 2] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=3010469, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2024983, WASH_BLOCK_END=985486], seq:[0, 1, 2] [383]OB_MALLOC COST TOO MUCH TIME, cost_time=2993063, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=2008514, WASH_BLOCK_END=984548], seq:[0, 1, 2] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=2997053, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2011466, WASH_BLOCK_END=985587], seq:[0, 1, 2] [723]OB_MALLOC COST TOO MUCH TIME, cost_time=2958901, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1974196, WASH_BLOCK_END=984705], seq:[0, 1, 2] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=3002820, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2016990, WASH_BLOCK_END=985830], seq:[0, 1, 2] [528]OB_MALLOC COST TOO MUCH TIME, cost_time=2957925, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1973091, WASH_BLOCK_END=984833], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=2893236, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1908376, WASH_BLOCK_END=984860], seq:[0, 1, 2] [2025-08-14 09:03:20.837456] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=2][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2893306, owner: ObLog, click_count: 5, time dist:[FORMAT_END=41, ObMalloc#start=1, ObMalloc#end=2893244, ALLOC_END=20, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [400]OB_MALLOC COST TOO MUCH TIME, cost_time=2697620, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1712684, WASH_BLOCK_END=984936], seq:[0, 1, 2] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=4962461, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=956036, WASH_BLOCK_END=36200, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2958787], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:18.778548] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [663][T1001_StandbySc][T1001][YB42AC4D0007-00063C495C7A0334-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4962515, owner: ObLog, click_count: 5, time dist:[FORMAT_END=12, ObMalloc#start=0, ObMalloc#end=4962496, ALLOC_END=6, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:22.731867] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=979325] get wrs ts(ls_id={id:1}, delta=481953407, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:22.755338] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23432] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:22.757004] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=481953407, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:22.757018] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:22.773499] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:22.781311] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:22.851129] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:22.860259] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:22.734468] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:03:22.738302] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=42339] gc stale ls task succ [2025-08-14 09:03:22.742124] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:22.753585] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1046523] get wrs ts(ls_id={id:1}, delta=653990070, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:23.726311] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=971249] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:23.727668] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:20.733122] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2997100, owner: ObLog, click_count: 11, time dist:[FORMAT_END=18, ObMalloc#start=1, ObMalloc#end=2997061, ALLOC_END=20, APPEND_END=0, ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=112, ALLOC_BLOCK#start=11468, ALLOC_BLOCK#end=96, ALLOC_BLOCK#start=12828, ALLOC_BLOCK#end=35], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [998]OB_MALLOC COST TOO MUCH TIME, cost_time=1020987, tenant_id=1002, label=SqlExecutor, ctx_id=0, prio=0, size=245760, [994]OB_MALLOC COST TOO MUCH TIME, cost_time=1013783, tenant_id=1002, label=SqlExecutor, ctx_id=0, prio=0, size=245760, [2025-08-14 09:03:20.736373] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=1001633][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3989300, owner: ObLog, click_count: 5, time dist:[FORMAT_END=19, ObMalloc#start=0, ObMalloc#end=2986797, ALLOC_END=1002483, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:20.750011] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3975743, owner: ObLog, click_count: 5, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=2973385, ALLOC_END=1002347, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:21.033489] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=1778262][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2697709, owner: ObLog, click_count: 5, time dist:[FORMAT_END=17, ObMalloc#start=1, ObMalloc#end=2697628, ALLOC_END=62, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [717]OB_MALLOC COST TOO MUCH TIME, cost_time=3014770, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1031210, WASH_BLOCK_END=1983560], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=4897943, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2912875, WASH_BLOCK_END=1985068], seq:[0, 1, 2] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=1008918, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1008916], seq:[0] [2025-08-14 09:03:15.755161] EDIAG [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=1009275][errcode=0] LOGGER COST TOO MUCH TIME, cost: 6984269, owner: ObLog, click_count: 7, time dist:[FORMAT_END=1985522, ObMalloc#start=1, ObMalloc#end=2012710, ObMalloc#start=2017114, ObMalloc#end=968918, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x18bdb084 0x18bda6b1 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [729]OB_MALLOC COST TOO MUCH TIME, cost_time=1890950, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:03:20.735715] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [477][T1_StandbySchem][T1][YB42AC4D0007-00063C495B6A034B-0-0] [lt=12][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4010549, owner: ObLog, click_count: 6, time dist:[FORMAT_END=11, ObMalloc#start=0, ObMalloc#end=2987499, ObMalloc#start=22, ObMalloc#end=112, FREE_END=1022905], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:20.735641] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4010667, owner: ObLog, click_count: 6, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=2988309, ObMalloc#start=20, ObMalloc#end=35, FREE_END=1022291], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:20.754392] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3992110, owner: ObLog, click_count: 6, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=2969302, ObMalloc#start=23, ObMalloc#end=92, FREE_END=1022682], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:18.769599] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5974204, owner: ObLog, click_count: 8, time dist:[FORMAT_END=31, ObMalloc#start=1, ObMalloc#end=1001213, ObMalloc#start=1980334, ObMalloc#end=1969764, ObMalloc#start=111, ObMalloc#end=35, FREE_END=1022715], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:23.747344] EDIAG [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=992022][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1005525, owner: ObLog, click_count: 3, time dist:[FORMAT_END=985601, ALLOC_END=19923, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x99721a1 0x997a72e 0x2006ed17 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:22.755317] EDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=990764][errcode=-4015] LOGGER COST TOO MUCH TIME, cost: 1997915, owner: ObLog, click_count: 8, time dist:[FORMAT_END=1976143, ObMalloc#start=1, ObMalloc#end=102, ObMalloc#start=25, ObMalloc#end=38, ObMalloc#start=1462, ObMalloc#end=56, FREE_END=20088], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x279e59be 0x971b1c1 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:20.736410] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4014070, owner: ObLog, click_count: 6, time dist:[FORMAT_END=7, ObMalloc#start=1, ObMalloc#end=2993373, ObMalloc#start=21, ObMalloc#end=55, FREE_END=1020613], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:20.737042] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4013601, owner: ObLog, click_count: 6, time dist:[FORMAT_END=8, ObMalloc#start=0, ObMalloc#end=2993081, ObMalloc#start=20, ObMalloc#end=35, FREE_END=1020456], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:20.719516] EDIAG [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4031221, owner: ObLog, click_count: 6, time dist:[FORMAT_END=16, ObMalloc#start=2, ObMalloc#end=3010479, ObMalloc#start=21, ObMalloc#end=40, FREE_END=1020663], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a1f867 0x27a135c3 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:23.753469] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1002303, owner: ObLog, click_count: 6, time dist:[FORMAT_END=6, ObMalloc#start=0, ObMalloc#end=47, ObMalloc#start=0, ObMalloc#end=35, FREE_END=1002215], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:19.756556] EDIAG [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=980448][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4989014, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1956578, ObMalloc#start=2, ObMalloc#end=2009206, ObMalloc#start=21, ObMalloc#end=48, FREE_END=1023159], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x1d0f5b0a 0x1d0f57c6 0x1d17ad7c 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [567]OB_MALLOC COST TOO MUCH TIME, cost_time=1033366, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [2025-08-14 09:03:19.830213] EDIAG try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=1652018][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4918331, owner: ObLog, click_count: 6, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=4897954, ObMalloc#start=19, ObMalloc#end=51, FREE_END=20296], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x26499505 0x97074a3 0x26498fa9 0x7f9973e081cf 0x7f9973a39dd3 [658]OB_MALLOC COST TOO MUCH TIME, cost_time=1046860, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1008930], seq:[0, 1] [2025-08-14 09:03:23.725534] EDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=4890672][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1046912, owner: ObLog, click_count: 5, time dist:[FORMAT_END=27, ObMalloc#start=2, ObMalloc#end=1046878, ALLOC_END=4, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ed35c 0x97aead0 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:19.758017] EDIAG [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=1000036][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4990427, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1955146, ObMalloc#start=0, ObMalloc#end=3014787, ObMalloc#start=22, ObMalloc#end=69, FREE_END=20403], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x1d0f5b0a 0x1d0f57c6 0x1d17ad7c 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:23.753313] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1001704, owner: ObLog, click_count: 6, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=65, ObMalloc#start=2, ObMalloc#end=39, FREE_END=1001588], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:19.845597] EDIAG eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4903532, owner: ObLog, click_count: 6, time dist:[FORMAT_END=15, ObMalloc#start=1, ObMalloc#end=3882668, ObMalloc#start=38, ObMalloc#end=103, FREE_END=1020707], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x27a8a8c8 0x97b7e1a 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:19.979724] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=363722][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4789347, owner: ObLog, click_count: 3, time dist:[FORMAT_END=4789346, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:17.743643] EDIAG [PALF] log_loop_ (log_loop_thread.cpp:154) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4005949, owner: ObLog, click_count: 5, time dist:[FORMAT_END=6, ObMalloc#start=0, ObMalloc#end=4005941, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xfabb534 0x993e263 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:19.734835] EDIAG [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=14847][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3994223, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1036263, ObMalloc#start=1, ObMalloc#end=2957933, ALLOC_END=26, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x12377491 0x1236e8e9 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [568]OB_MALLOC COST TOO MUCH TIME, cost_time=1986188, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1986179], seq:[0] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=5902557, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2939620, WASH_BLOCK_END=976704, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1986227], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:19.803336] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5902585, owner: ObLog, click_count: 5, time dist:[FORMAT_END=15, ObMalloc#start=1, ObMalloc#end=5902566, ALLOC_END=2, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:23.734527] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:20.733122] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:19.734835] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=14847] finish to resize log external storage handler(current_ts=1755162197778500, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-08-14 09:03:20.837456] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:23.735159] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162203735159, start=1755162200837454, delta=2897705) [2025-08-14 09:03:21.033489] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=1778262][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:22.750916] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:22.753816] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:23.748624] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:23.748663] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=985002] gc stale ls task succ [2025-08-14 09:03:23.750791] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:23.753446] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:23.759374] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2997100][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:23.759397] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:23.759411] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:23.775118] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:23.781204] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:24.723595] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=986835] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:03:20.735678] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=1012851] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:20.736373] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=1001633] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:20.750011] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:24.731827] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3975743][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:24.731848] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:24.731869] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:24.732076] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:19.744812] INFO dump_info (ob_timer_service.cpp:728) [694][T1001_TimerSvr][T1001][Y0-0000000000000000-0-0] [lt=22] dump info [summary](this=0x7f99356b0af0, this={this:0x7f99356b0af0, tenant_id_:1001, priority_task_queue_.size():22, running_task_set_.size():24, uncanceled_task_set_.size():0, worker_thread_pool_.get_queue_num():12}) [2025-08-14 09:03:24.750757] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:24.751256] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:24.750132] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=9] gc stale ls task succ [2025-08-14 09:03:24.729827] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:24.752291] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22428][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:24.752562] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3992110][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:24.752745] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=481953407, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:24.752767] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:24.752764] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:24.752824] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=53] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:24.753381] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=481953407, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:24.753391] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:23.747344] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=992022] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:23.809329] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [741][T1001_MultiVers][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:24.752866] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=5974204][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:24.754419] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1005525] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:18.729834] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 6978801, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:18.740701] EDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=960701][errcode=0] LOGGER COST TOO MUCH TIME, cost: 6968031, , BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:17.754030] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=21][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4988586, owner: ObLog, click_count: 7, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=2002402, ObMalloc#start=2017392, ObMalloc#end=968775, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:23.050224] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [402][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=319788][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2665168, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2665167, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:19.742800] EDIAG LIBalloc (ob_malloc_allocator.cpp:924) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=2029300][errcode=-4013] LOGGER COST TOO MUCH TIME, cost: 5007140, owner: ObLog, click_count: 3, time dist:[FORMAT_END=4980102, ALLOC_END=27038, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0xf9b6d19 0xf9b6367 0xf9b6058 0xf989dfb 0x973c6f6 0x973c18d 0x7f9973af5dfa 0x7f9973af5efd 0x260b8ade 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:20.797265] EDIAG [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4934823, owner: ObLog, click_count: 3, time dist:[FORMAT_END=4914619, ALLOC_END=20204, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a5bbf2 0x94da542 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:24.756324] EDIAG [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=4977808][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1002867, owner: ObLog, click_count: 3, time dist:[FORMAT_END=985646, ALLOC_END=17220, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1bdd73e2 0x1bdd6e36 0x986bc6f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:18.760899] EDIAG [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 6989263, owner: ObLog, click_count: 3, time dist:[FORMAT_END=6989262, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x1f99e657 0x1f981d5d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:24.743960] EDIAG [RS.COMPACTION] try_idle (ob_major_merge_scheduler.cpp:226) [1275][T1001_MergeSche][T1001][YB42AC4D0007-00063C4963E99CBE-0-0] [lt=41918][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1021913, , BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1125d502 0x1125cf81 0x1125a4c3 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:17.735361] EDIAG [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [723][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 7010826, owner: ObLog, click_count: 6, time dist:[FORMAT_END=3031568, ObMalloc#start=1, ObMalloc#end=2958910, ObMalloc#start=19, ObMalloc#end=46, FREE_END=1020282], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x973b640 0x973b132 0x973a32d 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [659]OB_MALLOC COST TOO MUCH TIME, cost_time=992744, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=44, WASH_BLOCK_END=935335, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=57359], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:24.744020] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1032721, owner: ObLog, click_count: 11, time dist:[ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=146, ALLOC_BLOCK#start=10117, ALLOC_BLOCK#end=95, ALLOC_BLOCK#start=14132, ALLOC_BLOCK#end=68, ALLOC_BLOCK#start=10777, ALLOC_BLOCK#end=46, FORMAT_END=997333, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:24.768517] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=3898568][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1008291, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=75, FORMAT_END=1008210, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:18.759940] EDIAG [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:233) [529][T1_RFLWorker][T1][YB42AC4D0007-00063C495B199C16-0-0] [lt=712290][errcode=0] LOGGER COST TOO MUCH TIME, cost: 7007957, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=965928, ALLOC_BLOCK#end=2, FORMAT_END=6007807, ObMalloc#start=1, ObMalloc#end=34200, ALLOC_END=19, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x11359281 0x98b739c 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:22.057956] EDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=49][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 3714949, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=17, ALLOC_BLOCK#end=15, FORMAT_END=3714916, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x9757ced 0x97569df 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:24.768079] EDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15188][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 1009979, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=72, ALLOC_BLOCK#start=10069, ALLOC_BLOCK#end=32, FORMAT_END=999800, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1b53d414 0x97ad9aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:21.713325] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1035092, owner: ObLog, click_count: 5, time dist:[FORMAT_END=60440, ObMalloc#start=1, ObMalloc#end=974645, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:20.727602] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4023110, owner: ObLog, click_count: 6, time dist:[FORMAT_END=14, ObMalloc#start=1, ObMalloc#end=3002829, ObMalloc#start=19, ObMalloc#end=47, FREE_END=1020200], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:15.800537] EDIAG [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 9965161, owner: ObLog, click_count: 3, time dist:[FORMAT_END=9965160, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x989c577 0x989bc96 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:18.065797] EDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 7704046, , BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x9757ced 0x97569df 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:24.769275] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1011708, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=6, ALLOC_BLOCK#end=66, FORMAT_END=1011635, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:23.725534] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=4890672][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:24.774056] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1046912][errcode=-4389] timer task use too much time(end=1755162204774056, start=1755162197849306, delta=6924750) [2025-08-14 09:03:24.755966] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [741][T1001_MultiVers][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162204755966, start=1755162203809326, delta=946640) [2025-08-14 09:03:19.979724] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=363722][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:24.760528] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:24.779452] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18894][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:24.781231] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:25.611552] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=51627][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:19.803336] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=14] gc stale ls task succ [2025-08-14 09:03:25.714881] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=5902585] gc stale ls task succ [2025-08-14 09:03:24.774080] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:25.715929] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] timer task use too much time(end=1755162205715928, start=1755162204774078, delta=941850) [2025-08-14 09:03:24.774003] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:25.716847] WDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=941418][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:24.844071] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:25.717008] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162205717007, start=1755162204844069, delta=872938) [2025-08-14 09:03:18.729834] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:25.719223] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6978801][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:25.719251] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:25.719263] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:18.740701] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=960701][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:25.719303] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6968031] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:25.719316] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:25.719398] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=4988586] gc stale ls task succ [2025-08-14 09:03:23.050224] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [402][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=319788][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:25.720158] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=157] [sql nio session](*s={this:0x7f98f72b3030, session_id:3221562808, trace_id:YB42AC4D0007-00063C4965DA1F2E-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:267}, err:0, last_decode_time:1755161748766816, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1890, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:03:25.720177] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=18] [sql nio session](*s={this:0x7f98f72ab030, session_id:3221589435, trace_id:YB42AC4D0007-00063C4965DA1F35-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:246}, err:0, last_decode_time:1755161762721798, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:2456, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:03:25.720187] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=9] [sql nio session](*s={this:0x7f98f72a9030, session_id:3221590528, trace_id:YB42AC4D0007-00063C4965DA1F34-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:250}, err:0, last_decode_time:1755161762715415, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:205, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:03:24.844832] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=1899769][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:03:24.845484] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=1923401][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:03:25.725374] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=35] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:03:25.728659] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:24.757185] INFO [STORAGE] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:123) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=983945] wead read timestamp is too old(timestamp={val:1755161749741193000, v:0}, gts_scn={val:1755161762721798002, v:0}, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1755161749741193000, v:0}}) [2025-08-14 09:03:24.770313] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161101770923, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755161572801635) [2025-08-14 09:03:24.734649] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [578][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=25] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:25.736440] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=976724] get_max_applied_scn(scn={val:1755161744727824001, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:49, palf_committed_end_lsn_:{lsn:7323686009}, palf_committed_end_scn_:{val:1755161747742105001, v:0}, last_check_scn_:{val:1755161749741193000, v:0}, max_applied_cb_scn_:{val:1755161744727824001, v:0}}) [2025-08-14 09:03:20.797265] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=9] [sql nio session](*s={this:0x7f98f733f030, session_id:3221589417, trace_id:YB42AC4D0007-00063C496359B268-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:252}, err:0, last_decode_time:1755161813752329, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1778, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:03:25.739586] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=4934823] [sql nio session](*s={this:0x7f98f733e030, session_id:3221589980, trace_id:YB42AC4D0007-00063C496359B261-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:251}, err:0, last_decode_time:1755161748778070, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1681, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:03:25.739603] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=17] [sql nio session](*s={this:0x7f98f7332030, session_id:3221590132, trace_id:YB42AC4D0007-00063C496359B265-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:266}, err:0, last_decode_time:1755161768783802, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:2125, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:03:24.768319] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=3913748] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:03:25.720195] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=8] [sql nio session](*s={this:0x7f98f72b5030, session_id:3221590961, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:10, sql_initiative_shutdown:false, reader:{fd:278}, err:5, last_decode_time:1755162075745307, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:03:25.739945] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=19719] [sql nio session](*s={this:0x7f98f72a6030, session_id:3221590972, trace_id:YB42AC4D0007-00063C4965DA1F38-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:264}, err:0, last_decode_time:1755162176744872, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:221, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:03:24.775822] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:25.739995] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] timer task use too much time(end=1755162205739994, start=1755162204775819, delta=964175) [2025-08-14 09:03:25.717083] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] [sql nio session](*s={this:0x7f98f72d0030, session_id:3221589415, trace_id:YB42AC4D0007-00063C4965FA43DC-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:245}, err:0, last_decode_time:1755161751714366, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1662, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:03:25.742021] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=24899] [sql nio session](*s={this:0x7f98f72d5030, session_id:3221590200, trace_id:YB42AC4D0007-00063C4965FA43E0-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:248}, err:0, last_decode_time:1755161776777167, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1593, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:03:24.765613] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:25.743742] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] timer task use too much time(end=1755162205743741, start=1755162204765611, delta=978130) [2025-08-14 09:03:25.739491] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=3989300] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:25.739524] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:25.716352] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [564][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=959277] XA scheduler heartbeat task statistics(avg_time=8524) [2025-08-14 09:03:25.735429] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=963642] get_max_applied_scn(scn={val:1755161547738516001, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:50, palf_committed_end_lsn_:{lsn:5282737655}, palf_committed_end_scn_:{val:1755161547738516001, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161547738516001, v:0}}) [2025-08-14 09:03:25.744773] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9319] get_min_unreplayed_log_info(lsn={lsn:5282737655}, scn={val:1755100947825033001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161624848121, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:13063726, ret_code_:0}}, next_to_submit_lsn_:{lsn:5282737655}, next_to_submit_scn_:{val:1755100947825033001, v:0}, base_lsn_:{lsn:4831543296}, base_scn_:{val:1755063445733337001, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7605000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:5282737655}, end_lsn:{lsn:5282737655}, read_buf:{buf_len_:3747840, buf_:0x7f98f7605000}, block_size:67104768, log_storage_:0x7f993e529bf0, log_storage_:{log_tail:{lsn:5282737655}, readable_log_tail:{lsn:5282737655}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:5234171904}, min_scn:{val:1755152244802535004, v:0}, curr_block_id:78, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:206, min_block_id:42, max_block_id:79, curr_writable_block_id:78}, logical_block_size_:67104768, curr_block_writable_size_:18539017, block_header_serialize_buf_:0x7f993e52a8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:85, read_io_size:301367296, read_disk_cost_ts:2364604, start_lsn:{lsn:4831543296}}}}}}) [2025-08-14 09:03:25.736464] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get_min_unreplayed_log_info(lsn={lsn:7323685240}, scn={val:1755100946796590002, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873485, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:8946764, ret_code_:0}}, next_to_submit_lsn_:{lsn:7323685240}, next_to_submit_scn_:{val:1755100946796590002, v:0}, base_lsn_:{lsn:7180210176}, base_scn_:{val:1753840803166611000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:7323685240}, end_lsn:{lsn:7323685240}, read_buf:{buf_len_:3747840, buf_:0x7f98f7a05000}, block_size:67104768, log_storage_:0x7f9901131bf0, log_storage_:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:28, read_io_size:98508800, read_disk_cost_ts:645339, start_lsn:{lsn:7180210176}}}}}}) [2025-08-14 09:03:25.746367] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9292] get wrs ts(ls_id={id:1001}, delta=461017916, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:25.746388] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:25.748262] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=85] get wrs ts(ls_id={id:1}, delta=658009219, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:25.751587] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=245] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:25.755951] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=244] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:25.756468] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:25.756553] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:25.758672] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=179] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:25.759811] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=191] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:25.760182] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:24.756324] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=4977808] finish update min start scn(max_decided_scn={val:1755161719775179002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1755161287778691002, v:0}, keep_alive_scn_:{val:1755161287778691002, v:0}, update_ts_:1755162201728586}) [2025-08-14 09:03:18.760899] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:03:19.722496] WDIAG [SQL] open_plan (ob_result_set.cpp:104) [813][T1002_DupTbLeas][T1001][YB42AC4D0007-00063C496039A468-0-0] [lt=1982870][errcode=-4012] query is timeout(ret=-4012, timeout_ts=1755162037733722, start_time=1755162013720066) [2025-08-14 09:03:25.764555] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:25.732263] WARN [LIB] alloc (ob_malloc_allocator.cpp:924) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=5007140][errcode=-4013] [OOPS]: alloc failed reason is that server memory has reached the upper limit(server_hold: 4293763072, server_limit: 4294967296, alloc_size: 2097152) [2025-08-14 09:03:25.748294] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:25.756499] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [412][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=951385] [sql nio session](*s={this:0x7f98f733a030, session_id:3221590255, trace_id:YB42AC4D0007-00063C49651A3232-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:257}, err:0, last_decode_time:1755161952736711, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1716, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:03:25.774689] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:25.774713] WDIAG [DETECT] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [723][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=7010826][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:17.774777] EDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] LOGGER COST TOO MUCH TIME, cost: 7996648, , BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x13b06ffb 0x958a39e 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:19.759112] EDIAG [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:171) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2987002, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2018837, ObMalloc#start=1, ObMalloc#end=968158, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1b538df0 0x1b4f8d72 0x1b4fbf53 0x1b5014e0 0x1b5009d1 0x1b50387c 0x1b8905de 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:17.775131] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4970975, owner: ObLog, click_count: 5, time dist:[FORMAT_END=984216, ObMalloc#start=1, ObMalloc#end=3963611, ALLOC_END=23142, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:20.833390] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4946225, , BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:20.735678] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=1012851][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3988759, owner: ObLog, click_count: 5, time dist:[FORMAT_END=11, ObMalloc#start=0, ObMalloc#end=2987417, ALLOC_END=1001330, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:09.780599] EDIAG [ELECT] prepare (election_proposer.cpp:380) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=2675][errcode=0] LOGGER COST TOO MUCH TIME, cost: 15969090, owner: ObLog, click_count: 5, time dist:[FORMAT_END=14976312, ObMalloc#start=1, ObMalloc#end=992773, ALLOC_END=3, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0xf9d67b1 0xf9d4184 0xfa09ca2 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:24.770396] EDIAG [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15951][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1951616, , BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x99721a1 0x997a72e 0x2006ed17 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:24.817377] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_timer.cpp:125) [862][T1002_ArcTimer][T1002][YB42AC4D0007-00063C495E599C16-0-0] [lt=1953160][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1912062, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1912058, ALLOC_END=3, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x11406c8f 0x114063aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:25.756361] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:25.774757] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18367][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:25.756248] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=1979] gc stale ls task succ [2025-08-14 09:03:24.776196] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1001704][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:25.777374] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=999651] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:25.777387] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:25.735329] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=4903532] PNIO [ratelimit] time: 1755162204776247, bytes: 0, bw: 0.000000 MB/s, add_ts: 4930651, add_bytes: 0 [2025-08-14 09:03:24.744020] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:25.778300] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=42895] PNIO [ratelimit] time: 1755162205778299, bytes: 0, bw: 0.000000 MB/s, add_ts: 1002052, add_bytes: 0 [2025-08-14 09:03:24.768517] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=3898568] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:03:25.778336] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1032721][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:25.778364] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:25.741842] WDIAG ev_invoke_pending (ev.c:2136) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=4031221][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:25.778426] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=36521] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:25.778440] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=9] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:18.759940] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:233) [529][T1_RFLWorker][T1][YB42AC4D0007-00063C495B199C16-0-0] [lt=712290] ObRemoteFetchWorker is running(thread_index=0) [2025-08-14 09:03:22.057956] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=49][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a6178, timer:0x7f9966fddee0, task:0x7f994f1b2fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161720715668, pushed_time:0, delay:900000000}, running_task_token={this:0x7f994e1a65d8, timer:0x7f9966fddee0, task:0x7f994f1bbcc0, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161694824965, pushed_time:1755161925772608, delay:5000000}, total_delay=481342286, threashold=1000000, tenant_id=1) [2025-08-14 09:03:25.732288] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:199) [664][T1001_COMMONLSS][T1001][YB42AC4D0007-00063C496439FDEB-0-0] [lt=163723302][errcode=-4283] result set close failed(ret=-4283) [2025-08-14 09:03:25.755839] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=1001132] current monitor number(seq_=-1) [2025-08-14 09:03:25.778565] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:951) [664][T1001_COMMONLSS][T1001][YB42AC4D0007-00063C496439FDEB-0-0] [lt=46204][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2025-08-14 09:03:21.773699] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=1016033] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:03:24.768079] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15188][errcode=-4389] timer task use too much time(end=1755162204768078, start=1755162198769597, delta=5998481) [2025-08-14 09:03:25.779612] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1009979][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:25.779659] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=1035092][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:25.779684] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4389] timer task use too much time(end=1755162205779683, start=1755162201713321, delta=4066362) [2025-08-14 09:03:25.779733] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4023110][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:25.779749] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:25.779760] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:15.800537] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=17] [thread pool stat](pool_stat={pool:0x564d6e8922c0, last_check_time:1755162195800535, last_idle_time:102477764956, last_thread_count:12}, *pool={name:TimerWK, this:0x564d6e8922c0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:10, threads_idle_time:102477764956, tenant_id:500}, qsize=8, current_idle_time=102477764956) [2025-08-14 09:03:18.065797] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc4658, timer:0x7f9970f40f80, task:0x564d6d07a468, task_type:"N9oceanbase8observer23ObSchemaReleaseTimeTaskE", scheduled_time:1755161716573087, pushed_time:0, delay:0}, running_task_token={this:0x7f9970fc5fb8, timer:0x7f9970f40f80, task:0x564d727182f0, task_type:"N9oceanbase5share16ObAshRefreshTaskE", scheduled_time:1755161611717613, pushed_time:1755161614757742, delay:120000000}, total_delay=481492709, threashold=1000000, tenant_id=500) [2025-08-14 09:03:25.089254] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:24.997064] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:25.782282] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162205782281, start=1755162204997062, delta=785219) [2025-08-14 09:03:25.756646] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=2955975] blacklist_loop exec finished(cost_time=15, is_enabled=true, send_cnt=0) [2025-08-14 09:03:24.769275] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=10] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:03:17.774777] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:03:25.784831] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119812272, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-08-14 09:03:25.784870] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=456042971, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:25.784882] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:25.786062] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1001}, delta=461057714, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:25.786076] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:25.786507] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:25.786528] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [578][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=999305][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:03:25.786544] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [578][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:03:25.786613] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=2987002][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162197795659], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:25.786634] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162197795659], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:20.833390] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:25.787125] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=4946225][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:25.787119] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=4970975][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:25.787137] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task use too much time(end=1755162205787136, start=1755162200833387, delta=4953749) [2025-08-14 09:03:25.787138] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4389] timer task use too much time(end=1755162205787138, start=1755162197775129, delta=8012009) [2025-08-14 09:03:25.789675] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4389] run this task use too much time(*p_task_={this:0x7f996f40b7f0, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40bad0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:0, time_interval:10.00s, expected_run_time:09:03:28.802, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:25.807348] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:25.816010] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:25.831062] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:25.887352] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:25.917158] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:26.182110] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=691606][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:26.247048] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:26.715230] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=7] gc stale ls task succ [2025-08-14 09:03:26.720467] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=4] gc stale ls task succ [2025-08-14 09:03:26.721257] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:26.721573] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:24.770396] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15951] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:26.715278] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=991264] [COMPACTION DAG STATUS] (start_time=1755162191760242, end_time=1755162206715274, dag_status={}) [2025-08-14 09:03:26.725779] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=16] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:03:26.720334] INFO [RPC.OBRPC] do_client_loop (ob_net_keepalive.cpp:709) [406][KeepAliveClient][T0][Y0-0000000000000000-0-0] [lt=18] mark black, addr: "172.77.0.7:2882", last_write_ts_=1755162204823510, last_read_ts_=1755162201728586 [2025-08-14 09:03:25.732042] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:26.728159] WDIAG ev_invoke_pending (ev.c:2136) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=994611][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:26.729509] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:24.817377] INFO [ARCHIVE] do_thread_task_ (ob_archive_timer.cpp:125) [862][T1002_ArcTimer][T1002][YB42AC4D0007-00063C495E599C16-0-0] [lt=1953160] archive round not in doing status, just skip(key={incarnation:-1, dest_id:-1, round:-1}, state={status:"INVALID"}) [2025-08-14 09:03:26.733305] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4776] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:26.733890] INFO [RPC.OBRPC] try_read_response (ob_net_keepalive.cpp:624) [406][KeepAliveClient][T0][Y0-0000000000000000-0-0] [lt=6510] whitewash, addr: "172.77.0.7:2882", last_write_ts_=1755162204823510, last_read_ts_=1755162206725955 [2025-08-14 09:03:21.773699] EDIAG [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=1016033][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3999744, owner: ObLog, click_count: 3, time dist:[FORMAT_END=3999743, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9826e08 0x98267df 0x982650b 0x98262e0 0x1d188ec1 0x1d17b8ec 0x1d17ad88 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:26.742672] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:26.733876] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=4005949][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:26.744025] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=3994223] snapshot_traffic_info once [2025-08-14 09:03:26.733603] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=4998837][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:26.733798] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=4989014] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:03:26.733809] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=4990427] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:03:26.744312] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:26.744415] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=4782] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:26.746790] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7996648][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:03:26.746833] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:03:26.746846] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:03:26.746866] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:03:26.746877] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:03:26.746894] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:03:26.746912] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:03:26.746923] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:03:26.750667] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:26.751106] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:26.751609] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:26.754256] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:26.754650] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:26.756709] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=18505] gc stale ls task succ [2025-08-14 09:03:26.758261] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:26.758653] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:25.780800] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9965161] [thread pool stat](pool_stat={pool:0x7f99667cbcf0, last_check_time:1755162205780799, last_idle_time:34814952535, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x7f99667cbcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:34815954572, tenant_id:500}, qsize=-1, current_idle_time=34814952535) [2025-08-14 09:03:26.760401] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=978118] [thread pool stat](pool_stat={pool:0x7f99667ccaf0, last_check_time:1755162206760399, last_idle_time:26590329821, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x7f99667ccaf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:26590329821, tenant_id:500}, qsize=-1, current_idle_time=26590329821) [2025-08-14 09:03:26.760420] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=17] [thread pool stat](pool_stat={pool:0x7f9969de56b0, last_check_time:1755162206760419, last_idle_time:102938681542, last_thread_count:17}, *pool={name:TimerWK, this:0x7f9969de56b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:16, threads_idle_time:102938681542, tenant_id:1}, qsize=10, current_idle_time=102938681542) [2025-08-14 09:03:26.760439] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=18] [thread pool stat](pool_stat={pool:0x7f99356b0e70, last_check_time:1755162206760438, last_idle_time:102366343177, last_thread_count:15}, *pool={name:TimerWK, this:0x7f99356b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:12, threads_idle_time:102366343177, tenant_id:1001}, qsize=12, current_idle_time=102366343177) [2025-08-14 09:03:26.760451] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] [thread pool stat](pool_stat={pool:0x7f99230b0e70, last_check_time:1755162206760451, last_idle_time:102534969158, last_thread_count:17}, *pool={name:TimerWK, this:0x7f99230b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:16, threads_idle_time:102534969158, tenant_id:1002}, qsize=15, current_idle_time=102534969158) [2025-08-14 09:03:26.763680] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:26.595416] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:26.765252] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] timer task use too much time(end=1755162206765251, start=1755162206595415, delta=169836) [2025-08-14 09:03:26.768323] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1951616] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:03:26.774342] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:26.778046] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:26.779259] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:26.779259] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=22697] current monitor number(seq_=-1) [2025-08-14 09:03:26.779256] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:26.780252] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:26.781374] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:26.784226] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=11] PNIO [ratelimit] time: 1755162206784223, bytes: 0, bw: 0.000000 MB/s, add_ts: 1005924, add_bytes: 0 [2025-08-14 09:03:26.784849] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=6984269] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:03:26.787290] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=3988759] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:26.787788] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:26.808262] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:26.816906] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:26.890601] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:26.925878] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a5d18, timer:0x7f99200ff180, task:0x7f99200ff1a8, task_type:"N9oceanbase8observer18ObTableLoadService17ObCheckTenantTaskE", scheduled_time:1755162085759970, pushed_time:0, delay:1000000}, running_task_token={this:0x7f991b7a5fb8, timer:0x7f99200ff180, task:0x7f99200ff1f0, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162050755138, pushed_time:1755162084760013, delay:1000000}, total_delay=121165900, threashold=1000000, tenant_id=1002) [2025-08-14 09:03:26.927763] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:27.018286] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:27.249494] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:27.715389] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=30] gc stale ls task succ [2025-08-14 09:03:27.720419] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:27.720530] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=14] gc stale ls task succ [2025-08-14 09:03:26.763194] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [578][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162197795415]}) [2025-08-14 09:03:27.725888] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:03:27.726274] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:27.730470] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:26.725512] EDIAG [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10214][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1004473, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1004462, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x99721a1 0x997a72e 0x2006ed17 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:26.725512] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10214] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:27.731547] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=1004473] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:27.731560] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:27.731573] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:27.731585] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:26.768206] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10389] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:03:26.746934] EDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] LOGGER COST TOO MUCH TIME, cost: 1008203, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1008202, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x13b06ffb 0x958a39e 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:25.755704] EDIAG do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=3007166][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1998901, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1998900, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x202c530b 0x202c6888 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [316]OB_MALLOC COST TOO MUCH TIME, cost_time=995400, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=78, WASH_BLOCK_END=173, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=21023], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:26.742458] EDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [316][pnio1][T1001][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] LOGGER COST TOO MUCH TIME, cost: 1990748, owner: ObLog, click_count: 5, time dist:[FORMAT_END=995319, ObMalloc#start=1, ObMalloc#end=995422, ALLOC_END=5, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1c02872e 0x1c032753 0x1c034ae1 0x97bc914 0x97bc4ca 0x27ad7f11 0x97b81a7 0x27a8a6d6 0x97b7ba0 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [720]OB_MALLOC COST TOO MUCH TIME, cost_time=1007356, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1007349], seq:[0] [720]OB_MALLOC COST TOO MUCH TIME, cost_time=1007380, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1007376], seq:[0, 1] [2025-08-14 09:03:25.768614] EDIAG [OCCAM] operator() (ob_occam_timer.h:85) [720][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 2975571, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1968175, ObMalloc#start=0, ObMalloc#end=1007388, ALLOC_END=7, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ffa3b 0x9835318 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [832]OB_MALLOC COST TOO MUCH TIME, cost_time=1000583, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=24352], seq:[0, 1] [2025-08-14 09:03:27.746788] EDIAG [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=3999744][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1000622, owner: ObLog, click_count: 5, time dist:[FORMAT_END=18, ObMalloc#start=1, ObMalloc#end=1000600, ALLOC_END=3, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0xfd250b6 0xfd247d3 0xfd244ea 0xfd242a2 0x1d19629b 0x1d17baa3 0x1d17ad94 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:26.768345] EDIAG [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1980415, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1980413, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x2006a4c4 0x997aa00 0x2006ed17 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:26.768179] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10375] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:03:27.738419] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:27.739283] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:674) [812][T1002_TransTime][T1002][Y0-0000000000000000-0-0] [lt=47] rpc statistics(total_trans_msg_count=1, total_batch_msg_count=0) [2025-08-14 09:03:27.740260] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:27.744805] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:27.748042] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:27.750876] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:27.750903] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:27.753937] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:26.746934] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:03:27.756698] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=1008203][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:03:27.756713] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:03:27.756725] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:03:27.756853] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=19] gc stale ls task succ [2025-08-14 09:03:25.755704] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=3007166] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:27.756903] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:03:27.783462] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:27.818207] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:27.883184] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:28.094110] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:28.622690] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:28.720293] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:28.734987] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:26.742458] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [316][pnio1][T1001][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1001, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162189873283", warnings:[]}, tenant_id_=1001, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:03:27.772882] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=162] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:03:28.736315] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:28.744276] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:28.744913] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:25.768614] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [720][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571f9d70, caller:ob_table_lock_service.cpp:start:276, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE6$_1273, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa090, time_wheel:0x7f99375f7d10, thread_pool:0x7f99375f74f0, is_running:0, time_interval:10.00s, expected_run_time:09:03:32.332, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:28.730889] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:288) [570][T1_ArcFetcher][T1][YB42AC4D0007-00063C495B999C16-0-0] [lt=984425] ObArchiveFetcher is running(thread_index=0) [2025-08-14 09:03:28.748710] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [720][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=2975571][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:27.746788] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=3999744] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:03:27.760401] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=3488][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:03:28.751262] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] task executed too delay(delay_time=24951063, *p_task_={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:0, time_interval:500.00ms, expected_run_time:09:03:28.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:26.768345] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=20] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [389]OB_MALLOC COST TOO MUCH TIME, cost_time=1004638, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=15368], seq:[0, 1] [2025-08-14 09:03:27.751419] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1004678, owner: ObLog, click_count: 5, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=1004658, ALLOC_END=1, APPEND_END=7], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:27.751419] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=34] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:28.757643] WDIAG ev_invoke_pending (ev.c:2136) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1004678][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:28.757665] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:27.767678] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [762][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB90F7-0-0] [lt=27][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1001, svr="172.77.0.7:2882", version={val:1755161719775179002, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162189873277) [2025-08-14 09:03:28.759411] INFO [PALF] register_parent_ (log_config_mgr.cpp:2707) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=12885] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.77.0.7:2882", reason=1, leader="172.77.0.7:2882", register_time_us=1755162208736282, last_submit_register_req_time_us=1755162208736282, register_parent_reason=1) [2025-08-14 09:03:28.759436] INFO [PALF] check_parent_health (log_config_mgr.cpp:2884) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=21] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.77.0.7:2882") [2025-08-14 09:03:27.763430] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=447627048, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=251761, succ_cluster_heartbeat_count=1502, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161006924036004, v:0}, local_cluster_delta=1198924810, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:03:28.759953] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=995002] get_max_applied_scn(scn={val:1755161749741193000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:50, palf_committed_end_lsn_:{lsn:1604090090}, palf_committed_end_scn_:{val:1755161749741193000, v:0}, last_check_scn_:{val:1755161749741193001, v:0}, max_applied_cb_scn_:{val:1755161749741193000, v:0}}) [2025-08-14 09:03:28.759977] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get_min_unreplayed_log_info(lsn={lsn:1604088821}, scn={val:1755100948296325001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873479, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:12492179, ret_code_:0}}, next_to_submit_lsn_:{lsn:1604088821}, next_to_submit_scn_:{val:1755100948296325001, v:0}, base_lsn_:{lsn:1409200128}, base_scn_:{val:1753840802540914000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:1604088821}, end_lsn:{lsn:1604088821}, read_buf:{buf_len_:3747840, buf_:0x7f98f6a05000}, block_size:67104768, log_storage_:0x7f990bfebbf0, log_storage_:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:42, read_io_size:147709952, read_disk_cost_ts:1061450, start_lsn:{lsn:1409200128}}}}}}) [2025-08-14 09:03:28.760046] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=66] get wrs ts(ls_id={id:1}, delta=459007858, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:28.760063] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:28.760086] INFO [STORAGE] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:123) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] wead read timestamp is too old(timestamp={val:1755161749741193000, v:0}, gts_scn={val:1755161762721798002, v:0}, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1755161749741193000, v:0}}) [2025-08-14 09:03:28.760633] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=464021227, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:28.760652] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:28.760712] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119812272, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-08-14 09:03:28.760736] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=459007858, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:28.760747] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:27.760168] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21119] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=658032752, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=253211, succ_cluster_heartbeat_count=2213, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161000606550000, v:0}, local_cluster_delta=1205164718, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:03:28.761329] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1001}, delta=464021227, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:28.761343] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:28.761446] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=999115] get wrs ts(ls_id={id:1}, delta=661010535, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:28.761467] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:28.761890] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:28.761922] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:28.762000] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:28.762073] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=8] gc stale ls task succ [2025-08-14 09:03:28.762096] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=661010535, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:28.762096] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=303] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:28.762111] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:28.736153] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:981) [664][T1001_COMMONLSS][T1001][YB42AC4D0007-00063C496439FDEB-0-0] [lt=12][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"select a.tenant_id as tenant_id, a.ls_id as ls_id, a.primary_zone as primary_zone, a.ls_group_id as ls_group_id, b.zone_priority as zone_priority from __all_ls_status as a left join __all_ls_election_reference_info b on a.tenant_id = b.tenant_id and a.ls_id = b.ls_id where a.tenant_id = 1002 order by a.ls_group_id"}, record_ret=-4283, ret=-4283) [2025-08-14 09:03:28.753670] WDIAG [MVCC] study (ob_multi_version_garbage_collector.cpp:319) [578][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=959045][errcode=-4012] study min unallocated GTS failed(ret=-4012) [2025-08-14 09:03:28.762621] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=340] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:28.762650] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=711] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:28.762674] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=372] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:27.963778] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [869][T1002_MultiVers][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:28.762918] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=4918331] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:28.762931] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [869][T1002_MultiVers][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162208762931, start=1755162207963776, delta=799155) [2025-08-14 09:03:27.763464] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=485941863, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=252358, succ_cluster_heartbeat_count=1905, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161003593738002, v:0}, local_cluster_delta=1202123304, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:03:28.762999] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=997987] get_max_applied_scn(scn={val:1755161719775179002, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:45, palf_committed_end_lsn_:{lsn:8461098908}, palf_committed_end_scn_:{val:1755161719775179002, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161719775179002, v:0}}) [2025-08-14 09:03:28.763023] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] get_min_unreplayed_log_info(lsn={lsn:8461098908}, scn={val:1755100947600130001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161920778323, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:23084275, ret_code_:0}}, next_to_submit_lsn_:{lsn:8461098908}, next_to_submit_scn_:{val:1755100947600130001, v:0}, base_lsn_:{lsn:7515734016}, base_scn_:{val:1753840867171901045, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6e05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:8461098908}, end_lsn:{lsn:8461098908}, read_buf:{buf_len_:3747840, buf_:0x7f98f6e05000}, block_size:67104768, log_storage_:0x7f99297a3bf0, log_storage_:{log_tail:{lsn:8461098908}, readable_log_tail:{lsn:8461098908}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8455200768}, min_scn:{val:1755160384851871002, v:0}, curr_block_id:126, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:215, min_block_id:93, max_block_id:127, curr_writable_block_id:126}, logical_block_size_:67104768, curr_block_writable_size_:61206628, block_header_serialize_buf_:0x7f99297a48b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:198, read_io_size:699822080, read_disk_cost_ts:4178035, start_lsn:{lsn:7515734016}}}}}}) [2025-08-14 09:03:27.780853] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:28.763122] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] timer task use too much time(end=1755162208763122, start=1755162207780850, delta=982272) [2025-08-14 09:03:28.764264] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=62] get wrs ts(ls_id={id:1}, delta=488987966, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:28.764291] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:28.774295] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=186] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:28.780307] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=190] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:28.781490] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=249] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:28.781522] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=201] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:28.783015] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:28.783246] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=209] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:28.799454] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=231] PNIO [ratelimit] time: 1755162208799452, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007611, add_bytes: 0 [2025-08-14 09:03:28.819739] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:28.861266] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:28.863040] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=13] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:28.894468] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:28.948248] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:29.212553] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:29.256177] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:29.274386] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:29.720899] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:29.720943] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=218] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:29.720968] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=270] gc stale ls task succ [2025-08-14 09:03:28.780527] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=192] current monitor number(seq_=-1) [2025-08-14 09:03:24.770854] EDIAG [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=4004817][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4942778, owner: ObLog, click_count: 5, time dist:[FORMAT_END=4942767, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=1, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b51df0f 0x986bcde 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:24.770854] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=4004817] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136826952869001, v:0}, clog_base_lsn:{lsn:5099962368}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1755071646732636000, v:0}, all_id_meta:{id_meta:[{limited_id:1755161555795565000, latest_log_ts:{val:1755161545796642001, v:0}}, {limited_id:22000001, latest_log_ts:{val:1755135645787806003, v:0}}, {limited_id:60000001, latest_log_ts:{val:1755135646400653001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:50, palf_env_:0x7f9950bf4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-08-14 09:03:25.761613] EDIAG [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=1002867][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3954265, owner: ObLog, click_count: 5, time dist:[FORMAT_END=3954244, ObMalloc#start=1, ObMalloc#end=13, ALLOC_END=1, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b51df0f 0x986bcde 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:27.739330] EDIAG [STORAGE.TRANS] handle_tx_commit_timeout (ob_trans_service_v4.cpp:372) [812][T1002_TransTime][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1979811, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1979804, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1b82c52f 0x1b886e6d 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:25.761613] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=1002867] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136829134139001, v:0}, clog_base_lsn:{lsn:8186781696}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1753840867372367034, v:0}, all_id_meta:{id_meta:[{limited_id:1755161729775179000, latest_log_ts:{val:1755161719775179001, v:0}}, {limited_id:18000001, latest_log_ts:{val:1755157440846556001, v:0}}, {limited_id:60000001, latest_log_ts:{val:1755135657266496001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:46, palf_env_:0x7f99321fa030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-08-14 09:03:29.715513] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=294] gc stale ls task succ [2025-08-14 09:03:28.759556] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8572] [IO STATUS SERVER] net_in=0kB/s, net_out=0kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=1280000kB/s [2025-08-14 09:03:27.739330] INFO [STORAGE.TRANS] handle_tx_commit_timeout (ob_trans_service_v4.cpp:372) [812][T1002_TransTime][T1002][Y0-0000000000000000-0-0] [lt=25] handle tx commit timeout(ret=0, tx_id={txid:8095834}, ref_cnt=4, cb_executed=false) [2025-08-14 09:03:27.767678] EDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [762][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB90F7-0-0] [lt=27][errcode=-4341] LOGGER COST TOO MUCH TIME, cost: 1953725, owner: ObLog, click_count: 5, time dist:[FORMAT_END=989238, ObMalloc#start=2, ObMalloc#end=5, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0x1c028aa9 0x1c028835 0x1c031b7e 0x1c03844f 0x984d055 0x9280839 0x926c154 0x12c5a0c9 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:29.724351] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=987243][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:03:29.724386] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:03:29.724406] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:03:29.724418] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:03:29.724432] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:03:29.724456] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:03:29.724473] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:03:29.724497] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:03:29.724524] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:03:29.724543] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:03:29.724559] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:03:29.724577] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:03:29.724591] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:03:29.724609] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:03:28.762511] INFO [MVCC] study (ob_multi_version_garbage_collector.cpp:392) [578][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8822] study multi version garabage collector end(ret=-4012, this={this:0x7f994b3ff2a0, last_study_timestamp:1755161000972061, last_refresh_timestamp:1755161553782070, last_reclaim_timestamp:1755161631803871, last_sstable_overflow_timestamp:0, has_error_when_study:true, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1755161000606550000, v:0}, is_inited:true}, min_unallocated_GTS={val:4611686018427387903, v:0}, min_unallocated_GTS={val:4611686018427387903, v:0}, max_committed_txn_version={val:4611686018427387903, v:0}, min_active_txn_version={val:4611686018427387903, v:0}) [2025-08-14 09:03:29.728732] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=231] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:03:29.728812] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=210] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:29.732178] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:29.733556] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [567]OB_MALLOC COST TOO MUCH TIME, cost_time=881539, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1], seq:[0, 1] [857]OB_MALLOC COST TOO MUCH TIME, cost_time=882812, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [381]OB_MALLOC COST TOO MUCH TIME, cost_time=998106, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=15, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=998070], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:26.742198] EDIAG [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1023792, , BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1b5046cb 0x1b5010b8 0x1b501581 0x1b5009d1 0x1b50387c 0x1b8905de 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:20.772811] EDIAG [RS] get_valid_sts_after (ob_tenant_info_loader.cpp:454) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=9119][errcode=0] LOGGER COST TOO MUCH TIME, cost: 7963898, , BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1001d942 0x996d843 0x996d15a 0x996cfd8 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:27.788450] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1958483, , BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:27.891253] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1855926, , BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:27.787800] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1959291, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:29.724628] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:03:29.740744] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15925][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:03:29.740785] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:03:29.741797] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [762][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB90F7-0-0] [lt=1953725][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1001, req={req_server:"172.77.0.7:2882", version:{val:1755161719775179002, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162189873277}, twrs={inited:true, tenant_id:1001, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162208762990}, server_version_for_stat:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162208762990}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:03:29.741993] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:29.744956] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=459998279, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:29.744983] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:29.745077] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=662000956, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:29.745094] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:29.745597] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] get wrs ts(ls_id={id:1001}, delta=465011648, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:29.745616] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:29.745643] INFO [STORAGE.TRANS] amend (ob_tenant_weak_read_server_version_mgr.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [WRS] tenant has no valid partition on current server, generate weak read snapshot version based on current timestamp(tenant_id=1002, epoch_tstamp_=1755162209744938, total_part_count_=2, valid_inner_part_count_=0, valid_user_part_count_=0, version_={val:1755161758221798000, v:0}, base_version_when_no_valid_partition={val:1755161758221798000, v:0}) [2025-08-14 09:03:29.745678] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=459998279, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:29.745689] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:29.745712] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=662000956, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:29.745725] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:28.901324] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=3714949][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a5fb8, timer:0x7f994b3feee0, task:0x7f994b3fef80, task_type:"N9oceanbase8observer18ObTableLoadService21ObClientTaskPurgeTaskE", scheduled_time:1755162027715356, pushed_time:0, delay:1000000}, running_task_token={this:0x7f994e1a5df8, timer:0x7f994b3feee0, task:0x7f994b3fef50, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162023758648, pushed_time:1755162122757495, delay:1000000}, total_delay=181185966, threashold=1000000, tenant_id=1) [2025-08-14 09:03:29.746310] WDIAG [MVCC] repeat_study (ob_multi_version_garbage_collector.cpp:186) [578][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=960708][errcode=-4012] repeat study failed, we will retry immediately(ret=-4012, this={this:0x7f994b3ff2a0, last_study_timestamp:1755161000972061, last_refresh_timestamp:1755161553782070, last_reclaim_timestamp:1755161631803871, last_sstable_overflow_timestamp:0, has_error_when_study:true, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1755161000606550000, v:0}, is_inited:true}, current_timestamp=1755162201780478) [2025-08-14 09:03:29.748956] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=12] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:27.788450] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:29.749918] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1958483][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:29.749936] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] timer task use too much time(end=1755162209749935, start=1755162207788448, delta=1961487) [2025-08-14 09:03:27.891253] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:29.750006] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=1855926][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:29.750018] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task use too much time(end=1755162209750018, start=1755162207891251, delta=1858767) [2025-08-14 09:03:27.787800] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:28.764318] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] [WRS] update tenant weak read server version(tenant_id=1001, server_version={version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162208762990}, version_delta=488989139) [2025-08-14 09:03:29.753326] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=986873] get wrs ts(ls_id={id:1}, delta=489964293, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:29.753347] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:29.753956] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=489964293, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:29.753968] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:29.754732] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:29.754833] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1001}, delta=465026935, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:29.754859] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:29.755442] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:29.758420] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:29.760495] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [858]OB_MALLOC COST TOO MUCH TIME, cost_time=971656, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=971648], seq:[0] [2025-08-14 09:03:29.761260] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:29.763281] WDIAG ev_invoke_pending (ev.c:2136) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11689][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:29.777107] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=969839] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:03:29.777354] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=1008291] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:03:29.866087] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=22] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:30.721417] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:30.721451] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:30.721473] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=10] gc stale ls task succ [2025-08-14 09:03:30.733533] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=30] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:30.740891] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=490965707, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=252360, succ_cluster_heartbeat_count=1905, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161003593738002, v:0}, local_cluster_delta=1207147148, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:03:30.740992] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=47] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:30.734079] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=3954265] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:30.736145] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=4942778] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:30.746150] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=238][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:03:30.746176] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:03:30.746197] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:03:30.746209] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:03:30.746222] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:03:30.746250] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:03:30.746262] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:03:30.746281] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:03:30.746298] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:03:30.746328] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:03:30.746340] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:03:30.746352] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [528]OB_MALLOC COST TOO MUCH TIME, cost_time=1995074, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=56, WASH_BLOCK_END=998644, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=13236], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:28.730877] EDIAG [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=47][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2013779, owner: ObLog, click_count: 5, time dist:[FORMAT_END=18676, ObMalloc#start=1, ObMalloc#end=1995096, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x12377491 0x1236e8e9 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [858]OB_MALLOC COST TOO MUCH TIME, cost_time=983742, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=50, WASH_BLOCK_END=83, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=971676], seq:[0, 1, 2, 3, 4] [857]OB_MALLOC COST TOO MUCH TIME, cost_time=902483, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=903213, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [2025-08-14 09:03:28.730877] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=47] finish to resize log external storage handler(current_ts=1755162208730872, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-08-14 09:03:30.747735] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=2013779] snapshot_traffic_info once [2025-08-14 09:03:30.749057] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=16] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:30.749625] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=461005465, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:30.749648] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:30.749676] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] get_max_applied_scn(scn={val:1755161744727824001, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:49, palf_committed_end_lsn_:{lsn:7323686009}, palf_committed_end_scn_:{val:1755161747742105001, v:0}, last_check_scn_:{val:1755161749741193000, v:0}, max_applied_cb_scn_:{val:1755161744727824001, v:0}}) [2025-08-14 09:03:30.746377] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:03:30.750014] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=3619][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:03:30.750036] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:03:30.750210] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:03:30.750226] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [332]OB_MALLOC COST TOO MUCH TIME, cost_time=990083, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=990081], seq:[0] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=990104, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=990099], seq:[0, 1] [2025-08-14 09:03:27.736272] EDIAG [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=966630][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3016774, owner: ObLog, click_count: 3, time dist:[FORMAT_END=3016773, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1d17bb52 0x1d17ada0 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:27.735713] EDIAG [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=966020][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3018267, owner: ObLog, click_count: 6, time dist:[FORMAT_END=3017169, ObMalloc#start=1, ObMalloc#end=43, ObMalloc#start=22, ObMalloc#end=42, FREE_END=990], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1d17bb52 0x1d17ada0 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:28.761410] EDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1728][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1994580, , BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0xfc0e73d 0x993f560 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:28.748946] EDIAG [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=1000622][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2006596, owner: ObLog, click_count: 6, time dist:[FORMAT_END=2004825, ObMalloc#start=1, ObMalloc#end=62, ObMalloc#start=23, ObMalloc#end=41, FREE_END=1644], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1d17bb52 0x1d17ada0 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:30.750813] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:30.751397] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:30.752663] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8329] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:29.762756] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=1998901] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:30.754539] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=801] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:30.754566] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:30.745547] INFO [CLOG] update_upstream (ob_remote_location_adaptor.cpp:76) [829][T1002_LogRessvr][T1002][YB42AC4D0007-00063C495D199C16-0-0] [lt=13] primary tenant, just skip [2025-08-14 09:03:29.748918] WDIAG [RPC] handle_resp (ob_poc_rpc_proxy.cpp:192) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=1990748][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:30.757569] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1980415] [COMPACTION DAG STATUS] (start_time=1755162191766259, end_time=1755162210757567, dag_status={}) [2025-08-14 09:03:27.736272] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=966630] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623095217325, flush_io_finished_ret_:0, flush_io_finished_round_:26347, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:03:30.750158] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=1959291] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:29.762579] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=9] gc stale ls task succ [2025-08-14 09:03:30.749805] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get_max_applied_scn(scn={val:1755161547738516001, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:50, palf_committed_end_lsn_:{lsn:5282737655}, palf_committed_end_scn_:{val:1755161547738516001, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161547738516001, v:0}}) [2025-08-14 09:03:30.757790] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7966] get_min_unreplayed_log_info(lsn={lsn:5282737655}, scn={val:1755100947825033001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161624848121, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:13063726, ret_code_:0}}, next_to_submit_lsn_:{lsn:5282737655}, next_to_submit_scn_:{val:1755100947825033001, v:0}, base_lsn_:{lsn:4831543296}, base_scn_:{val:1755063445733337001, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7605000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:5282737655}, end_lsn:{lsn:5282737655}, read_buf:{buf_len_:3747840, buf_:0x7f98f7605000}, block_size:67104768, log_storage_:0x7f993e529bf0, log_storage_:{log_tail:{lsn:5282737655}, readable_log_tail:{lsn:5282737655}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:5234171904}, min_scn:{val:1755152244802535004, v:0}, curr_block_id:78, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:206, min_block_id:42, max_block_id:79, curr_writable_block_id:78}, logical_block_size_:67104768, curr_block_writable_size_:18539017, block_header_serialize_buf_:0x7f993e52a8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:85, read_io_size:301367296, read_disk_cost_ts:2364604, start_lsn:{lsn:4831543296}}}}}}) [2025-08-14 09:03:28.759455] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=11] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:12632581, max_lsn:8461098908, max_scn:{val:1755161719775179002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:8214679957}, reuse_lsn:{lsn:8461098908}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:8419155868}}, last_submit_lsn:{lsn:8461098786}, last_submit_end_lsn:{lsn:8461098908}, last_submit_log_id:12632581, last_submit_log_pid:45, max_flushed_lsn:{lsn:8461098786}, max_flushed_end_lsn:{lsn:8461098908}, max_flushed_log_pid:45, committed_end_lsn:{lsn:8461098908}, last_slide_log_id:12632581, last_slide_scn:{val:1755161719775179002, v:0}, last_slide_lsn:{lsn:8461098786}, last_slide_end_lsn:{lsn:8461098908}, last_slide_log_pid:45, last_slide_log_accum_checksum:1091335677, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f992979c530}, [StateMgr]={this:0x7f99297a09b0, self:"172.77.0.7:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:46}, leader:"0.0.0.0:0", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:8461098908}, scan_disk_log_finished:true, last_check_start_id:12632582, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161899748977, palf_role_change_cb:0x7f99297a7470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:45, prev:{version:1, config:{config_version:{proposal_id:43, config_seq:41}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:45, config_seq:42}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:41, prev_lsn:{lsn:8461098218}, prev_mode_pid:45}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:41, prev_lsn:{lsn:8461098218}, prev_end_lsn:{lsn:8461098377}, prev_mode_pid:45}, reconfig_barrier:{prev_log_proposal_id:41, prev_lsn:{lsn:8461098218}, prev_end_lsn:{lsn:8461098377}, prev_mode_pid:45}, persistent_config_version:{proposal_id:45, config_seq:42}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755162208736282, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755162208736282, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f992979dc70}, [ModeMgr]={palf_id:1, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:45, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:46, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:46, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:1753719139394335001, v:0}, min_block_id:93, min_block_min_scn:{val:1753712284262753001, v:0}, base_lsn_for_block_gc:{lsn:8186781696}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:46}, log_config_meta:{version:3, proposal_id:45, prev:{version:1, config:{config_version:{proposal_id:43, config_seq:41}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:45, config_seq:42}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:41, prev_lsn:{lsn:8461098218}, prev_mode_pid:45}, log_snapshot_meta:{version:2, base_lsn:{lsn:8186781696}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:46, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:655360}, readable_log_tail:{lsn:655360}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8522305536}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/meta", dir_fd:214, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66449408, block_header_serialize_buf_:0x7f99297a2bb8, flashback_version:0}, log_storage:{log_tail:{lsn:8461098908}, readable_log_tail:{lsn:8461098908}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8455200768}, min_scn:{val:1755160384851871002, v:0}, curr_block_id:126, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:215, min_block_id:93, max_block_id:127, curr_writable_block_id:126}, logical_block_size_:67104768, curr_block_writable_size_:61206628, block_header_serialize_buf_:0x7f99297a48b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1755161900739661, this:0x7f99297a10f0}, [Reconfirm]={palf_id:1, self:"172.77.0.7:2882", state:"FETCH_MAX_LOG_LSN", new_proposal_id:46, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"172.77.0.7:2882", majority_max_accept_pid:45, majority_max_lsn:{lsn:8461098908}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:1755161954782466, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f99297a0b30}) [2025-08-14 09:03:30.749698] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get_min_unreplayed_log_info(lsn={lsn:7323685240}, scn={val:1755100946796590002, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873485, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:8946764, ret_code_:0}}, next_to_submit_lsn_:{lsn:7323685240}, next_to_submit_scn_:{val:1755100946796590002, v:0}, base_lsn_:{lsn:7180210176}, base_scn_:{val:1753840803166611000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:7323685240}, end_lsn:{lsn:7323685240}, read_buf:{buf_len_:3747840, buf_:0x7f98f7a05000}, block_size:67104768, log_storage_:0x7f9901131bf0, log_storage_:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:28, read_io_size:98508800, read_disk_cost_ts:645339, start_lsn:{lsn:7180210176}}}}}}) [2025-08-14 09:03:30.758788] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:30.759224] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8968] get wrs ts(ls_id={id:1001}, delta=466018834, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:30.759250] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:30.759280] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119812272, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-08-14 09:03:30.759307] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=461005465, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:30.759316] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:30.753274] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:30.762160] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:30.758365] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=72] get wrs ts(ls_id={id:1}, delta=663008142, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:30.763607] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5230] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:30.765180] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=663024414, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:30.765194] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:30.759855] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=466018834, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:30.767688] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7818] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:30.759876] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1023792] gts statistics(tenant_id=1002, gts_rpc_cnt=0, get_gts_cache_cnt=37, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:03:30.761308] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=46] get wrs ts(ls_id={id:1}, delta=490971479, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:30.767918] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6597] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:30.767937] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [WRS] update tenant weak read server version(tenant_id=1001, server_version={version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162210760762}, version_delta=490992757) [2025-08-14 09:03:30.762914] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=993685] gc stale ls task succ [2025-08-14 09:03:30.768474] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=490992643, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:30.770184] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1705] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:30.769795] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9362] tx gc loop thread is running(MTL_ID()=1) [2025-08-14 09:03:30.776972] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=192] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:30.778393] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=192] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:30.780415] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=186] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:30.781586] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=231] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:30.781627] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=118] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:30.801442] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:30.814696] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=207] PNIO [ratelimit] time: 1755162210814694, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007626, add_bytes: 0 [2025-08-14 09:03:30.820311] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=874069][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:03:30.820384] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=873441][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:03:30.859342] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=30] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=452637540, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=251764, succ_cluster_heartbeat_count=1502, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161006924036004, v:0}, local_cluster_delta=1203935302, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:03:30.864729] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=663126212, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=253214, succ_cluster_heartbeat_count=2213, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161000606550000, v:0}, local_cluster_delta=1210258178, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:03:30.867622] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=11] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:30.960835] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:31.216993] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:31.250424] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:00:20.756, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:50.756], idx=338, next_allow_print_ts="09:00:50.756") [2025-08-14 09:03:31.250458] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:03:31.250476] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:03:31.250485] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:03:31.250502] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:03:31.250531] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:03:31.250542] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=561, timeout ts=09:00:58.739, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:00:48.739], idx=561, next_allow_print_ts="09:01:28.739") [2025-08-14 09:03:31.250556] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:03:31.250577] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:03:31.250599] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:03:31.250614] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:03:31.250627] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:03:31.250645] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:03:31.250664] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:03:31.250680] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:31.257320] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:31.304929] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:31.641007] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=959747, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=959745], seq:[0] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=959917, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=59, WASH_BLOCK_END=80, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=959773], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:30.747858] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7963898] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:03:28.759455] EDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1995940, owner: ObLog, click_count: 7, time dist:[FORMAT_END=1995938, ALLOC_END=2, APPEND_END=0, ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=81, ALLOC_BLOCK#start=12285, ALLOC_BLOCK#end=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0xfc0e73d 0x993f560 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:29.748918] EDIAG [RPC] handle_resp (ob_poc_rpc_proxy.cpp:192) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=1990748][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1006899, owner: ObLog, click_count: 7, time dist:[FORMAT_END=1006898, ALLOC_END=1, APPEND_END=0, ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=79, ALLOC_BLOCK#start=10080, ALLOC_BLOCK#end=373], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x27a861af 0x97bcba5 0x97bc4ca 0x27ad7f11 0x97b81a7 0x27a8a6d6 0x97b7ba0 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:30.715556] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=6530][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1001833, owner: ObLog, click_count: 9, time dist:[ALLOC_BLOCK#start=13, ALLOC_BLOCK#end=102, ALLOC_BLOCK#start=10129, ALLOC_BLOCK#end=106, ALLOC_BLOCK#start=10345, ALLOC_BLOCK#end=4, FORMAT_END=981130, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:30.757585] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:31.717438] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=10] new mysql sessid created(conn.sessid_=3221590977, support_ssl=false) [2025-08-14 09:03:30.746075] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=36210] log compressor cycles once. (ret=0, cost_time=14978229, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25898061824, fast_delete_log_mode=false) [857]OB_MALLOC COST TOO MUCH TIME, cost_time=886689, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1], seq:[0, 1] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=881406, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [729]OB_MALLOC COST TOO MUCH TIME, cost_time=879260, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:03:31.718000] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=1995940] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161956777957, last_flushed_end_lsn:{lsn:8461098908}}]) [2025-08-14 09:03:31.718029] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=23] current log_task status(palf_id=1, self="172.77.0.7:2882", log_id=12632582, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161956777957, last_flushed_end_lsn:{lsn:8461098908}}]) [2025-08-14 09:03:30.769893] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=11323] tx gc loop thread is running(MTL_ID()=1001) [2025-08-14 09:03:31.718729] WDIAG stat_cleanup (debug.c:35) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=1006899][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:30.715556] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=6530] gc stale ls task succ [2025-08-14 09:03:31.718929] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=1001833] gc stale ls task succ [2025-08-14 09:03:31.720603] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:31.721551] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=15] gc stale ls task succ [2025-08-14 09:03:31.727893] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=233] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:31.740964] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:31.729162] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=230] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:03:31.744398] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=961963] [COMPACTION DAG STATUS] (start_time=1755162206715274, end_time=1755162211744397, dag_status={}) [2025-08-14 09:03:31.731988] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2148] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:31.747305] WDIAG ev_invoke_pending (ev.c:2136) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15292][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:31.717022] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=957984] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:31.748050] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=41] get wrs ts(ls_id={id:1}, delta=664008604, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:31.748064] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:31.748094] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161101770923, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755161572801635) [2025-08-14 09:03:31.748601] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=664008604, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:31.748609] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:31.749131] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=491971941, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:31.749140] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:30.770605] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=808] try gc retain ctx [2025-08-14 09:03:31.718624] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=947295] try gc retain ctx [2025-08-14 09:03:31.749803] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=31132] try to retry start_working [2025-08-14 09:03:31.750316] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8] get wrs ts(ls_id={id:1}, delta=491971941, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:31.750334] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:31.751658] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:31.754412] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=990125] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:31.747381] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=30312] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:31.762102] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14698] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:31.762113] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:31.762119] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:31.762126] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:31.762131] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:31.762137] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:31.762145] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:03:31.762151] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:03:31.762977] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=6881] gc stale ls task succ [2025-08-14 09:03:31.735487] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:31.736062] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=2411] current monitor number(seq_=-1) [2025-08-14 09:03:31.750555] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=8802] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:31.751556] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:31.773293] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=22702][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:31.751618] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:31.773333] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21680][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:31.755269] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:31.773366] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18069][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:31.755326] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:31.773399] WDIAG ev_invoke_pending (ev.c:2136) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18046][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:31.775360] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:31.778389] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:31.779405] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=16] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:30.770630] EDIAG [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [854][T1002_ObXAHbWor][T1002][Y0-0000000000000000-0-0] [lt=2989382][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1008322, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=3, FORMAT_END=988194, ObMalloc#start=1, ObMalloc#end=20116, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b64d8ba 0x1b64d154 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:31.757346] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:31.780412] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23030][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:31.758455] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:31.780449] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21961][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:30.770630] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [854][T1002_ObXAHbWor][T1002][Y0-0000000000000000-0-0] [lt=2989382] XA scheduler heartbeat task statistics(avg_time=5258) [2025-08-14 09:03:31.768139] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=966][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:31.781563] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:31.781598] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:31.804884] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=46][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:31.808730] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:31.822300] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=29] PNIO [ratelimit] time: 1755162211822299, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007605, add_bytes: 0 [2025-08-14 09:03:31.867541] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=2697709][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:31.875245] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:31.964345] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:32.252059] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:32.261333] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:32.307119] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:32.494625] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:32.719527] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=7] gc stale ls task succ [2025-08-14 09:03:32.721261] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:32.721616] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=9] gc stale ls task succ [279]OB_MALLOC COST TOO MUCH TIME, cost_time=982419, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=32896, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=982412], seq:[0] [2025-08-14 09:03:30.820928] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=2][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1903510, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=8, ALLOC_BLOCK#end=2, FORMAT_END=1903492, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [729]OB_MALLOC COST TOO MUCH TIME, cost_time=875819, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [822]OB_MALLOC COST TOO MUCH TIME, cost_time=1006771, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=58, WASH_BLOCK_END=1006710, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:31.720420] EDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1994580][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1006809, owner: ObLog, click_count: 7, time dist:[FORMAT_END=17, ObMalloc#start=1, ObMalloc#end=1006786, ALLOC_END=1, APPEND_END=3, ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=2], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xfc0e8a4 0x993f693 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:30.784395] EDIAG [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=25674][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1941615, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1941610, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1fce4e91 0x1fce455d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [567]OB_MALLOC COST TOO MUCH TIME, cost_time=879477, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [575]OB_MALLOC COST TOO MUCH TIME, cost_time=977861, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 16, time dist:[ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=977783, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=967088, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 16, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=967020, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] [857]OB_MALLOC COST TOO MUCH TIME, cost_time=988689, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=21011], seq:[0, 1] [2025-08-14 09:03:30.850876] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1884125, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1884122, ALLOC_END=0, APPEND_END=3], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:32.725941] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=970425] log compressor cycles once. (ret=0, cost_time=1008271, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25898319872, fast_delete_log_mode=false) [2025-08-14 09:03:30.820928] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:32.727354] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1903510][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:32.727373] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4389] timer task use too much time(end=1755162212727371, start=1755162210820926, delta=1906445) [2025-08-14 09:03:32.728035] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=492952018, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:32.728062] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:32.728669] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] get wrs ts(ls_id={id:1}, delta=492953062, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:32.728683] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:31.720420] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1994580] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161974728707, last_flushed_end_lsn:{lsn:1604090331}}]) [2025-08-14 09:03:32.728820] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1006809] current log_task status(palf_id=1, self="172.77.0.7:2882", log_id=8607756, log_task={header:{begin_lsn:{lsn:1604090090}, end_lsn:{lsn:1604090203}, log_id:8607756, min_scn:{val:1755161749741193001, v:0}, max_scn:{val:1755161749741193001, v:0}, data_len:57, proposal_id:50, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:1604090090}, data_checksum:2031283194, accum_checksum:759186326, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1755161750716165, freeze_ts:1755161750716173, submit_ts:1755161750716183, flushed_ts:1755161853765770, gen_to_freeze cost time:8, gen_to_submit cost time:18, submit_to_flush cost time:103049587}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161974728707, last_flushed_end_lsn:{lsn:1604090331}}]) [2025-08-14 09:03:30.784395] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=25674] blacklist_loop exec finished(cost_time=24, is_enabled=true, send_cnt=0) [2025-08-14 09:03:32.731405] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:31.747490] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=46] get wrs ts(ls_id={id:1}, delta=462005927, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:32.737728] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=988737] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:30.850876] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:32.737888] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=1884125][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:32.737904] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4389] timer task use too much time(end=1755162212737904, start=1755162210850873, delta=1887031) [2025-08-14 09:03:31.751967] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:31.768552] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=27] sm conn init succ(conn.sessid_=3221590977, sess.client_addr_="172.77.0.1:48098") [2025-08-14 09:03:31.759174] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=7578] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:32.738262] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [719][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [387]OB_MALLOC COST TOO MUCH TIME, cost_time=1022596, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=49491], seq:[0, 1] [2025-08-14 09:03:31.720577] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1022629, owner: ObLog, click_count: 5, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=1022613, ALLOC_END=4, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:30.767731] EDIAG [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7840][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1975588, owner: ObLog, click_count: 9, time dist:[ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=85, ALLOC_BLOCK#start=10039, ALLOC_BLOCK#end=1, FORMAT_END=1965454, ObMalloc#start=1, ObMalloc#end=4, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1b503c0a 0x1b503925 0x1b8905de 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [382]OB_MALLOC COST TOO MUCH TIME, cost_time=962403, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=962401], seq:[0] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=993801, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=993790], seq:[0] [2025-08-14 09:03:32.741718] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] get wrs ts(ls_id={id:1001}, delta=468013420, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:32.741735] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:32.742284] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:31.720577] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:32.744777] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1022629][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:32.744794] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:32.737951] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162212737950, start=1755162211751965, delta=985985) [2025-08-14 09:03:32.739793] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1}, delta=665000295, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:32.746272] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6465] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:30.767731] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7840] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162197795659]}) [2025-08-14 09:03:32.749819] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=665004847, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:32.749837] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:32.751900] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:32.752705] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:32.753387] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [721][T1001_OBJLockGC][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:32.753417] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [721][T1001_OBJLockGC][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162212753417, start=1755162212406881, delta=346536) [2025-08-14 09:03:32.754497] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=17] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:32.763044] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=7] gc stale ls task succ [2025-08-14 09:03:32.844937] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=6854][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:33.252834] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:33.719589] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=36] gc stale ls task succ [2025-08-14 09:03:33.721983] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=16] gc stale ls task succ [2025-08-14 09:03:32.773400] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=21713] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:32.768361] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119812272, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-08-14 09:03:33.723532] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=953702] get wrs ts(ls_id={id:1}, delta=463981668, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:33.723551] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:33.724090] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1001}, delta=468995037, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:33.724116] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:32.726480] WDIAG [RPC] create (ob_poc_rpc_server.cpp:140) [316][pnio1][T0][YB42AC4D0007-00063C49607B92C6-0-0] [lt=12][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:33.733310] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:33.738311] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=977375] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:33.741018] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2429] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:33.723422] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:33.747929] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24468][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:33.751341] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:33.751989] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:33.754974] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=9] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:33.763116] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=12] gc stale ls task succ [2025-08-14 09:03:28.751281] EDIAG [ELECT] operator() (election_proposer.cpp:232) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5007773, owner: ObLog, click_count: 5, time dist:[FORMAT_END=5007751, ObMalloc#start=1, ObMalloc#end=4, ALLOC_END=1, APPEND_END=16], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xfa07f77 0xfa07a14 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:33.770933] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=12332][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:34.719658] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=21] gc stale ls task succ [2025-08-14 09:03:34.720530] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [398]OB_MALLOC COST TOO MUCH TIME, cost_time=1963320, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1963314], seq:[0] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=1963347, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1963336], seq:[0, 1] [2025-08-14 09:03:32.754336] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1963368, owner: ObLog, click_count: 5, time dist:[FORMAT_END=9, ObMalloc#start=0, ObMalloc#end=1963355, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [382]OB_MALLOC COST TOO MUCH TIME, cost_time=1966424, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=57, WASH_BLOCK_END=8487, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=962424], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:32.751327] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1966451, owner: ObLog, click_count: 5, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=1966434, ALLOC_END=2, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [857]OB_MALLOC COST TOO MUCH TIME, cost_time=995179, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=1964596, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1964566], seq:[0, 1, 2, 3, 4] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=1964612, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1964611], seq:[0, 1] [2025-08-14 09:03:32.754411] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1964630, owner: ObLog, click_count: 5, time dist:[FORMAT_END=6, ObMalloc#start=0, ObMalloc#end=1964619, ALLOC_END=2, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [729]OB_MALLOC COST TOO MUCH TIME, cost_time=1894885, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1], seq:[0, 1] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=1873897, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1], seq:[0, 1] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=1995708, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=59, WASH_BLOCK_END=95, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=993823], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:32.728882] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1995737, owner: ObLog, click_count: 5, time dist:[FORMAT_END=7, ObMalloc#start=0, ObMalloc#end=1995725, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:32.754336] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:34.720698] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1963368][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:34.720719] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:32.751327] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:34.720774] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1966451][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:34.720819] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=35] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:34.721468] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=-4389] run this task use too much time(*p_task_={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:0, time_interval:500.00ms, expected_run_time:09:03:33.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:33.766754] EDIAG [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1003869, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1003867, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x18bdb084 0x18bda6b1 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [856]OB_MALLOC COST TOO MUCH TIME, cost_time=2011728, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=2011693], seq:[0, 1, 2, 3, 4] [723]OB_MALLOC COST TOO MUCH TIME, cost_time=2010279, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=2010050, ALLOC_CHUNK_END=212], seq:[0, 1, 2, 3] [723]OB_MALLOC COST TOO MUCH TIME, cost_time=2010342, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2010295, WASH_BLOCK_END=47], seq:[0, 1, 2] [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=2001200, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=2001178, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=2001225, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2001212, WASH_BLOCK_END=12], seq:[0, 1, 2] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=2000139, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=2000111, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=2000156, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=2000147, WASH_BLOCK_END=8], seq:[0, 1, 2] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=1996470, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1996447, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=1996487, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1996478, WASH_BLOCK_END=9], seq:[0, 1, 2] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=1992517, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1992496, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=1992534, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1992525, WASH_BLOCK_END=8], seq:[0, 1, 2] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=1990850, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1990820, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=1990879, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1990870, WASH_BLOCK_END=9], seq:[0, 1, 2] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=1989633, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1989611, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=1989650, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1989641, WASH_BLOCK_END=9], seq:[0, 1, 2] [386]OB_MALLOC COST TOO MUCH TIME, cost_time=1989642, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1989623, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [386]OB_MALLOC COST TOO MUCH TIME, cost_time=1989658, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1989650, WASH_BLOCK_END=8], seq:[0, 1, 2] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=1984407, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1984387, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=1984424, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1984415, WASH_BLOCK_END=9], seq:[0, 1, 2] [383]OB_MALLOC COST TOO MUCH TIME, cost_time=2010600, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=16, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=2010563, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=1962718, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=1962694, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=1962738, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1962726, WASH_BLOCK_END=11], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=1942217, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1942190, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=1942233, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1942225, WASH_BLOCK_END=8], seq:[0, 1, 2] [400]OB_MALLOC COST TOO MUCH TIME, cost_time=1899008, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1898984, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [400]OB_MALLOC COST TOO MUCH TIME, cost_time=1899023, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1899015, WASH_BLOCK_END=8], seq:[0, 1, 2] [383]OB_MALLOC COST TOO MUCH TIME, cost_time=2012839, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2012723, WASH_BLOCK_END=116], seq:[0, 1, 2] [571]OB_MALLOC COST TOO MUCH TIME, cost_time=1842596, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=1842570, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [571]OB_MALLOC COST TOO MUCH TIME, cost_time=1842617, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1842607, WASH_BLOCK_END=10], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=1707490, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=18, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1707450, ALLOC_CHUNK_END=13], seq:[0, 1, 2, 3] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=1707507, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1707498, WASH_BLOCK_END=9], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=1510989, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=19, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1510953, ALLOC_CHUNK_END=8], seq:[0, 1, 2, 3] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=1511007, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1510998, WASH_BLOCK_END=9], seq:[0, 1, 2] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=1251733, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=27, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1251691, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=1251749, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1251740, WASH_BLOCK_END=9], seq:[0, 1, 2] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=1027841, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=17, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1027794, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=1027858, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1027850, WASH_BLOCK_END=8], seq:[0, 1, 2] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=1025938, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1025896, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=1025956, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1025947, WASH_BLOCK_END=9], seq:[0, 1, 2] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=943673, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=15, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=943622, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=943701, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=943686, WASH_BLOCK_END=15], seq:[0, 1, 2] [656]OB_MALLOC COST TOO MUCH TIME, cost_time=935461, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=935421, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [656]OB_MALLOC COST TOO MUCH TIME, cost_time=935480, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=935471, WASH_BLOCK_END=9], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=927369, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=927329, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=927388, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=927379, WASH_BLOCK_END=9], seq:[0, 1, 2] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=833196, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=833148, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=833217, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=833205, WASH_BLOCK_END=12], seq:[0, 1, 2] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=521154, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=15, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=521110, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=521172, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=521164, WASH_BLOCK_END=8], seq:[0, 1, 2] [2025-08-14 09:03:34.721501] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:32.754411] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:34.722045] WDIAG ev_invoke_pending (ev.c:2136) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1964630][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:34.722065] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:34.722686] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=76] gc stale ls task succ [2025-08-14 09:03:34.724389] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=948658] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:32.728882] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:34.727661] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1995737][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:34.738446] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:34.727682] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:34.750535] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:34.752478] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:34.752530] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:34.754121] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:33.766754] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=23] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:03:32.742890] EDIAG [STORAGE.TRANS] refresh_elr_tenant_config_ (ob_tx_elr_util.cpp:42) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2029896, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2013921, ObMalloc#start=1, ObMalloc#end=15970, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b878a47 0x1b8787a8 0x1b878197 0x986bac9 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:34.742336] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:34.775779] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=33390][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:34.750395] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:34.755130] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=23] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:32.742890] INFO [STORAGE.TRANS] refresh_elr_tenant_config_ (ob_tx_elr_util.cpp:42) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=6] refresh tenant config success(tenant_id=1001, *this={last_refresh_ts:1755162212742301, can_tenant_elr:true}) [2025-08-14 09:03:34.763165] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=35] gc stale ls task succ [2025-08-14 09:03:34.813555] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:34.882258] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:34.961453] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [402][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=2665168][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:35.720064] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=30] gc stale ls task succ [2025-08-14 09:03:35.720259] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:35.722749] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=12] gc stale ls task succ [2025-08-14 09:03:35.724504] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=30] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:35.729461] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [414]OB_MALLOC COST TOO MUCH TIME, cost_time=2010748, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=2010727, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [2025-08-14 09:03:35.741824] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:35.753530] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=0] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:35.755076] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=31] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:35.755112] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [567]OB_MALLOC COST TOO MUCH TIME, cost_time=1034410, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1034408], seq:[0] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=1034614, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=64, WASH_BLOCK_END=110, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1034431], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:34.727204] EDIAG [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1034650, owner: ObLog, click_count: 5, time dist:[FORMAT_END=18, ObMalloc#start=1, ObMalloc#end=1034624, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1c03a517 0x97a8d73 0x97a88d2 0x97a7481 0x97a6a43 0x97a42c5 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:31.760568] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=212][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3998883, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=967109, ALLOC_BLOCK#start=10192, ALLOC_BLOCK#end=2, FORMAT_END=3021571, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:32.742316] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=12595][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3018846, owner: ObLog, click_count: 3, time dist:[FORMAT_END=3018834, ALLOC_END=1, APPEND_END=11], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [856]OB_MALLOC COST TOO MUCH TIME, cost_time=3003546, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2011749], seq:[0, 1] [2025-08-14 09:03:34.727204] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=666988330, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:35.765192] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] timer task use too much time(end=1755162215765191, start=1755162214882255, delta=882936) [2025-08-14 09:03:35.765251] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] timer task use too much time(end=1755162215765251, start=1755162214813551, delta=951700) [2025-08-14 09:03:35.765462] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=12723] gc stale ls task succ [2025-08-14 09:03:31.760568] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=212] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:35.765530] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3998883][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:35.765557] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:35.765571] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:32.742316] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=12595] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [414]OB_MALLOC COST TOO MUCH TIME, cost_time=3005243, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2974273, WASH_BLOCK_END=30968, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:32.737991] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1090) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=967966] accept one succ(*s={this:0x7f98f733b030, session_id:3221590977, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:211}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:03:34.725987] EDIAG [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1043745, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1043725, ObMalloc#start=2, ObMalloc#end=3, ALLOC_END=0, APPEND_END=14], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0xfebfd41 0x94edd90 0x94edb97 0x97a8fbd 0x97a8cc7 0x97a88d2 0x97a7481 0x97a6a43 0x97a42c5 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:34.720904] EDIAG [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1048897, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1048890, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0xfebfd41 0x94edd90 0x94edb97 0x97a8fbd 0x97a8cc7 0x97a88d2 0x97a7481 0x97a6a43 0x97a42c5 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:34.725987] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get_max_applied_scn(scn={val:1755161719775179002, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:45, palf_committed_end_lsn_:{lsn:8461098908}, palf_committed_end_scn_:{val:1755161719775179002, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161719775179002, v:0}}) [2025-08-14 09:03:34.720904] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get_max_applied_scn(scn={val:1755161749741193000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:50, palf_committed_end_lsn_:{lsn:1604090090}, palf_committed_end_scn_:{val:1755161749741193000, v:0}, last_check_scn_:{val:1755161749741193001, v:0}, max_applied_cb_scn_:{val:1755161749741193000, v:0}}) [2025-08-14 09:03:35.772437] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=62][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:35.780032] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=20677] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:35.920476] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=25][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:03:35.921245] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=14][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:03:35.962103] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [402][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:36.720357] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=35] gc stale ls task succ [2025-08-14 09:03:36.720408] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:31.875341] EDIAG try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4841678, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=852545, ALLOC_BLOCK#end=90, FORMAT_END=3989021, ObMalloc#start=3, ObMalloc#end=10, ALLOC_END=1, APPEND_END=7], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x26499505 0x97074a3 0x26498fa9 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:31.875341] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=28] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:36.724741] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:03:33.853732] EDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=843195][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 2867322, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2867314, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x9757ced 0x97569df 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:36.724806] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:33.853732] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=843195][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a5b58, timer:0x7f994b3feee0, task:0x7f994b3fef08, task_type:"N9oceanbase8observer18ObTableLoadService17ObCheckTenantTaskE", scheduled_time:1755162032742702, pushed_time:0, delay:1000000}, running_task_token={this:0x7f994e1a5df8, timer:0x7f994b3feee0, task:0x7f994b3fef50, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162023758648, pushed_time:1755162122757495, delay:1000000}, total_delay=181111029, threashold=1000000, tenant_id=1) [2025-08-14 09:03:36.729420] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:32.737991] EDIAG [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1090) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=967966][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3027150, owner: ObLog, click_count: 5, time dist:[FORMAT_END=21889, ObMalloc#start=1, ObMalloc#end=3005255, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a59fb0 0x27a59614 0x94da7e4 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:31.749723] EDIAG [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=977637][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4973899, owner: ObLog, click_count: 9, time dist:[ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=977877, ALLOC_BLOCK#start=10246, ALLOC_BLOCK#end=5, FORMAT_END=3985757, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=6], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x1b51d27b 0x986d830 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:35.210312] EDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=767][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 1519125, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1519120, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x9757ced 0x97569df 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:31.749723] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=977637] try to retry start_working [2025-08-14 09:03:35.210312] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=767][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc4738, timer:0x7f9970f40f80, task:0x564d6cd5b940, task_type:"N9oceanbase3sql15ObSQLSessionMgrE", scheduled_time:1755161613781775, pushed_time:0, delay:5000000}, running_task_token={this:0x7f9970fc5fb8, timer:0x7f9970f40f80, task:0x564d727182f0, task_type:"N9oceanbase5share16ObAshRefreshTaskE", scheduled_time:1755161611717613, pushed_time:1755161614757742, delay:120000000}, total_delay=601428535, threashold=1000000, tenant_id=500) [2025-08-14 09:03:32.759283] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3971401, owner: ObLog, click_count: 7, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=2012848, ObMalloc#start=1958536, ObMalloc#end=3, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:36.735374] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [576][T1_TransferServ][T1][YB42AC4D0007-00063C4961D99CE5-0-0] [lt=978140][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:32.759283] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:36.736726] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3971401][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:36.736747] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:36.736763] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:34.731389] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:34.731389] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2002745, owner: ObLog, click_count: 7, time dist:[FORMAT_END=18, ObMalloc#start=1, ObMalloc#end=44219, ObMalloc#start=1958497, ObMalloc#end=2, ALLOC_END=0, APPEND_END=8], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:36.737185] WDIAG ev_invoke_pending (ev.c:2136) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2002745][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:36.737207] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:32.782326] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3948970, owner: ObLog, click_count: 7, time dist:[FORMAT_END=6, ObMalloc#start=0, ObMalloc#end=1989664, ObMalloc#start=1959294, ObMalloc#end=3, ALLOC_END=0, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:33.745938] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2986895, owner: ObLog, click_count: 5, time dist:[FORMAT_END=39, ObMalloc#start=1, ObMalloc#end=1027864, ALLOC_END=1958987, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:36.737221] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:32.782326] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:36.737290] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3948970][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:36.737308] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:36.737323] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:33.745938] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:36.737374] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=2986895][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:36.737396] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task use too much time(end=1755162216737395, start=1755162213745934, delta=2991461) [381]OB_MALLOC COST TOO MUCH TIME, cost_time=966447, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=966446], seq:[0] [2025-08-14 09:03:31.770250] EDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34709][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4960637, owner: ObLog, click_count: 9, time dist:[ALLOC_BLOCK#start=957585, ALLOC_BLOCK#end=66, FORMAT_END=3031402, ObMalloc#start=0, ObMalloc#end=4348, ObMalloc#start=967225, ObMalloc#end=4, ALLOC_END=1, APPEND_END=6], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:32.809348] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3923739, owner: ObLog, click_count: 7, time dist:[FORMAT_END=31, ObMalloc#start=0, ObMalloc#end=1962745, ObMalloc#start=1960956, ObMalloc#end=3, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:32.787604] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [776][T1002_StandbySc][T1002][YB42AC4D0007-00063C495DFA0348-0-0] [lt=23][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3945733, owner: ObLog, click_count: 7, time dist:[FORMAT_END=12, ObMalloc#start=0, ObMalloc#end=1984430, ObMalloc#start=1961285, ObMalloc#end=2, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:33.941120] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2794048, owner: ObLog, click_count: 7, time dist:[FORMAT_END=19, ObMalloc#start=2, ObMalloc#end=833223, ALLOC_END=1960800, APPEND_END=4, ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=53], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:33.830316] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=2][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2904765, owner: ObLog, click_count: 5, time dist:[FORMAT_END=37, ObMalloc#start=1, ObMalloc#end=943707, ALLOC_END=1961016, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:31.770250] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34709][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:36.738431] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4960637] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:36.738449] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:32.809348] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:36.739045] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=3923739][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:36.739066] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4389] timer task use too much time(end=1755162216739066, start=1755162212809347, delta=3929719) [2025-08-14 09:03:33.941120] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:33.830316] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:36.739493] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=2904765][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:36.739509] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4389] timer task use too much time(end=1755162216739508, start=1755162213830313, delta=2909195) [2025-08-14 09:03:36.744497] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:36.751319] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:36.751411] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:36.752261] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:34.649460] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=59][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:34.649460] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=59][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2106273, owner: ObLog, click_count: 7, time dist:[FORMAT_END=1114759, ObMalloc#start=0, ObMalloc#end=274, ObMalloc#start=991227, ObMalloc#end=4, ALLOC_END=0, APPEND_END=9], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:36.758954] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=2106273][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:36.758982] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4389] timer task use too much time(end=1755162216758982, start=1755162214649457, delta=2109525) [2025-08-14 09:03:36.759049] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=4973899] get_max_applied_scn(scn={val:1755161547738516001, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:50, palf_committed_end_lsn_:{lsn:5282737655}, palf_committed_end_scn_:{val:1755161547738516001, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161547738516001, v:0}}) [2025-08-14 09:03:32.782287] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3970810, owner: ObLog, click_count: 7, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=1989660, ObMalloc#start=1981132, ObMalloc#end=2, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:32.781011] EDIAG [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3972165, owner: ObLog, click_count: 5, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=1990885, ALLOC_END=1981263, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a1f867 0x27a135c3 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:32.779290] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3973914, owner: ObLog, click_count: 5, time dist:[FORMAT_END=8, ObMalloc#start=7, ObMalloc#end=1992540, ALLOC_END=1981355, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:32.775297] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3977985, owner: ObLog, click_count: 7, time dist:[FORMAT_END=8, ObMalloc#start=0, ObMalloc#end=1996493, ObMalloc#start=1981477, ObMalloc#end=3, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:32.770463] EDIAG [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=34266][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3982888, owner: ObLog, click_count: 5, time dist:[FORMAT_END=13, ObMalloc#start=1, ObMalloc#end=2001232, ALLOC_END=1981638, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1f832f91 0x97c36fe 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:33.521662] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3232942, owner: ObLog, click_count: 5, time dist:[FORMAT_END=43, ObMalloc#start=2, ObMalloc#end=1251755, ALLOC_END=1981138, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:33.065122] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3688861, owner: ObLog, click_count: 5, time dist:[FORMAT_END=49, ObMalloc#start=2, ObMalloc#end=1707514, ALLOC_END=1981293, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:32.782287] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:36.759129] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3970810][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:36.759150] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:36.759165] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:32.781011] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=15] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:32.779290] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:36.759245] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3973914][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:36.759262] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:36.759277] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:32.775297] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:36.759319] WDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3977985][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:36.759335] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:36.759349] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:32.770463] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=34266] current monitor number(seq_=-1) [2025-08-14 09:03:33.521662] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:36.759516] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=3232942][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:36.759534] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4389] timer task use too much time(end=1755162216759533, start=1755162213521660, delta=3237873) [2025-08-14 09:03:33.065122] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:36.759581] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=3688861][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:36.759594] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task use too much time(end=1755162216759594, start=1755162213065119, delta=3694475) [391]OB_MALLOC COST TOO MUCH TIME, cost_time=1007669, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1007662], seq:[0] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=1018630, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=51, WASH_BLOCK_END=10886, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1007687], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:35.755157] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1018648, owner: ObLog, click_count: 5, time dist:[FORMAT_END=6, ObMalloc#start=0, ObMalloc#end=1018640, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:34.757449] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2017957, owner: ObLog, click_count: 9, time dist:[ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=92, ALLOC_BLOCK#start=10111, ALLOC_BLOCK#end=95, ALLOC_BLOCK#start=10347, ALLOC_BLOCK#end=2, FORMAT_END=1997302, ALLOC_END=3, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:33.261976] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3512400, owner: ObLog, click_count: 5, time dist:[FORMAT_END=37, ObMalloc#start=2, ObMalloc#end=1511013, ALLOC_END=2001348, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [381]OB_MALLOC COST TOO MUCH TIME, cost_time=1009872, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=966459], seq:[0, 1] [2025-08-14 09:03:33.765286] EDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1975588][errcode=-4038] LOGGER COST TOO MUCH TIME, cost: 3011160, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2001274, ObMalloc#start=1, ObMalloc#end=1009883, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1ba227a3 0x1b89078a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:34.253677] EDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4016] LOGGER COST TOO MUCH TIME, cost: 2525052, owner: ObLog, click_count: 7, time dist:[FORMAT_END=23, ObMalloc#start=1, ObMalloc#end=521179, ObMalloc#start=1980754, ObMalloc#end=19409, ALLOC_END=3686, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x13b06ffb 0x958a39e 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:35.755157] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:36.775372] WDIAG ev_invoke_pending (ev.c:2136) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1018648][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:36.775391] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:36.777320] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=12] gc stale ls task succ [2025-08-14 09:03:34.757449] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:36.778489] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2017957][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:36.778505] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:36.778516] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:56.769647] WDIAG [STORAGE] init (ob_multiple_scan_merge.cpp:72) [14945][T1_TimerWK8_MergeLoop][T1][YB42AC4D0007-00063C4978199C2C-0-0] [lt=18625][errcode=-4012] failed to init ObMultipleMerge(ret=-4012, param={iter_param:{table_id:88888, tablet_id:{id:50345}, ls_id:{id:1}, cg_idx:0, read_info:{schema_column_count:4, schema_rowkey_cnt:2, rowkey_cnt:4, trans_col_index:-1, mview_old_new_col_index:-1, group_idx_col_index:-1, seq_read_column_count:2, max_col_index:5, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 4, 5], data:0x7f98c5026740, count:4, allocator:0x7f97efad31e8, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 3], data:0x7f98c5026760, count:4, allocator:0x7f97efad31e8, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=16 {type:"TINYINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"CHAR", collation:"binary", coercibility:"INVALID"} order=0, column_id=18 {type:"CHAR", collation:"binary", coercibility:"INVALID"} order=0, column_id=19 {type:"CHAR", collation:"binary", coercibility:"INVALID"} order=0], data:0x7f98c5026700, count:4, allocator:0x7f97efad31e8, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:4, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}], data:0x7f98c50267b0, count:4, allocator:0x7f97efad31e8, capacity:4, init_cnt:4}, has_all_column_group:true, need_truncate_filter:false}, rowkey_read_info:{is_inited:true, compat_version:2, is_oracle_mode:true, is_cs_replica_compat:0, is_delete_insert_table:0, schema_column_count:4, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:6}, cols_desc:{ObIArray:[cnt:4, column_id=16 {type:"TINYINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"CHAR", collation:"binary", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9956fc6150, count:4, allocator:0x564d6ef37d68, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:true, rowkey_cnt:4, is_inited:true, is_oracle_mode:true}, memtable_cols_index:{rowkey_mode:true, for_memtable:true, schema_rowkey_cnt:2, column_cnt:6}}, out_cols_project:[0, 1, 2, 3], agg_cols_project:[], group_by_cols_project:[], pushdown_filter:NULL, op:null, sstable_index_filter:null, cg_read_infos:null, output_exprs:NULL, aggregate_exprs:NULL, output_sel_mask:[true, true, true, true], is_multi_version_minor_merge:false, need_scn:false, is_same_schema_column:true, pd_storage_flag:{pd_flag:0}, vectorized_enabled:false, has_virtual_columns:false, has_lob_column_out:false, is_for_foreign_check:false, limit_prefetch:true, is_mds_query:true, is_non_unique_local_index:false, ss_rowkey_prefix_cnt:0, table_scan_opt:{io_read_batch_size:0, io_read_gap_size:0, storage_rowsets_size:1}, auto_split_filter_type:18446744073709551615, auto_split_filter:null, auto_split_params:NULL, is_tablet_spliting:false, is_column_replica_table:false, is_delete_insert:false, need_update_tablet_param:0x7f97efad2fc8}, padding_cols:[], projector_size:0, output_exprs:NULL, op_filters:null, row2exprs_projector:null, output_sel_mask:[true, true, true, true], is_inited:true}, context={is_inited:true, use_fuse_row_cache:false, need_scn:false, need_release_mview_scan_info:true, need_release_truncate_part_filter:true, timeout:1755161926786042, ls_id:{id:1}, tablet_id:{id:50345}, query_flag:{scan_order:0, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_bare_row_scan:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:1, is_select_follower:0, enable_lob_prefetch:0, is_bare_row_scan:0, mr_mv_scan:0, is_plain_insert_gts_opt:0, reserved:0}, sql_mode:134217760, store_ctx:0x7f97efacde38, limit_param:null, stmt_allocator:0x7f97efad31e8, allocator:0x7f97efad31e8, range_allocator:null, table_scan_stat:0x7f97efad2578, out_cnt:0, trans_version_range:{multi_version_start:0, base_version:0, snapshot_version:9223372036854775807}, merge_scn:{val:4611686018427387903, v:0}, lob_allocator:pages=0, used=0, total=0, lob_locator_helper:null, cached_iter_node:null, stmt_iter_pool:null, cg_iter_pool:null, cg_param_pool:null, block_row_store:null, sample_filter:null, mview_scan_info:NULL, table_store_stat:{row_cache_hit_cnt:0, row_cache_miss_cnt:0, row_cache_put_cnt:0, bf_filter_cnt:0, bf_access_cnt:0, block_cache_hit_cnt:0, block_cache_miss_cnt:0, fuse_row_cache_hit_cnt:0, fuse_row_cache_miss_cnt:0, fuse_row_cache_put_cnt:0, micro_access_cnt:0, pushdown_micro_access_cnt:0, empty_read_cnt:0, rowkey_prefix:0, logical_read_cnt:0, physical_read_cnt:0, in_row_cache_threshold:100}, truncate_part_filter:null, mds_collector:0x7f97efad3050}, get_table_param={frozen_version:9223372036854775807, sample_info:{method:0, percent:"1.000000000000000000e+02", seed:-1, table_id:0, scope:0, force_block:false}, tablet_iter:{tablet_handle:{obj:0x7f994f3e0a10, obj_pool:0x7f9969be3930, allocator:null, wash_priority:0, allow_copy_and_assign:true}, transfer_src_handle:null, table_store_iter:{table_ptr_array:[], sstable_handle_array:[{phy_addr:{[ver=1,mode=0,seq=131117][2nd=6330][3rd=0][offset=1786186,size=921,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, io_handle:{addr:{[ver=1,mode=0,seq=0][2nd=0][3rd=0][offset=0,size=0,type=5,seq=0][trans_seq=0, sec_part=0][6th=0]}, object_handle:{macro_id:{[ver=1,mode=0,seq=131117][2nd=6330]}, io_handle:{io_result:"{is_inited_:true, is_finished_:false, is_canceled_:false, has_estimated_:false, complete_size_:0, offset_:1786186, size_:921, timeout_us_:10000000, result_ref_cnt_:2, out_ref_cnt_:1, flag_:{mode:"READ", group_id_:0, func_type_:0, wait_event_id_:3, is_sync_:false, is_unlimited_:false, is_detect_:false, is_write_through_:false, is_sealed_:true, is_time_detect_:false, need_close_dev_and_fd_:false, reserved_:0}, ret_code_:{io_ret_:0, fs_errno_:0}, tenant_id_:1, tenant_io_mgr_:{ptr:0x7f994b36a030}, user_data_buf_:null, buf_:null, io_callback_:0x7f9858858e30, time_log:{begin_ts:1755161929718639, enqueue_used:0, dequeue_used:-1, submit_used:-1, return_used:-1, callback_enqueue_used:-1, callback_dequeue_used:-1, callback_finish_used:-1, end_used:-1}}"}}}, cache_handle:{cache_value:{ref_cnt:2, value:NULL, cache_handle:{hazptr_holder:{hazptr:null}}}, allocator:0x7f9969be56b0}}], pos:9223372036854775807, step:1, memstore_retired:null, need_load_sstable:true, table_store_handle:{phy_addr:{[ver=1,mode=0,seq=0][2nd=0][3rd=0][offset=0,size=0,type=5,seq=0][trans_seq=0, sec_part=0][6th=0]}, io_handle:{addr:{[ver=1,mode=0,seq=0][2nd=0][3rd=0][offset=0,size=0,type=5,seq=0][trans_seq=0, sec_part=0][6th=0]}, object_handle:{macro_id:{[ver=1,mode=0,seq=0][2nd=9223372036854775807]}, io_handle:{io_result:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, transfer_src_table_store_handle:NULL, split_extra_table_store_handles:[]}, split_extra_tablet_handles:[]}, need_split_dst_table:true}) [2025-08-14 09:03:33.261976] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:36.779722] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=3512400][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:36.779751] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4389] timer task use too much time(end=1755162216779750, start=1755162213261974, delta=3517776) [2025-08-14 09:03:33.765286] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1975588][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1002}) [2025-08-14 09:03:31.716874] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=967548] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:03:34.253677] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:36.782580] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=2525052][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:31.778320] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [723][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=18] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=16, total_destructed_detector=16, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f99333562b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-08-14 09:03:32.873254] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3904203, owner: ObLog, click_count: 9, time dist:[FORMAT_END=18, ObMalloc#start=1, ObMalloc#end=1899029, ObMalloc#start=1961455, ObMalloc#end=43310, ObMalloc#start=382, ObMalloc#end=2, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:32.929790] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [571][T1_ArcSender][T1][YB42AC4D0007-00063C495BA99C16-0-0] [lt=269][errcode=-4018] LOGGER COST TOO MUCH TIME, cost: 3847991, owner: ObLog, click_count: 7, time dist:[FORMAT_END=18, ObMalloc#start=1, ObMalloc#end=1842623, ObMalloc#start=1961379, ObMalloc#end=43296, ALLOC_END=670, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3b4b 0x113d34aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:32.873254] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:32.929790] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [571][T1_ArcSender][T1][YB42AC4D0007-00063C495BA99C16-0-0] [lt=269][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:03:33.846753] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2935066, owner: ObLog, click_count: 7, time dist:[FORMAT_END=29, ObMalloc#start=1, ObMalloc#end=927394, ObMalloc#start=1980864, ObMalloc#end=26773, ALLOC_END=2, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:36.785414] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=23] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:33.846753] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:36.786292] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=2935066][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:36.786320] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4389] timer task use too much time(end=1755162216786319, start=1755162213846751, delta=2939568) [2025-08-14 09:03:33.838602] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:33.838602] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=2][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2945202, owner: ObLog, click_count: 7, time dist:[FORMAT_END=28, ObMalloc#start=2, ObMalloc#end=935490, ObMalloc#start=1980933, ObMalloc#end=28741, ALLOC_END=1, APPEND_END=6], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:36.788287] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=2945202][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:36.788317] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=-4389] timer task use too much time(end=1755162216788317, start=1755162213838599, delta=2949718) [2025-08-14 09:03:36.962707] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [402][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:37.132047] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=1519125][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc4ff8, timer:0x7f9970f40f80, task:0x564d6d07c170, task_type:"N9oceanbase8observer8ObServer25ObRefreshNetworkSpeedTaskE", scheduled_time:1755161615749643, pushed_time:0, delay:1000000}, running_task_token={this:0x7f9970fc5fb8, timer:0x7f9970f40f80, task:0x564d727182f0, task_type:"N9oceanbase5share16ObAshRefreshTaskE", scheduled_time:1755161611717613, pushed_time:1755161614757742, delay:120000000}, total_delay=601382402, threashold=1000000, tenant_id=500) [2025-08-14 09:03:34.773307] EDIAG [ELECT] operator() (election_acceptor.cpp:137) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2938609, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2938590, ObMalloc#start=3, ObMalloc#end=8, ALLOC_END=0, APPEND_END=8], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xfa03487 0x9835ae8 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:56.769647] EDIAG [STORAGE] init (ob_multiple_scan_merge.cpp:72) [14945][T1_TimerWK8_MergeLoop][T1][YB42AC4D0007-00063C4978199C2C-0-0] [lt=18625][errcode=-4012] LOGGER COST TOO MUCH TIME, cost: 39948657, owner: ObLog, click_count: 5, time dist:[FORMAT_END=39928984, ObMalloc#start=1, ObMalloc#end=4, ALLOC_END=19667, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1d7f3a76 0x94b4a18 0x9766028 0x9765ae5 0x976579e 0x9764de3 0x975ab8f 0x97f7c68 0x97f78b3 0x97f7636 0x97f5260 0x97f4a4a 0x97f4859 0x97f4305 0x1aee6975 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:37.720262] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:37.720420] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=32] gc stale ls task succ [2025-08-14 09:03:37.726744] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=16] gc stale ls task succ [2025-08-14 09:03:37.726790] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=20] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:37.736742] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:37.738265] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:37.738304] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:37.738459] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:37.740280] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:37.746986] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:37.751251] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:37.751405] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:37.751834] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=33] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:37.754229] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:37.727208] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=3018846] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:03:37.744066] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:37.744729] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:37.756561] WDIAG ev_invoke_pending (ev.c:2136) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=3972165][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:37.756592] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=23] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:37.756607] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=10] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:36.732616] EDIAG [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=1941615][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1023329, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1023305, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=1, APPEND_END=19], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1fce4e91 0x1fce455d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:35.739070] EDIAG [COMMON] loop_running_dag_net_list (ob_tenant_dag_scheduler.cpp:3845) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2015422, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1040732, ObMalloc#start=1, ObMalloc#end=69, ObMalloc#start=28, ObMalloc#end=50, FREE_END=974542], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x20061064 0x2006f829 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:36.732616] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=1941615] blacklist_loop exec finished(cost_time=22, is_enabled=true, send_cnt=0) [2025-08-14 09:03:32.759335] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4990599, owner: ObLog, click_count: 8, time dist:[FORMAT_END=12234, ObMalloc#start=1, ObMalloc#end=2000162, ObMalloc#start=1961591, ObMalloc#end=43413, FREE_END=973198, ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=3], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:37.757690] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4990599][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:37.757716] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:37.757742] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:35.738310] EDIAG [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=2006596][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2016431, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2016410, ObMalloc#start=0, ObMalloc#end=3, ALLOC_END=0, APPEND_END=18], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1d189030 0x1d18285e 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:31.778320] EDIAG [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [723][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4997184, owner: ObLog, click_count: 7, time dist:[FORMAT_END=981458, ObMalloc#start=1, ObMalloc#end=2010351, ObMalloc#start=1962036, ObMalloc#end=43334, ALLOC_END=3, APPEND_END=1], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x973b640 0x973b132 0x973a32d 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:32.829921] EDIAG eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4920559, owner: ObLog, click_count: 6, time dist:[FORMAT_END=34, ObMalloc#start=1, ObMalloc#end=1942240, ObMalloc#start=1961527, ObMalloc#end=43328, FREE_END=973429], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x27a8a8c8 0x97b7e1a 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:35.738310] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=2006596] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:03:37.757885] WDIAG [DETECT] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [723][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=4997184][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:35.738657] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2016684, owner: ObLog, click_count: 8, time dist:[ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=3, FORMAT_END=1038324, ObMalloc#start=2, ObMalloc#end=86, ObMalloc#start=21, ObMalloc#end=48, FREE_END=978195], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:36.738373] EDIAG [RS.COMPACTION] try_idle (ob_major_merge_scheduler.cpp:226) [1057][T1_MergeSchedul][T1][YB42AC4D0007-00063C4961599CC3-0-0] [lt=996329][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1018534, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1018525, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1125d502 0x1125cf81 0x1125a4c3 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:37.758598] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=979689][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:33.721259] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4031389, owner: ObLog, click_count: 6, time dist:[FORMAT_END=26624, ObMalloc#start=1, ObMalloc#end=1025961, ObMalloc#start=1961129, ObMalloc#end=43269, FREE_END=974405], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:35.738287] EDIAG [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=3018267][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2017480, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2017473, ObMalloc#start=0, ObMalloc#end=3, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1d189030 0x1d18285e 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:37.742804] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:0, time_interval:250.00ms, expected_run_time:09:03:36.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:37.758771] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4031389][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:37.758791] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:37.758806] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:35.738287] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=3018267] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:03:37.759215] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=4841678] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:36.762303] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=12] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:03:31.716874] EDIAG [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=967548][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5057783, owner: ObLog, click_count: 7, time dist:[FORMAT_END=1039681, ObMalloc#start=2, ObMalloc#end=3003559, ObMalloc#start=1014536, ObMalloc#end=3, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b52e5d4 0x996d21d 0x996cfd8 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:35.739149] EDIAG [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=3016774][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2017893, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2017884, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1d189030 0x1d18285e 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:37.759743] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=5057783][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:35.739149] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=3016774] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:03:37.760776] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] timer task use too much time(end=1755162217760775, start=1755162216759548, delta=1001227) [2025-08-14 09:03:37.761034] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3011160] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1002}) [2025-08-14 09:03:36.515689] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1244305, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1244297, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:36.515689] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:37.761908] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=1244305][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:37.761923] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] timer task use too much time(end=1755162217761922, start=1755162216515686, delta=1246236) [2025-08-14 09:03:37.762299] WDIAG [STORAGE] init_and_open_multiple_scan_merge (ob_mds_row_iterator.cpp:315) [14945][T1_TimerWK8_MergeLoop][T1][YB42AC4D0007-00063C4978199C2C-0-0] [lt=39948657][errcode=-4012] fail to init multiple scan merge(ret=-4012) [2025-08-14 09:03:36.762307] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=15] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:03:37.766190] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=83] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:37.759112] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=33] log compressor cycles once. (ret=0, cost_time=996481, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25898201088, fast_delete_log_mode=false) [2025-08-14 09:03:36.760328] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=214][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1005461, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1005452, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:36.760328] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=214] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:03:37.767342] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=3982888] current monitor number(seq_=-1) [2025-08-14 09:03:37.767414] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:37.767439] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:37.777890] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=15] gc stale ls task succ [2025-08-14 09:03:37.779254] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:37.783430] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:37.785509] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=24] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:37.787002] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:37.861706] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:37.877986] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:37.878042] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:37.889037] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:37.959406] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=13] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:37.989428] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=3904203][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:38.012964] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:38.720263] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:38.720491] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=14] gc stale ls task succ [2025-08-14 09:03:38.729266] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:36.731980] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=961103] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:03:37.767392] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:37.786466] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=45][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:38.735500] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162218735499, start=1755162217786463, delta=949036) [2025-08-14 09:03:37.807320] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:38.735935] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] timer task use too much time(end=1755162218735935, start=1755162217807318, delta=928617) [2025-08-14 09:03:38.736342] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=4920559] PNIO [ratelimit] time: 1755162217757953, bytes: 0, bw: 0.000000 MB/s, add_ts: 4928033, add_bytes: 0 [2025-08-14 09:03:38.735426] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=966570][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:38.738336] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:38.739997] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:38.741164] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [576][T1_TransferServ][T1][YB42AC4D0007-00063C4961D99CE5-0-0] [lt=54][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:38.741257] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:38.742264] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:38.742299] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:38.738278] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:288) [860][T1002_ArcFetche][T1002][YB42AC4D0007-00063C495E399C16-0-0] [lt=45913] ObArchiveFetcher is running(thread_index=0) [2025-08-14 09:03:38.751921] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:38.754401] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=33] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:38.758966] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:38.780228] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:38.761282] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:38.761310] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:38.782283] WDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20939][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:38.762739] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=22] PNIO [ratelimit] time: 1755162218762737, bytes: 0, bw: 0.000000 MB/s, add_ts: 1004784, add_bytes: 0 [2025-08-14 09:03:38.767242] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=36] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:38.782349] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15083][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=948921, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=32, WASH_BLOCK_END=948889], seq:[0, 1, 2] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=978606, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=17606], seq:[0, 1] [2025-08-14 09:03:38.754645] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:39.734783] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162219734782, start=1755162218754642, delta=980140) [658]OB_MALLOC COST TOO MUCH TIME, cost_time=978637, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=17528], seq:[0, 1] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=977161, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=15977], seq:[0, 1] [2025-08-14 09:03:38.754728] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:39.734875] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162219734875, start=1755162218754726, delta=980149) [2025-08-14 09:03:38.756278] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [374]OB_MALLOC COST TOO MUCH TIME, cost_time=998351, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=998349], seq:[0] [2025-08-14 09:03:37.756683] EDIAG [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=3027150][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1976272, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1974843, ObMalloc#start=1, ObMalloc#end=54, ObMalloc#start=26, ObMalloc#end=25, FREE_END=1323], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a5bbf2 0x94da542 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [332]OB_MALLOC COST TOO MUCH TIME, cost_time=950461, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=32, WASH_BLOCK_END=950429], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=644824, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=32, WASH_BLOCK_END=644792], seq:[0, 1, 2] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=926585, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=39, WASH_BLOCK_END=926546], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=878796, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=28, WASH_BLOCK_END=878768], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=782247, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=28, WASH_BLOCK_END=782218], seq:[0, 1, 2] [400]OB_MALLOC COST TOO MUCH TIME, cost_time=752421, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=28, WASH_BLOCK_END=752392], seq:[0, 1, 2] [2025-08-14 09:03:36.778101] EDIAG [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=2029896][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2961452, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2961451, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1bdd73e2 0x1bdd6e36 0x986bc6f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [374]OB_MALLOC COST TOO MUCH TIME, cost_time=1010066, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=76, WASH_BLOCK_END=224, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=998360], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:37.746931] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1997143, owner: ObLog, click_count: 5, time dist:[FORMAT_END=987056, ObMalloc#start=1, ObMalloc#end=1010084, ALLOC_END=2, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:39.734942] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=977184][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:38.784284] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:38.785593] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=20] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:39.742892] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:39.743074] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:38.768313] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=2016684] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:39.720316] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=30] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:36.778101] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=2029896] finish update min start scn(max_decided_scn={val:1755161719775179002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1755161287778691002, v:0}, keep_alive_scn_:{val:1755161287778691002, v:0}, update_ts_:1755162216759845}) [2025-08-14 09:03:39.744693] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:37.746931] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:39.747120] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1997143][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:39.747150] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4389] timer task use too much time(end=1755162219747149, start=1755162217746928, delta=2000221) [2025-08-14 09:03:39.740672] WDIAG stat_cleanup (debug.c:35) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=996716][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:39.743879] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23525][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=975188, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=975182], seq:[0] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=975220, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=975215], seq:[0, 1] [2025-08-14 09:03:36.731980] EDIAG [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=961103][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2000388, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2000369, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=1, APPEND_END=14], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1f9d7fcf 0x99b8be2 0x1f981833 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [567]OB_MALLOC COST TOO MUCH TIME, cost_time=981252, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=37, WASH_BLOCK_END=949465, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=13878], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:36.732277] EDIAG [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1034650][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3028479, owner: ObLog, click_count: 7, time dist:[FORMAT_END=2024601, ObMalloc#start=1, ObMalloc#end=22574, ObMalloc#start=24, ObMalloc#end=981275, ALLOC_END=3, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0xfa06846 0xfa05f62 0xfa05c84 0xfa05a50 0x1b51bd85 0x1b51b9c4 0x97a8d82 0x97a88d2 0x97a7481 0x97a6a43 0x97a42c5 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:39.754605] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:39.744725] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:39.757590] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:38.782271] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20955][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:39.762069] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:39.762197] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:39.763260] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=2017480] temporary file module use no blocks [2025-08-14 09:03:39.744743] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=57] finish to resize log external storage handler(current_ts=1755162218756349, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-08-14 09:03:39.763731] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=18955] snapshot_traffic_info once [2025-08-14 09:03:36.732277] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1034650] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:39.765416] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [719][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=42][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571f9d70, caller:ob_table_lock_service.cpp:start:276, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE6$_1273, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa090, time_wheel:0x7f99375f7d10, thread_pool:0x7f99375f74f0, is_running:0, time_interval:10.00s, expected_run_time:09:03:42.332, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:39.765448] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [719][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:38.759523] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1005024, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1005013, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:38.759523] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:39.766093] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1005024][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:39.766115] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:39.756816] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=30] LogLoopThread round_cost_time(us)(round_cost_time=14999187) [2025-08-14 09:03:39.766323] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=9460][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:37.947247] EDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 1817311, owner: ObLog, click_count: 11, time dist:[ALLOC_BLOCK#start=7, ALLOC_BLOCK#end=3, FORMAT_END=1812564, ObMalloc#start=0, ObMalloc#end=50, ObMalloc#start=21, ObMalloc#end=47, ObMalloc#start=4612, ObMalloc#end=3, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x9757ced 0x97569df 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:38.727445] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=21][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1038356, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1034935, ObMalloc#start=0, ObMalloc#end=62, ObMalloc#start=21, ObMalloc#end=56, FREE_END=3282], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:39.767017] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=19551] PNIO [ratelimit] time: 1755162219767014, bytes: 0, bw: 0.000000 MB/s, add_ts: 1004277, add_bytes: 0 [2025-08-14 09:03:37.947247] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc49d8, timer:0x7f9970f40f80, task:0x564d6ef827d0, task_type:"N9oceanbase7storage20ObDDLCtrlSpeedHandle22RefreshSpeedHandleTaskE", scheduled_time:1755161615757725, pushed_time:0, delay:1000000}, running_task_token={this:0x7f9970fc5fb8, timer:0x7f9970f40f80, task:0x564d727182f0, task_type:"N9oceanbase5share16ObAshRefreshTaskE", scheduled_time:1755161611717613, pushed_time:1755161614757742, delay:120000000}, total_delay=602189520, threashold=1000000, tenant_id=500) [2025-08-14 09:03:39.767337] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=1817311][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc4b98, timer:0x7f9970f40f80, task:0x564d6d07c068, task_type:"N9oceanbase8observer16ObTenantDutyTaskE", scheduled_time:1755161617719061, pushed_time:0, delay:10000000}, running_task_token={this:0x7f9970fc5fb8, timer:0x7f9970f40f80, task:0x564d727182f0, task_type:"N9oceanbase5share16ObAshRefreshTaskE", scheduled_time:1755161611717613, pushed_time:1755161614757742, delay:120000000}, total_delay=602048275, threashold=1000000, tenant_id=500) [2025-08-14 09:03:39.767399] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=1038356] gc stale ls task succ [2025-08-14 09:03:38.782555] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26209][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:39.767445] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=983389] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:39.767486] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=414] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:39.767580] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:38.767441] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=16] current monitor number(seq_=-1) [2025-08-14 09:03:39.768150] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:38.777954] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=32] gc stale ls task succ [2025-08-14 09:03:39.768990] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=2016431] temporary file module use no blocks [2025-08-14 09:03:38.787643] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:39.769075] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162219769075, start=1755162218787641, delta=981434) [2025-08-14 09:03:39.769105] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=975269] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:38.744170] EDIAG [RS] get_valid_sts_after (ob_tenant_info_loader.cpp:454) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1023988, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1023984, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1001d942 0x996d843 0x996d15a 0x996cfd8 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:38.889644] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:39.769516] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162219769516, start=1755162218889642, delta=879874) [2025-08-14 09:03:39.749638] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16822][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:39.769731] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1023988] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:03:39.769755] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:03:39.769769] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:39.769781] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:39.770260] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=2017893] temporary file module use no blocks [2025-08-14 09:03:39.776298] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=14363] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:39.778317] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=989142] gc stale ls task succ [2025-08-14 09:03:39.780327] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=1003869] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:03:39.782297] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=31] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:39.783549] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=12321] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:03:39.787676] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=1005461] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:03:39.787676] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=999560] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:03:39.797166] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [880][T1002_ArcSender][T1002][YB42AC4D0007-00063C495EB99C16-0-0] [lt=202] ObArchiveSender is running(thread_index=5) [2025-08-14 09:03:39.834953] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:39.869858] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:39.944420] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:39.969214] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:39.969871] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:39.997530] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:40.029412] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:40.043841] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=782423] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:40.069941] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:40.170001] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:40.213498] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:40.236039] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:40.267678] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc50d8, timer:0x7f9970f40f80, task:0x564d6d07c190, task_type:"N9oceanbase8observer8ObServer24ObRefreshCpuFreqTimeTaskE", scheduled_time:1755161617737100, pushed_time:0, delay:10000000}, running_task_token={this:0x7f9970fc5fb8, timer:0x7f9970f40f80, task:0x564d727182f0, task_type:"N9oceanbase5share16ObAshRefreshTaskE", scheduled_time:1755161611717613, pushed_time:1755161614757742, delay:120000000}, total_delay=602530577, threashold=1000000, tenant_id=500) [2025-08-14 09:03:40.272261] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:40.372335] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:40.664677] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:40.721251] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=20010] gc stale ls task succ [2025-08-14 09:03:39.787875] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=995797] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:03:40.472419] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:40.723411] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=250609][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:40.737578] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19874] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:40.737973] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=950519] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:40.723413] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:03:40.738217] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14776] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:03:40.723427] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:03:40.738585] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [316][pnio1][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162193865599", warnings:[]}, tenant_id_=1, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:03:40.738600] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=15148] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:03:40.739378] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=275] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:39.776317] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=38204] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:40.739968] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=962193] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:40.739984] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:40.739996] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:40.740006] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:40.740017] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:36.762273] EDIAG [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=1048897][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4006819, owner: ObLog, click_count: 3, time dist:[FORMAT_END=4006818, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xff22be4 0x97a9cd7 0x97a9238 0x97a9307 0x97a8fe2 0x97a8cc7 0x97a88d2 0x97a7481 0x97a6a43 0x97a42c5 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:36.762290] EDIAG [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1043745][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4007122, owner: ObLog, click_count: 3, time dist:[FORMAT_END=4007121, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xff22be4 0x97a9cd7 0x97a9238 0x97a9307 0x97a8fe2 0x97a8cc7 0x97a88d2 0x97a7481 0x97a6a43 0x97a42c5 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:40.740996] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:40.741339] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=831] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:40.741429] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:40.755321] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:40.759348] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:40.723441] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:03:40.768455] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17101][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:36.762273] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=1048897] get_min_unreplayed_log_info(lsn={lsn:1604088821}, scn={val:1755100948296325001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873479, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:12492179, ret_code_:0}}, next_to_submit_lsn_:{lsn:1604088821}, next_to_submit_scn_:{val:1755100948296325001, v:0}, base_lsn_:{lsn:1409200128}, base_scn_:{val:1753840802540914000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:1604088821}, end_lsn:{lsn:1604088821}, read_buf:{buf_len_:3747840, buf_:0x7f98f6a05000}, block_size:67104768, log_storage_:0x7f990bfebbf0, log_storage_:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:42, read_io_size:147709952, read_disk_cost_ts:1061450, start_lsn:{lsn:1409200128}}}}}}) [2025-08-14 09:03:40.775193] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1049) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=4006819] get_min_unreplayed_log_info(lsn={lsn:1604088821}, scn={val:1755100948296325001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873479, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:12492179, ret_code_:0}}, next_to_submit_lsn_:{lsn:1604088821}, next_to_submit_scn_:{val:1755100948296325001, v:0}, base_lsn_:{lsn:1409200128}, base_scn_:{val:1753840802540914000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:1604088821}, end_lsn:{lsn:1604088821}, read_buf:{buf_len_:3747840, buf_:0x7f98f6a05000}, block_size:67104768, log_storage_:0x7f990bfebbf0, log_storage_:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:42, read_io_size:147709952, read_disk_cost_ts:1061450, start_lsn:{lsn:1409200128}}}}}}, ret=0) [2025-08-14 09:03:40.741380] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] [sql nio session](*s={this:0x7f98f72d0030, session_id:3221589415, trace_id:YB42AC4D0007-00063C4965FA43DC-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:245}, err:0, last_decode_time:1755161751714366, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1662, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:03:40.775268] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=33835] [sql nio session](*s={this:0x7f98f72d5030, session_id:3221590200, trace_id:YB42AC4D0007-00063C4965FA43E0-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:248}, err:0, last_decode_time:1755161776777167, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1593, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:03:36.762290] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1043745] get_min_unreplayed_log_info(lsn={lsn:8461098908}, scn={val:1755100947600130001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161920778323, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:23084275, ret_code_:0}}, next_to_submit_lsn_:{lsn:8461098908}, next_to_submit_scn_:{val:1755100947600130001, v:0}, base_lsn_:{lsn:7515734016}, base_scn_:{val:1753840867171901045, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6e05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:8461098908}, end_lsn:{lsn:8461098908}, read_buf:{buf_len_:3747840, buf_:0x7f98f6e05000}, block_size:67104768, log_storage_:0x7f99297a3bf0, log_storage_:{log_tail:{lsn:8461098908}, readable_log_tail:{lsn:8461098908}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8455200768}, min_scn:{val:1755160384851871002, v:0}, curr_block_id:126, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:215, min_block_id:93, max_block_id:127, curr_writable_block_id:126}, logical_block_size_:67104768, curr_block_writable_size_:61206628, block_header_serialize_buf_:0x7f99297a48b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:198, read_io_size:699822080, read_disk_cost_ts:4178035, start_lsn:{lsn:7515734016}}}}}}) [2025-08-14 09:03:40.775504] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1049) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=4007122] get_min_unreplayed_log_info(lsn={lsn:8461098908}, scn={val:1755100947600130001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161920778323, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:23084275, ret_code_:0}}, next_to_submit_lsn_:{lsn:8461098908}, next_to_submit_scn_:{val:1755100947600130001, v:0}, base_lsn_:{lsn:7515734016}, base_scn_:{val:1753840867171901045, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6e05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:8461098908}, end_lsn:{lsn:8461098908}, read_buf:{buf_len_:3747840, buf_:0x7f98f6e05000}, block_size:67104768, log_storage_:0x7f99297a3bf0, log_storage_:{log_tail:{lsn:8461098908}, readable_log_tail:{lsn:8461098908}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8455200768}, min_scn:{val:1755160384851871002, v:0}, curr_block_id:126, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:215, min_block_id:93, max_block_id:127, curr_writable_block_id:126}, logical_block_size_:67104768, curr_block_writable_size_:61206628, block_header_serialize_buf_:0x7f99297a48b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:198, read_io_size:699822080, read_disk_cost_ts:4178035, start_lsn:{lsn:7515734016}}}}}}, ret=0) [2025-08-14 09:03:40.738296] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=80] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623113209791, flush_io_finished_ret_:0, flush_io_finished_round_:26315, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:03:40.720564] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4700] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:40.776794] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=56143][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:36.778260] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=29] get_min_unreplayed_log_info(lsn={lsn:5282737655}, scn={val:1755100947825033001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161624848121, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:13063726, ret_code_:0}}, next_to_submit_lsn_:{lsn:5282737655}, next_to_submit_scn_:{val:1755100947825033001, v:0}, base_lsn_:{lsn:4831543296}, base_scn_:{val:1755063445733337001, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7605000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:5282737655}, end_lsn:{lsn:5282737655}, read_buf:{buf_len_:3747840, buf_:0x7f98f7605000}, block_size:67104768, log_storage_:0x7f993e529bf0, log_storage_:{log_tail:{lsn:5282737655}, readable_log_tail:{lsn:5282737655}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:5234171904}, min_scn:{val:1755152244802535004, v:0}, curr_block_id:78, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:206, min_block_id:42, max_block_id:79, curr_writable_block_id:78}, logical_block_size_:67104768, curr_block_writable_size_:18539017, block_header_serialize_buf_:0x7f993e52a8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:85, read_io_size:301367296, read_disk_cost_ts:2364604, start_lsn:{lsn:4831543296}}}}}}) [2025-08-14 09:03:40.777060] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=35635] [sql nio session](*s={this:0x7f98f72ab030, session_id:3221589435, trace_id:YB42AC4D0007-00063C4965DA1F35-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:246}, err:0, last_decode_time:1755161762721798, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:2456, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:03:40.777145] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=85] [sql nio session](*s={this:0x7f98f72a9030, session_id:3221590528, trace_id:YB42AC4D0007-00063C4965DA1F34-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:250}, err:0, last_decode_time:1755161762715415, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:205, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:03:40.777155] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=10] [sql nio session](*s={this:0x7f98f72b5030, session_id:3221590961, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:10, sql_initiative_shutdown:false, reader:{fd:278}, err:5, last_decode_time:1755162075745307, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:03:40.777164] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=9] [sql nio session](*s={this:0x7f98f72a6030, session_id:3221590972, trace_id:YB42AC4D0007-00063C4965DA1F38-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:264}, err:0, last_decode_time:1755162176744872, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:221, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:03:40.777276] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:40.782271] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=17] gc stale ls task succ [2025-08-14 09:03:40.782314] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:40.823472] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:40.835574] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:40.851464] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=754401][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:40.873758] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:40.879731] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=22][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:40.882890] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:40.923522] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:40.951490] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:40.973782] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:41.002322] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:41.020644] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=37][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:03:41.021392] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=22][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:03:41.023592] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:41.059748] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=26] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:41.124247] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:41.224310] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:41.238080] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:41.265269] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:41.275318] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:41.324389] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:41.424430] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:41.524488] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:41.626255] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:41.720272] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:41.721434] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=17] gc stale ls task succ [2025-08-14 09:03:41.726287] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:40.784053] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=25] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:03:40.774743] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=51221] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:03:41.727474] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=951290] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623113209583, flush_io_finished_ret_:0, flush_io_finished_round_:26348, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:03:40.775151] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=581] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:03:36.778260] EDIAG [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3992719, owner: ObLog, click_count: 5, time dist:[FORMAT_END=3992715, ALLOC_END=4, APPEND_END=0, ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=87], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xff22be4 0x97a9cd7 0x97a9238 0x97a9307 0x97a8fe2 0x986bc46 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:40.758735] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18066][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:41.727981] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=3992719] finish update min start scn(max_decided_scn={val:1755161547738516001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1755161493768447002, v:0}, keep_alive_scn_:{val:1755161493768447002, v:0}, update_ts_:1755162221727972}) [2025-08-14 09:03:41.729612] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=37] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:41.737306] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=2722][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:41.738311] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=21] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:41.739015] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:41.739060] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:41.739600] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=967749] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:41.742292] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:41.742613] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:41.755253] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6385] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:41.757518] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=168] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:41.742427] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [412][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=17219] [sql nio session](*s={this:0x7f98f733a030, session_id:3221590255, trace_id:YB42AC4D0007-00063C49651A3232-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:257}, err:0, last_decode_time:1755161952736711, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1716, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:03:41.761110] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=426] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:41.761133] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=496] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:41.759356] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=9562] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:41.759415] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9718] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:37.758735] EDIAG [ELECT] advance_ballot_number_and_reset_related_states_ (election_proposer.cpp:210) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3996711, owner: ObLog, click_count: 5, time dist:[FORMAT_END=3995180, ObMalloc#start=0, ObMalloc#end=1521, ALLOC_END=1, APPEND_END=9], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9d4e62 0xf9cf0e7 0xf9d3ec1 0xfa09ca2 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:41.759463] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:41.759813] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:40.767631] EDIAG [RPC] handle_resp (ob_poc_rpc_proxy.cpp:192) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1000347, owner: ObLog, click_count: 8, time dist:[FORMAT_END=996086, ObMalloc#start=1, ObMalloc#end=89, ObMalloc#start=25, ObMalloc#end=37, ObMalloc#start=3516, ObMalloc#end=76, FREE_END=517], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x27a861af 0x97bcba5 0x97bc4ca 0x27ad7f11 0x97b81a7 0x27a8a6d6 0x97b7ba0 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:40.779361] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2791][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162220776542], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:41.761044] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:41.772403] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=499] gc stale ls task succ [2025-08-14 09:03:41.761310] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9012] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:41.772505] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11176][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:41.773374] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [316][pnio1][T1][Y0-0000000000000000-0-0] [lt=701][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162196676151", warnings:[]}, tenant_id_=1, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:03:41.773636] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [316][pnio1][T1][Y0-0000000000000000-0-0] [lt=253][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162201781545", warnings:[]}, tenant_id_=1, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:03:41.773662] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [316][pnio1][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1002, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162208860757", warnings:[]}, tenant_id_=1002, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:03:41.773685] WDIAG pkts_handle_req_queue (pkts_post.h:51) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:41.773804] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=325] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:41.773854] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=247] current monitor number(seq_=-1) [2025-08-14 09:03:40.774778] EDIAG [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2000388][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1001809, owner: ObLog, click_count: 5, time dist:[FORMAT_END=951376, ObMalloc#start=1, ObMalloc#end=50427, ALLOC_END=2, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1f9ebc06 0x97c9052 0x1f9819be 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:41.761100] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=1023329] blacklist_loop exec finished(cost_time=17, is_enabled=true, send_cnt=0) [2025-08-14 09:03:41.777835] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:40.774778] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2000388] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:03:41.759350] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=8144] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:41.778170] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=1001809] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:03:41.778215] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:03:41.778249] INFO [COMMON] print_io_status (ob_io_manager.cpp:2639) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:0, norm_iops:0, bw:141, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:1013126, total:2970085) [2025-08-14 09:03:41.778276] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=1048] PNIO [ratelimit] time: 1755162221778274, bytes: 0, bw: 0.000000 MB/s, add_ts: 1004817, add_bytes: 0 [2025-08-14 09:03:41.778557] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=980705][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162220776542], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:39.756778] WDIAG [COMMON] estimate (ob_io_define.cpp:1996) [421][IO_HEALTH0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] io result wait too long(result_={is_inited_:true, is_finished_:true, is_canceled_:false, has_estimated_:true, complete_size_:4096, offset_:0, size_:4096, timeout_us_:5000000, result_ref_cnt_:1, out_ref_cnt_:1, flag_:{mode:"READ", group_id_:0, func_type_:0, wait_event_id_:1, is_sync_:false, is_unlimited_:true, is_detect_:true, is_write_through_:false, is_sealed_:true, is_time_detect_:true, need_close_dev_and_fd_:false, reserved_:0}, ret_code_:{io_ret_:0, fs_errno_:0}, tenant_id_:500, tenant_io_mgr_:{ptr:0x7f99667dc030}, user_data_buf_:null, buf_:null, io_callback_:null, time_log:{begin_ts:1755162215766304, enqueue_used:-1, dequeue_used:-1, submit_used:1755162217723263, return_used:1013126, callback_enqueue_used:-1, callback_dequeue_used:-1, callback_finish_used:-1, end_used:1755162218773075}}, result_delay=3006771) [2025-08-14 09:03:41.783279] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=24] gc stale ls task succ [2025-08-14 09:03:41.828308] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:41.837263] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:41.879461] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:41.893243] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [571][T1_ArcSender][T1][YB42AC4D0007-00063C495BA99C16-0-0] [lt=3847991][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:03:41.930272] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:41.957649] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:41.961156] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:41.979421] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:42.030294] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:42.061490] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=21] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:42.131260] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:42.231322] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:42.239527] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:42.240003] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:42.269299] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:42.277540] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:42.331381] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:42.431399] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:42.531456] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:42.631514] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:42.722407] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:42.722456] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:03:42.727778] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=941954] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:03:42.729257] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:41.783263] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:42.731068] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=946372][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:42.731971] INFO [PALF] check_parent_health (log_config_mgr.cpp:2875) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=13] re_register_parent reason(palf_id=1, self="172.77.0.7:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1755162208736282, last_first_register_time_us=1755158728414267, register_time_us=1755162208736282, curr_time_us=1755162221730192) [2025-08-14 09:03:42.738507] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=26] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:42.740739] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:42.740861] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:42.742314] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:42.745446] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:42.745811] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:42.745856] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:42.745883] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [411][sql_nio4][T0][Y0-0000000000000000-0-0] [lt=14877] [sql nio session](*s={this:0x7f98f72b9030, session_id:3221577643, trace_id:YB42AC4D0007-00063C49634A0940-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:270}, err:0, last_decode_time:1755162188774494, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1468, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:03:42.731589] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=71] get wrs ts(ls_id={id:1}, delta=472023656, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:42.751394] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2022] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:42.751439] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=2064] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:42.751902] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:42.754540] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1993] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:42.754538] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=37659] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:03:42.754567] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=21] temporary file module use no blocks [2025-08-14 09:03:42.754585] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=15] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:03:42.754598] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:03:42.754610] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623123215782, flush_io_finished_ret_:0, flush_io_finished_round_:26316, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:03:42.754643] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=38] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:03:42.754656] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] temporary file module use no blocks [2025-08-14 09:03:42.754662] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:03:42.754671] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:03:42.754680] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623124167282, flush_io_finished_ret_:0, flush_io_finished_round_:26349, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:03:42.761278] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:42.761509] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=15] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:42.761549] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=606] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:42.732806] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=51] get wrs ts(ls_id={id:1}, delta=501999932, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:42.762942] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2888] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:42.762986] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:42.763012] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:42.764141] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:41.778586] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=24] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162220776542]}) [2025-08-14 09:03:42.727884] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=951238] [COMPACTION DAG STATUS] (start_time=1755162211744397, end_time=1755162222727883, dag_status={}) [2025-08-14 09:03:42.762644] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=29788] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:41.773707] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] [COMPACTION DAG STATUS] (start_time=1755162210757567, end_time=1755162221765193, dag_status={}) [2025-08-14 09:03:42.768832] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=993618] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:42.768849] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:42.768867] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=17] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:42.768876] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:42.768885] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:42.768894] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:42.768902] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:42.768911] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:42.768938] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=19] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:42.768950] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:03:42.768959] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:03:42.749294] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17671] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:40.775458] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=226] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1001, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:8571658, max_lsn:7323687369, max_scn:{val:1755161749741193002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:7278610047}, reuse_lsn:{lsn:7323686131}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:7281744329}}, last_submit_lsn:{lsn:7323686131}, last_submit_end_lsn:{lsn:7323687369}, last_submit_log_id:8571658, last_submit_log_pid:49, max_flushed_lsn:{lsn:7323686009}, max_flushed_end_lsn:{lsn:7323686131}, max_flushed_log_pid:49, committed_end_lsn:{lsn:7323686009}, last_slide_log_id:8571656, last_slide_scn:{val:1755161747742105001, v:0}, last_slide_lsn:{lsn:7323685887}, last_slide_end_lsn:{lsn:7323686009}, last_slide_log_pid:49, last_slide_log_accum_checksum:2505458373, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f990112a530}, [StateMgr]={this:0x7f990112e9b0, self:"172.77.0.7:2882", palf_id:1001, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"PENDING", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:49}, leader:"0.0.0.0:0", leader_epoch:-1, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:7323687369}, scan_disk_log_finished:true, last_check_start_id:8571657, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161767744317, palf_role_change_cb:0x7f9901135470, allow_vote:true}, [ConfigMgr]={palf_id:1001, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:49, prev:{version:1, config:{config_version:{proposal_id:48, config_seq:47}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:49, config_seq:48}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_mode_pid:49}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_end_lsn:{lsn:7323685240}, prev_mode_pid:49}, reconfig_barrier:{prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_end_lsn:{lsn:7323685240}, prev_mode_pid:49}, persistent_config_version:{proposal_id:49, config_seq:48}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755161739755312, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755161739755312, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f990112bc70}, [ModeMgr]={palf_id:1001, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, accepted_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1001, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1753069671378640001, v:0}, base_lsn_for_block_gc:{lsn:7247314944}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:49}, log_config_meta:{version:3, proposal_id:49, prev:{version:1, config:{config_version:{proposal_id:48, config_seq:47}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:49, config_seq:48}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_mode_pid:49}, log_snapshot_meta:{version:2, base_lsn:{lsn:7247314944}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1753069671378640000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}}, log_meta_storage:{log_tail:{lsn:765952}, readable_log_tail:{lsn:765952}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7381524480}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/meta", dir_fd:232, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66338816, block_header_serialize_buf_:0x7f9901130bb8, flashback_version:0}, log_storage:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, palf_epoch:2, last_purge_throttling_ts:1755161769777679, this:0x7f990112f0f0}, [Reconfirm]={palf_id:1001, self:"172.77.0.7:2882", state:"WAITING_LOG_FLUSHED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:1755161769777679, this:0x7f990112eb30}) [2025-08-14 09:03:40.775458] EDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=226][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1992419, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1992391, ObMalloc#start=2, ObMalloc#end=8, ALLOC_END=6, APPEND_END=10], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0xfc0e73d 0x993f560 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:42.772469] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=11] gc stale ls task succ [2025-08-14 09:03:42.775538] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=13] current monitor number(seq_=-1) [2025-08-14 09:03:42.775589] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:42.777948] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=17] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:42.778248] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=18786] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:42.779159] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:42.783213] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=16] PNIO [ratelimit] time: 1755162222783209, bytes: 0, bw: 0.000000 MB/s, add_ts: 1004935, add_bytes: 0 [2025-08-14 09:03:42.845985] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:42.958225] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:42.980180] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:43.063385] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=27] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:43.186229] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc43b8, timer:0x7f9970f40f80, task:0x564d6d07a448, task_type:"N9oceanbase5share17ObServerTraceTaskE", scheduled_time:1755161621733806, pushed_time:0, delay:8000000}, running_task_token={this:0x7f9970fc5fb8, timer:0x7f9970f40f80, task:0x564d727182f0, task_type:"N9oceanbase5share16ObAshRefreshTaskE", scheduled_time:1755161611717613, pushed_time:1755161614757742, delay:120000000}, total_delay=601452422, threashold=1000000, tenant_id=500) [2025-08-14 09:03:43.249927] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:43.277734] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:43.512058] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:43.676189] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:43.720263] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=35] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:43.723277] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=16] gc stale ls task succ [2025-08-14 09:03:43.728274] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=25] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:03:43.729258] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:43.738101] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=37286] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:03:43.738182] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=72] temporary file module use no blocks [2025-08-14 09:03:43.738196] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:03:43.738215] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=18] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:03:43.738245] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=28] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623123215730, flush_io_finished_ret_:0, flush_io_finished_round_:26338, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:03:43.738255] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:43.738858] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:43.739285] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=27] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:43.739329] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:43.739766] INFO [STORAGE] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:123) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20565] wead read timestamp is too old(timestamp={val:1755161749741193000, v:0}, gts_scn={val:1755161762721798002, v:0}, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1755161749741193000, v:0}}) [2025-08-14 09:03:42.783341] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=13] gc stale ls task succ [2025-08-14 09:03:43.249209] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:43.739857] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] timer task use too much time(end=1755162223739856, start=1755162223249207, delta=490649) [2025-08-14 09:03:43.740967] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=33] get_max_applied_scn(scn={val:1755161744727824001, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:49, palf_committed_end_lsn_:{lsn:7323686009}, palf_committed_end_scn_:{val:1755161747742105001, v:0}, last_check_scn_:{val:1755161749741193000, v:0}, max_applied_cb_scn_:{val:1755161744727824001, v:0}}) [2025-08-14 09:03:43.741006] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26] get_min_unreplayed_log_info(lsn={lsn:7323685240}, scn={val:1755100946796590002, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873485, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:8946764, ret_code_:0}}, next_to_submit_lsn_:{lsn:7323685240}, next_to_submit_scn_:{val:1755100946796590002, v:0}, base_lsn_:{lsn:7180210176}, base_scn_:{val:1753840803166611000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:7323685240}, end_lsn:{lsn:7323685240}, read_buf:{buf_len_:3747840, buf_:0x7f98f7a05000}, block_size:67104768, log_storage_:0x7f9901131bf0, log_storage_:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:28, read_io_size:98508800, read_disk_cost_ts:645339, start_lsn:{lsn:7180210176}}}}}}) [2025-08-14 09:03:43.741615] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=81] get wrs ts(ls_id={id:1001}, delta=479011141, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:43.741639] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:39.771355] EDIAG [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=2961452][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3965253, owner: ObLog, click_count: 5, time dist:[FORMAT_END=3965243, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b51df0f 0x986bcde 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:43.741857] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=46][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:39.771355] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=2961452] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136829134139001, v:0}, clog_base_lsn:{lsn:8186781696}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1753840867372367034, v:0}, all_id_meta:{id_meta:[{limited_id:1755161729775179000, latest_log_ts:{val:1755161719775179001, v:0}}, {limited_id:18000001, latest_log_ts:{val:1755157440846556001, v:0}}, {limited_id:60000001, latest_log_ts:{val:1755135657266496001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:46, palf_env_:0x7f99321fa030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-08-14 09:03:43.742597] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1992419] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [2025-08-14 09:03:41.728669] EDIAG [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2020081, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2013813, ALLOC_END=6267, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b51df0f 0x986bcde 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:42.767252] EDIAG [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3028479][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1945099, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1945097, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x9c6174d 0xfd347d6 0xfd344b7 0xfd34254 0x1c029398 0x97a1ad0 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [381]OB_MALLOC COST TOO MUCH TIME, cost_time=970574, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=970566], seq:[0] [2025-08-14 09:03:43.742625] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=24] current log_task status(palf_id=1001, self="172.77.0.7:2882", log_id=8571657, log_task={header:{begin_lsn:{lsn:7323686009}, end_lsn:{lsn:7323686131}, log_id:8571657, min_scn:{val:1755161749741193000, v:0}, max_scn:{val:1755161749741193000, v:0}, data_len:66, proposal_id:49, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:7323686009}, data_checksum:2222604892, accum_checksum:1531236791, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1755161749745909, freeze_ts:1755161749745914, submit_ts:1755161749745940, flushed_ts:1755161785758706, gen_to_freeze cost time:5, gen_to_submit cost time:31, submit_to_flush cost time:36012766}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [2025-08-14 09:03:43.741679] INFO [STORAGE.TRANS] amend (ob_tenant_weak_read_server_version_mgr.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25] [WRS] tenant has no valid partition on current server, generate weak read snapshot version based on current timestamp(tenant_id=1002, epoch_tstamp_=1755162214720883, total_part_count_=2, valid_inner_part_count_=0, valid_user_part_count_=0, version_={val:1755161758221798000, v:0}, base_version_when_no_valid_partition={val:1755161758221798000, v:0}) [2025-08-14 09:03:43.746418] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:41.728669] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=20] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136826952869001, v:0}, clog_base_lsn:{lsn:5099962368}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1755071646732636000, v:0}, all_id_meta:{id_meta:[{limited_id:1755161555795565000, latest_log_ts:{val:1755161545796642001, v:0}}, {limited_id:22000001, latest_log_ts:{val:1755135645787806003, v:0}}, {limited_id:60000001, latest_log_ts:{val:1755135646400653001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:50, palf_env_:0x7f9950bf4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-08-14 09:03:43.746452] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:43.753745] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=33] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:43.766778] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:43.766803] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:43.780266] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=10] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:43.785303] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:43.863890] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [381]OB_MALLOC COST TOO MUCH TIME, cost_time=992687, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=51, WASH_BLOCK_END=103, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=970587], seq:[0, 1, 2, 3, 4] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=975243, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=975236], seq:[0] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=982882, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=49, WASH_BLOCK_END=7569, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=975259], seq:[0, 1, 2, 3, 4] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=987236, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=987205, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=987282, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=987258, WASH_BLOCK_END=24], seq:[0, 1, 2] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=987314, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=987279, ALLOC_CHUNK_END=13], seq:[0, 1, 2, 3, 4, 5] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=987334, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=987325, WASH_BLOCK_END=9], seq:[0, 1, 2] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=987313, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=987285, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=987330, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=987321, WASH_BLOCK_END=9], seq:[0, 1, 2] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=987320, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=987284, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=987338, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=987330, WASH_BLOCK_END=8], seq:[0, 1, 2] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=987311, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=987283, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=987329, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=987320, WASH_BLOCK_END=9], seq:[0, 1, 2] [815]OB_MALLOC COST TOO MUCH TIME, cost_time=986905, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=986874, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [815]OB_MALLOC COST TOO MUCH TIME, cost_time=986950, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=986923, WASH_BLOCK_END=26], seq:[0, 1, 2] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=987615, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=987584, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=987658, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=987630, WASH_BLOCK_END=28], seq:[0, 1, 2] [2025-08-14 09:03:42.840922] EDIAG [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1914495, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1913498, ObMalloc#start=1, ObMalloc#end=850, ObMalloc#start=32, ObMalloc#end=60, FREE_END=54], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1b52e1b7 0x996d441 0x996cfd8 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [776]OB_MALLOC COST TOO MUCH TIME, cost_time=979438, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=979400, ALLOC_CHUNK_END=14], seq:[0, 1, 2, 3, 4, 5] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=979467, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=979455, WASH_BLOCK_END=12], seq:[0, 1, 2] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=987444, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=987411, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=987493, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=987463, WASH_BLOCK_END=30], seq:[0, 1, 2] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=974964, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=974929, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=974992, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=974976, WASH_BLOCK_END=16], seq:[0, 1, 2] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=980517, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=980486, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=980549, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=980531, WASH_BLOCK_END=18], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=970376, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=970344, ALLOC_CHUNK_END=7], seq:[0, 1, 2, 3, 4, 5] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=970419, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=970392, WASH_BLOCK_END=27], seq:[0, 1, 2] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=914462, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=914431, ALLOC_CHUNK_END=7], seq:[0, 1, 2, 3, 4, 5] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=914481, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=914472, WASH_BLOCK_END=9], seq:[0, 1, 2] [466]OB_MALLOC COST TOO MUCH TIME, cost_time=875391, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=875357, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [466]OB_MALLOC COST TOO MUCH TIME, cost_time=875412, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=875402, WASH_BLOCK_END=10], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=697594, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=697562, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3, 4, 5] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=697614, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=697604, WASH_BLOCK_END=10], seq:[0, 1, 2] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=509673, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=509643, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=509692, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=509682, WASH_BLOCK_END=10], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=171136, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=171095, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=171162, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=171146, WASH_BLOCK_END=16], seq:[0, 1, 2] [2025-08-14 09:03:43.882343] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:43.899354] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:43.958863] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:43.985320] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:44.250905] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:44.302970] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:42.767252] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3028479] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=674038796, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=253217, succ_cluster_heartbeat_count=2213, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161000606550000, v:0}, local_cluster_delta=1221170762, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:03:43.745455] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=5461] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=503970274, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=252364, succ_cluster_heartbeat_count=1905, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161003593738002, v:0}, local_cluster_delta=1220151715, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:03:43.743196] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=1508] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=465521395, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=251767, succ_cluster_heartbeat_count=1502, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161006924036004, v:0}, local_cluster_delta=1216819157, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:03:44.720434] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:43.743518] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=984244] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=72, get_gts_with_stc_cnt=1, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:03:44.737746] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=992724] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162220776542]}) [2025-08-14 09:03:44.738825] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:43.754603] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:44.738997] WDIAG ev_invoke_pending (ev.c:2136) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=982906][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:43.783406] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=955015] gc stale ls task succ [2025-08-14 09:03:43.751849] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=7184] log compressor cycles once. (ret=0, cost_time=1974116, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25896906752, fast_delete_log_mode=false) [2025-08-14 09:03:44.741277] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5789] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:44.750754] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:44.751084] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=23] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:44.751899] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=990093][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:44.753107] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=992648][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:44.739409] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:44.753505] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=843] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:44.754329] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=3965253] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:44.754405] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=19] tx gc loop thread is running(MTL_ID()=1001) [2025-08-14 09:03:44.758271] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:44.760533] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:44.754414] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=8] try gc retain ctx [2025-08-14 09:03:44.761027] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=6602] try to retry start_working [2025-08-14 09:03:44.761409] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:43.772689] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=19] gc stale ls task succ [2025-08-14 09:03:44.761741] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:44.064443] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=19] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:44.729262] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:44.729393] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=25] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:03:44.763659] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34343][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:44.723338] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=25] gc stale ls task succ [2025-08-14 09:03:44.737485] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:44.763698] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26173][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:44.738855] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=32] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:44.763713] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24819][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:44.763736] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=40] LogLoopThread round_cost_time(us)(round_cost_time=19983281) [2025-08-14 09:03:44.753411] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13980][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:44.758669] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=329][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:44.763804] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=5124][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:44.683477] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=2867322][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a4658, timer:0x7f9956bffee0, task:0x7f9969bb3038, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr12TabletGCTaskE", scheduled_time:1755161923366542, pushed_time:0, delay:20000}, running_task_token={this:0x7f994e1a4498, timer:0x7f9956bffee0, task:0x7f9969bb3010, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr11TableGCTaskE", scheduled_time:1755161923356444, pushed_time:1755161923356659, delay:20000}, total_delay=301316934, threashold=1000000, tenant_id=1) [2025-08-14 09:03:44.591892] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:44.764381] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] timer task use too much time(end=1755162224764380, start=1755162224591890, delta=172490) [2025-08-14 09:03:43.886244] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:44.764426] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162224764425, start=1755162223886243, delta=878182) [2025-08-14 09:03:43.846964] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:44.764486] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] timer task use too much time(end=1755162224764483, start=1755162223846961, delta=917522) [2025-08-14 09:03:44.252710] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:44.764517] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162224764517, start=1755162224252708, delta=511809) [2025-08-14 09:03:44.764553] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=982703][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:44.764591] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=2799] temporary file module use no blocks [2025-08-14 09:03:44.764603] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:03:44.764616] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:03:44.764630] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623126194477, flush_io_finished_ret_:0, flush_io_finished_round_:26318, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:03:43.764421] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:44.754868] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=2020081] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:44.765382] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10491] tx gc loop thread is running(MTL_ID()=1) [2025-08-14 09:03:44.765391] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] try gc retain ctx [2025-08-14 09:03:44.765399] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] try to retry start_working [2025-08-14 09:03:44.763772] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:44.767163] WDIAG stat_cleanup (debug.c:35) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:44.767165] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=3378] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:03:44.767218] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=44] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:8607757, max_lsn:1604090331, max_scn:{val:1755161756730847001, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:1556737504}, reuse_lsn:{lsn:1604090331}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:1562147291}}, last_submit_lsn:{lsn:1604090203}, last_submit_end_lsn:{lsn:1604090331}, last_submit_log_id:8607757, last_submit_log_pid:50, max_flushed_lsn:{lsn:1604090203}, max_flushed_end_lsn:{lsn:1604090331}, max_flushed_log_pid:50, committed_end_lsn:{lsn:1604090090}, last_slide_log_id:8607755, last_slide_scn:{val:1755161749741193000, v:0}, last_slide_lsn:{lsn:1604089968}, last_slide_end_lsn:{lsn:1604090090}, last_slide_log_pid:50, last_slide_log_accum_checksum:1446576759, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f990bfe4530}, [StateMgr]={this:0x7f990bfe89b0, self:"172.77.0.7:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"PENDING", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:51}, leader:"0.0.0.0:0", leader_epoch:-1, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:1604090331}, scan_disk_log_finished:true, last_check_start_id:8607756, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161901828039, palf_role_change_cb:0x7f990bfef470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:50, prev:{version:1, config:{config_version:{proposal_id:49, config_seq:45}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:50, config_seq:46}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_mode_pid:50}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_end_lsn:{lsn:1604088821}, prev_mode_pid:50}, reconfig_barrier:{prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_end_lsn:{lsn:1604088821}, prev_mode_pid:50}, persistent_config_version:{proposal_id:50, config_seq:46}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755161738854289, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755161738854289, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f990bfe5c70}, [ModeMgr]={palf_id:1, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:50, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:1543409664}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:51}, log_config_meta:{version:3, proposal_id:50, prev:{version:1, config:{config_version:{proposal_id:49, config_seq:45}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:50, config_seq:46}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_mode_pid:50}, log_snapshot_meta:{version:2, base_lsn:{lsn:1543409664}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:712704}, readable_log_tail:{lsn:712704}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:1610514432}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/meta", dir_fd:225, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66392064, block_header_serialize_buf_:0x7f990bfeabb8, flashback_version:0}, log_storage:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1755161901828095, this:0x7f990bfe90f0}, [Reconfirm]={palf_id:1, self:"172.77.0.7:2882", state:"RECONFIRM_MODE_META", new_proposal_id:51, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"172.77.0.7:2882", majority_max_accept_pid:50, majority_max_lsn:{lsn:1604090331}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:1755161901828373, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f990bfe8b30}) [2025-08-14 09:03:44.767473] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=248] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161974728707, last_flushed_end_lsn:{lsn:1604090331}}]) [2025-08-14 09:03:44.767492] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16] current log_task status(palf_id=1, self="172.77.0.7:2882", log_id=8607756, log_task={header:{begin_lsn:{lsn:1604090090}, end_lsn:{lsn:1604090203}, log_id:8607756, min_scn:{val:1755161749741193001, v:0}, max_scn:{val:1755161749741193001, v:0}, data_len:57, proposal_id:50, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:1604090090}, data_checksum:2031283194, accum_checksum:759186326, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1755161750716165, freeze_ts:1755161750716173, submit_ts:1755161750716183, flushed_ts:1755161853765770, gen_to_freeze cost time:8, gen_to_submit cost time:18, submit_to_flush cost time:103049587}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161974728707, last_flushed_end_lsn:{lsn:1604090331}}]) [2025-08-14 09:03:44.767537] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:03:44.753708] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [854][T1002_ObXAHbWor][T1002][Y0-0000000000000000-0-0] [lt=37324] XA scheduler heartbeat task statistics(avg_time=977151) [2025-08-14 09:03:44.767615] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] get_max_applied_scn(scn={val:1755161547738516001, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:50, palf_committed_end_lsn_:{lsn:5282737655}, palf_committed_end_scn_:{val:1755161547738516001, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161547738516001, v:0}}) [2025-08-14 09:03:44.767628] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [414][sql_nio7][T1002][Y0-0000000000000000-0-0] [lt=1094] succeed to dispatch to tenant mysql queue(tenant_id=1002) [2025-08-14 09:03:44.767647] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:281) [414][sql_nio7][T1002][Y0-0000000000000000-0-0] [lt=19] mysql login queue(mysql_queue->queue_.size()=1) [2025-08-14 09:03:44.767641] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=19] get_min_unreplayed_log_info(lsn={lsn:5282737655}, scn={val:1755100947825033001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161624848121, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:13063726, ret_code_:0}}, next_to_submit_lsn_:{lsn:5282737655}, next_to_submit_scn_:{val:1755100947825033001, v:0}, base_lsn_:{lsn:4831543296}, base_scn_:{val:1755063445733337001, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7605000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:5282737655}, end_lsn:{lsn:5282737655}, read_buf:{buf_len_:3747840, buf_:0x7f98f7605000}, block_size:67104768, log_storage_:0x7f993e529bf0, log_storage_:{log_tail:{lsn:5282737655}, readable_log_tail:{lsn:5282737655}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:5234171904}, min_scn:{val:1755152244802535004, v:0}, curr_block_id:78, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:206, min_block_id:42, max_block_id:79, curr_writable_block_id:78}, logical_block_size_:67104768, curr_block_writable_size_:18539017, block_header_serialize_buf_:0x7f993e52a8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:85, read_io_size:301367296, read_disk_cost_ts:2364604, start_lsn:{lsn:4831543296}}}}}}) [2025-08-14 09:03:44.767708] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=64] finish update min start scn(max_decided_scn={val:1755161547738516001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1755161493768447002, v:0}, keep_alive_scn_:{val:1755161493768447002, v:0}, update_ts_:1755162221727972}) [2025-08-14 09:03:44.768441] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136826952869001, v:0}, clog_base_lsn:{lsn:5099962368}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1755071646732636000, v:0}, all_id_meta:{id_meta:[{limited_id:1755161555795565000, latest_log_ts:{val:1755161545796642001, v:0}}, {limited_id:22000001, latest_log_ts:{val:1755135645787806003, v:0}}, {limited_id:60000001, latest_log_ts:{val:1755135646400653001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:50, palf_env_:0x7f9950bf4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-08-14 09:03:44.768514] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=69] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:44.764674] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=998738][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:44.768617] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=3936][errcode=-4389] timer task use too much time(end=1755162224768616, start=1755162223764419, delta=1004197) [2025-08-14 09:03:44.768933] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:44.769790] WDIAG [COORDINATOR] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=964077][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:44.739223] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B92C6-0-0] [lt=44909][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162193865592) [2025-08-14 09:03:44.761931] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=15] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:03:44.777289] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=15331] temporary file module use no blocks [2025-08-14 09:03:44.777300] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:03:44.777312] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:03:44.777327] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623126194532, flush_io_finished_ret_:0, flush_io_finished_round_:26351, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:03:44.777730] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=987539] gc stale ls task succ [2025-08-14 09:03:44.777752] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=19] current monitor number(seq_=-1) [2025-08-14 09:03:44.781474] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:44.784269] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=954352] gc stale ls task succ [2025-08-14 09:03:44.787332] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=9] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:44.798437] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=972481] PNIO [ratelimit] time: 1755162224798435, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007624, add_bytes: 0 [2025-08-14 09:03:44.863907] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:44.869256] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:44.883093] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:44.961048] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:44.963896] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:44.979251] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:44.990425] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:45.065286] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:45.070428] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=697665] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:45.168260] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:45.256455] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:45.268342] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:45.309314] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:45.372272] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:45.473259] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:45.573388] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:45.591540] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:45.673418] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:44.781613] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:45.720555] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:44.869323] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:45.723348] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=852723] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:44.767733] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [720][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571f9d70, caller:ob_table_lock_service.cpp:start:276, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE6$_1273, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa090, time_wheel:0x7f99375f7d10, thread_pool:0x7f99375f74f0, is_running:0, time_interval:10.00s, expected_run_time:09:03:52.332, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:45.723939] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [720][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=954759][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:45.729568] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:45.730413] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [316][pnio1][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1001, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162210868038", warnings:[]}, tenant_id_=1001, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:03:44.755624] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:03:45.763122] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:45.747603] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=934097] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:03:45.723584] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=40293] gc stale ls task succ [2025-08-14 09:03:45.756448] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=999316] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:03:45.738614] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:45.767592] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28932][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:45.739529] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10337] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:45.767810] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28237][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:45.768098] INFO [PALF] register_parent_ (log_config_mgr.cpp:2707) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=39] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.77.0.7:2882", reason=1, leader="172.77.0.7:2882", register_time_us=1755162222770967, last_submit_register_req_time_us=1755162222770967, register_parent_reason=1) [2025-08-14 09:03:45.768123] INFO [PALF] check_parent_health (log_config_mgr.cpp:2884) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=23] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.77.0.7:2882") [2025-08-14 09:03:44.769190] INFO [COMMON] init (ob_diagnostic_info_summary.cpp:102) [982][T1002_L0_G0][T1002][YB42AC4D0007-00063C4962CA3040-0-0] [lt=1010444] Successfully init diagnostic info collector(slot_num=32, tenant_id=1002, group_id=100) [2025-08-14 09:03:45.723856] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [621][T1_L0_G0][T1][YB42AC4D0007-00063C49607B92C6-0-0] [lt=33205][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162193865592}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162214726670}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162214726670}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:03:45.768707] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27630][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:45.768524] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=26988][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:45.768365] WDIAG ev_invoke_pending (ev.c:2136) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27191][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:45.766349] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1421) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=1] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-08-14 09:03:44.767751] EDIAG [STORAGE.TRANS] refresh_elr_tenant_config_ (ob_tx_elr_util.cpp:42) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1010330, owner: ObLog, click_count: 6, time dist:[FORMAT_END=983735, ObMalloc#start=0, ObMalloc#end=25900, ObMalloc#start=17, ObMalloc#end=52, FREE_END=626], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b878a47 0x1b8787a8 0x1b878197 0x986bac9 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:45.750271] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=971462] get_max_applied_scn(scn={val:1755161749741193000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:50, palf_committed_end_lsn_:{lsn:1604090090}, palf_committed_end_scn_:{val:1755161749741193000, v:0}, last_check_scn_:{val:1755161749741193001, v:0}, max_applied_cb_scn_:{val:1755161749741193000, v:0}}) [2025-08-14 09:03:45.772743] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22424] get wrs ts(ls_id={id:1}, delta=476028735, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:45.772766] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:45.768140] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=11] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:12632581, max_lsn:8461098908, max_scn:{val:1755161719775179002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:8214679957}, reuse_lsn:{lsn:8461098908}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:8419155868}}, last_submit_lsn:{lsn:8461098786}, last_submit_end_lsn:{lsn:8461098908}, last_submit_log_id:12632581, last_submit_log_pid:45, max_flushed_lsn:{lsn:8461098786}, max_flushed_end_lsn:{lsn:8461098908}, max_flushed_log_pid:45, committed_end_lsn:{lsn:8461098908}, last_slide_log_id:12632581, last_slide_scn:{val:1755161719775179002, v:0}, last_slide_lsn:{lsn:8461098786}, last_slide_end_lsn:{lsn:8461098908}, last_slide_log_pid:45, last_slide_log_accum_checksum:1091335677, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f992979c530}, [StateMgr]={this:0x7f99297a09b0, self:"172.77.0.7:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:46}, leader:"0.0.0.0:0", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:8461098908}, scan_disk_log_finished:true, last_check_start_id:12632582, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161899748977, palf_role_change_cb:0x7f99297a7470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:45, prev:{version:1, config:{config_version:{proposal_id:43, config_seq:41}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:45, config_seq:42}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:41, prev_lsn:{lsn:8461098218}, prev_mode_pid:45}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:41, prev_lsn:{lsn:8461098218}, prev_end_lsn:{lsn:8461098377}, prev_mode_pid:45}, reconfig_barrier:{prev_log_proposal_id:41, prev_lsn:{lsn:8461098218}, prev_end_lsn:{lsn:8461098377}, prev_mode_pid:45}, persistent_config_version:{proposal_id:45, config_seq:42}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755162222770967, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755162222770967, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f992979dc70}, [ModeMgr]={palf_id:1, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:45, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:46, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:46, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:1753719139394335001, v:0}, min_block_id:93, min_block_min_scn:{val:1753712284262753001, v:0}, base_lsn_for_block_gc:{lsn:8186781696}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:46}, log_config_meta:{version:3, proposal_id:45, prev:{version:1, config:{config_version:{proposal_id:43, config_seq:41}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:45, config_seq:42}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:41, prev_lsn:{lsn:8461098218}, prev_mode_pid:45}, log_snapshot_meta:{version:2, base_lsn:{lsn:8186781696}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:46, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:655360}, readable_log_tail:{lsn:655360}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8522305536}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/meta", dir_fd:214, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66449408, block_header_serialize_buf_:0x7f99297a2bb8, flashback_version:0}, log_storage:{log_tail:{lsn:8461098908}, readable_log_tail:{lsn:8461098908}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8455200768}, min_scn:{val:1755160384851871002, v:0}, curr_block_id:126, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:215, min_block_id:93, max_block_id:127, curr_writable_block_id:126}, logical_block_size_:67104768, curr_block_writable_size_:61206628, block_header_serialize_buf_:0x7f99297a48b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1755161900739661, this:0x7f99297a10f0}, [Reconfirm]={palf_id:1, self:"172.77.0.7:2882", state:"FETCH_MAX_LOG_LSN", new_proposal_id:46, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"172.77.0.7:2882", majority_max_accept_pid:45, majority_max_lsn:{lsn:8461098908}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:1755161954782466, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f99297a0b30}) [2025-08-14 09:03:45.773078] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=4925] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161956777957, last_flushed_end_lsn:{lsn:8461098908}}]) [2025-08-14 09:03:45.773117] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=38] current log_task status(palf_id=1, self="172.77.0.7:2882", log_id=12632582, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161956777957, last_flushed_end_lsn:{lsn:8461098908}}]) [2025-08-14 09:03:45.773276] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:45.774010] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:45.775209] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1001}, delta=481042104, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:45.775255] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=23] LogLoopThread round_cost_time(us)(round_cost_time=6006860) [2025-08-14 09:03:45.775281] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:45.777787] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=15] gc stale ls task succ [2025-08-14 09:03:45.750782] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:45.778362] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=27537][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:45.751532] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:45.778529] WDIAG ev_invoke_pending (ev.c:2136) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26956][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:45.747376] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=22] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:03:45.761258] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [385]OB_MALLOC COST TOO MUCH TIME, cost_time=653561, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 16, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=653494, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=648371, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 16, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=648306, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=963934, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=963923], seq:[0] [2025-08-14 09:03:45.779119] WDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17833][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:45.763835] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=34212] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:03:45.750229] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=23] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:03:45.775229] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:45.780642] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=5381] get wrs ts(ls_id={id:1}, delta=476028735, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:45.780659] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:45.781203] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=481042104, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:45.781212] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:45.784461] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=20] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:45.787512] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=17] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:45.866381] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:03:45.866410] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=23] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:03:45.874077] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:03:45.874100] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:45.968641] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:45.980328] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:45.985533] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:46.081244] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:46.120791] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=27][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:03:46.181313] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:46.281335] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:46.381394] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:46.528396] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:45.880755] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] get_min_unreplayed_log_info(lsn={lsn:1604088821}, scn={val:1755100948296325001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873479, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:12492179, ret_code_:0}}, next_to_submit_lsn_:{lsn:1604088821}, next_to_submit_scn_:{val:1755100948296325001, v:0}, base_lsn_:{lsn:1409200128}, base_scn_:{val:1753840802540914000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:1604088821}, end_lsn:{lsn:1604088821}, read_buf:{buf_len_:3747840, buf_:0x7f98f6a05000}, block_size:67104768, log_storage_:0x7f990bfebbf0, log_storage_:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:42, read_io_size:147709952, read_disk_cost_ts:1061450, start_lsn:{lsn:1409200128}}}}}}) [2025-08-14 09:03:46.721826] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=839796] get wrs ts(ls_id={id:1}, delta=476976179, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:46.721846] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:46.723412] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=481989548, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:46.723428] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:46.723483] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=476976179, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:46.481451] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:46.725037] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=243216][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:46.720511] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:46.498249] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:46.725524] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162226725523, start=1755162226498247, delta=227276) [2025-08-14 09:03:46.731423] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:46.723492] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:46.733243] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9234] get wrs ts(ls_id={id:1001}, delta=481989548, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:46.733259] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:46.738748] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:46.739577] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:42.756053] EDIAG [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3978404, owner: ObLog, click_count: 5, time dist:[FORMAT_END=3978394, ObMalloc#start=0, ObMalloc#end=4, ALLOC_END=0, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0xfad91f4 0x99f635e 0xfad88de 0xfad867a 0x1f9ebaba 0x97c8ff1 0x1f981a55 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [729]OB_MALLOC COST TOO MUCH TIME, cost_time=751941, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:03:42.756053] INFO [COMMON] print_io_status (ob_io_manager.cpp:2756) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=3, io_config={group_configs_cnt:3, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, param_config_:{memory_limit:536870912, callback_thread_count:0, enable_io_tracer:false, object_storage_io_timeout_ms:20000}, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=930848, [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-08-14 09:03:46.740752] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [758][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB9102-0-0] [lt=45][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1001, svr="172.77.0.7:2882", version={val:1755161719775179002, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162194674462) [2025-08-14 09:03:46.740786] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [758][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB9102-0-0] [lt=31][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1001, req={req_server:"172.77.0.7:2882", version:{val:1755161719775179002, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162194674462}, twrs={inited:true, tenant_id:1001, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162225887440}, server_version_for_stat:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162225887440}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [393]OB_MALLOC COST TOO MUCH TIME, cost_time=971967, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=41, WASH_BLOCK_END=43, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=963952], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:46.741518] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [622][T1_L0_G0][T1][YB42AC4D0007-00063C49607B92E1-0-0] [lt=22][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162196676144) [2025-08-14 09:03:45.763206] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3067] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:46.747594] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=982900][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:46.741531] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [952][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8FAC-0-0] [lt=20][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1002, svr="172.77.0.7:2882", version={val:1755161758221798000, v:0}, valid_part_count=0, total_part_count=2, generate_timestamp=1755162194450593) [2025-08-14 09:03:46.747662] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [952][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8FAC-0-0] [lt=6117][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1002, req={req_server:"172.77.0.7:2882", version:{val:1755161758221798000, v:0}, valid_part_count:0, total_part_count:2, generate_timestamp:1755162194450593}, twrs={inited:true, tenant_id:1002, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162226723475}, server_version_for_stat:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162226723475}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:03:46.759262] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2082] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:46.759680] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:46.741902] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=956243][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162225767813], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:46.766905] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8882] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=10, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:46.768052] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162226756776}, version_delta=679029535) [2025-08-14 09:03:46.768679] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=679028870, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:46.768700] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:46.744304] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=26] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:03:46.766481] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=24538][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162225767813], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:46.741973] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [955][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8FB3-0-0] [lt=364][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1002, svr="172.77.0.7:2882", version={val:1755161758221798000, v:0}, valid_part_count=0, total_part_count=2, generate_timestamp=1755162208860737) [2025-08-14 09:03:46.770464] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [955][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8FB3-0-0] [lt=28446][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1002, req={req_server:"172.77.0.7:2882", version:{val:1755161758221798000, v:0}, valid_part_count:0, total_part_count:2, generate_timestamp:1755162208860737}, twrs={inited:true, tenant_id:1002, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162226723475}, server_version_for_stat:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755162226723475}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:03:46.777850] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=10] gc stale ls task succ [2025-08-14 09:03:46.781500] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3860] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162225767813]}) [2025-08-14 09:03:46.781516] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] gts statistics(tenant_id=1001, gts_rpc_cnt=0, get_gts_cache_cnt=57, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:03:47.125660] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=652720] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [857]OB_MALLOC COST TOO MUCH TIME, cost_time=956152, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=956111, ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=18, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [857]OB_MALLOC COST TOO MUCH TIME, cost_time=956199, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=956173, WASH_BLOCK_END=26], seq:[0, 1, 2] [832]OB_MALLOC COST TOO MUCH TIME, cost_time=955854, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=955805, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [832]OB_MALLOC COST TOO MUCH TIME, cost_time=955873, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=955864, WASH_BLOCK_END=9], seq:[0, 1, 2] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=953364, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=953326, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=953391, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=953378, WASH_BLOCK_END=13], seq:[0, 1, 2] [759]OB_MALLOC COST TOO MUCH TIME, cost_time=943999, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=943998], seq:[0] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=946332, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=946296, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=946354, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=946344, WASH_BLOCK_END=10], seq:[0, 1, 2] [622]OB_MALLOC COST TOO MUCH TIME, cost_time=944952, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=944908, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [622]OB_MALLOC COST TOO MUCH TIME, cost_time=944981, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=944963, WASH_BLOCK_END=18], seq:[0, 1, 2] [764]OB_MALLOC COST TOO MUCH TIME, cost_time=945482, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=945441, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [764]OB_MALLOC COST TOO MUCH TIME, cost_time=945529, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=945504, WASH_BLOCK_END=25], seq:[0, 1, 2] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=945585, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=945545, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=945607, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=945597, WASH_BLOCK_END=10], seq:[0, 1, 2] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=945651, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=945612, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=945673, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=945663, WASH_BLOCK_END=10], seq:[0, 1, 2] [1001]OB_MALLOC COST TOO MUCH TIME, cost_time=945082, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=945061, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [552]OB_MALLOC COST TOO MUCH TIME, cost_time=945737, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=945697, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [1001]OB_MALLOC COST TOO MUCH TIME, cost_time=945109, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=945098, WASH_BLOCK_END=11], seq:[0, 1, 2] [552]OB_MALLOC COST TOO MUCH TIME, cost_time=945758, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=945750, WASH_BLOCK_END=8], seq:[0, 1, 2] [729]OB_MALLOC COST TOO MUCH TIME, cost_time=944860, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=944839, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [729]OB_MALLOC COST TOO MUCH TIME, cost_time=944878, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=944869, WASH_BLOCK_END=9], seq:[0, 1, 2] [626]OB_MALLOC COST TOO MUCH TIME, cost_time=944765, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=944747, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [626]OB_MALLOC COST TOO MUCH TIME, cost_time=944796, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=944783, WASH_BLOCK_END=13], seq:[0, 1, 2] [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=943216, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=943195, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [624]OB_MALLOC COST TOO MUCH TIME, cost_time=944887, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=944869, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3] [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=943233, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=943224, WASH_BLOCK_END=9], seq:[0, 1, 2] [624]OB_MALLOC COST TOO MUCH TIME, cost_time=944918, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=944907, WASH_BLOCK_END=10], seq:[0, 1, 2] [717]OB_MALLOC COST TOO MUCH TIME, cost_time=943206, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=943188, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [717]OB_MALLOC COST TOO MUCH TIME, cost_time=943223, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=943214, WASH_BLOCK_END=9], seq:[0, 1, 2] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=943272, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=943254, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=943289, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=943281, WASH_BLOCK_END=8], seq:[0, 1, 2] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=943296, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=943277, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=943313, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=943305, WASH_BLOCK_END=8], seq:[0, 1, 2] [864]OB_MALLOC COST TOO MUCH TIME, cost_time=943333, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=943302, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [864]OB_MALLOC COST TOO MUCH TIME, cost_time=943351, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=943342, WASH_BLOCK_END=9], seq:[0, 1, 2] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=942390, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=942370, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=942414, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=942398, WASH_BLOCK_END=16], seq:[0, 1, 2] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=939526, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=939508, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=939547, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=939538, WASH_BLOCK_END=9], seq:[0, 1, 2] [276]OB_MALLOC COST TOO MUCH TIME, cost_time=937383, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=937362, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [276]OB_MALLOC COST TOO MUCH TIME, cost_time=937400, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=937392, WASH_BLOCK_END=8], seq:[0, 1, 2] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=935736, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=935704, ALLOC_CHUNK_END=12], seq:[0, 1, 2, 3] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=935753, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=935745, WASH_BLOCK_END=8], seq:[0, 1, 2] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=934173, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=934141, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=934196, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=934181, WASH_BLOCK_END=15], seq:[0, 1, 2] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=927958, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=927937, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=927978, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=927968, WASH_BLOCK_END=10], seq:[0, 1, 2] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=920796, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=920773, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=920816, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=920805, WASH_BLOCK_END=11], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=914246, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=914225, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=914262, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=914254, WASH_BLOCK_END=8], seq:[0, 1, 2] [575]OB_MALLOC COST TOO MUCH TIME, cost_time=890232, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=890208, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [575]OB_MALLOC COST TOO MUCH TIME, cost_time=890248, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=890240, WASH_BLOCK_END=8], seq:[0, 1, 2] [466]OB_MALLOC COST TOO MUCH TIME, cost_time=855864, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=17, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=855833, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [466]OB_MALLOC COST TOO MUCH TIME, cost_time=855886, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=855874, WASH_BLOCK_END=12], seq:[0, 1, 2] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=859486, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=859463, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=859510, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=859496, WASH_BLOCK_END=14], seq:[0, 1, 2] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=855929, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=855910, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=855950, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=855939, WASH_BLOCK_END=11], seq:[0, 1, 2] [822]OB_MALLOC COST TOO MUCH TIME, cost_time=761688, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=761667, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [822]OB_MALLOC COST TOO MUCH TIME, cost_time=761708, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=761699, WASH_BLOCK_END=9], seq:[0, 1, 2] [400]OB_MALLOC COST TOO MUCH TIME, cost_time=742731, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=742710, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [400]OB_MALLOC COST TOO MUCH TIME, cost_time=742750, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=742741, WASH_BLOCK_END=9], seq:[0, 1, 2] [571]OB_MALLOC COST TOO MUCH TIME, cost_time=735551, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=735528, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [571]OB_MALLOC COST TOO MUCH TIME, cost_time=735571, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=735559, WASH_BLOCK_END=12], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=759298, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=759278, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=759326, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=759307, WASH_BLOCK_END=19], seq:[0, 1, 2] [513]OB_MALLOC COST TOO MUCH TIME, cost_time=655070, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=655047, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3] [513]OB_MALLOC COST TOO MUCH TIME, cost_time=655087, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=655079, WASH_BLOCK_END=8], seq:[0, 1, 2] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=201677, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=201649, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=201704, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=201690, WASH_BLOCK_END=14], seq:[0, 1, 2] [623]OB_MALLOC COST TOO MUCH TIME, cost_time=948155, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=948139, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [623]OB_MALLOC COST TOO MUCH TIME, cost_time=948173, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=948165, WASH_BLOCK_END=8], seq:[0, 1, 2] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=945441, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=945416, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=945475, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=945459, WASH_BLOCK_END=16], seq:[0, 1, 2] [2025-08-14 09:03:45.764368] EDIAG [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=1402][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1967241, , BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1d189030 0x1d18285e 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:45.784328] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1947519, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [759]OB_MALLOC COST TOO MUCH TIME, cost_time=952677, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=944010], seq:[0, 1] [2025-08-14 09:03:46.684687] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1049003, owner: ObLog, click_count: 12, time dist:[ALLOC_BLOCK#start=7, ALLOC_BLOCK#end=44714, ALLOC_BLOCK#start=10763, ALLOC_BLOCK#end=1, FORMAT_END=40189, ObMalloc#start=0, ObMalloc#end=945539, ObMalloc#start=26, ObMalloc#end=50, ObMalloc#start=7506, ObMalloc#end=55, FREE_END=153], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:45.779910] EDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16538][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1952921, , BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [396]OB_MALLOC COST TOO MUCH TIME, cost_time=959672, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=959665], seq:[0] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=1000321, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=53, WASH_BLOCK_END=40570, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=959693], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:46.740579] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=225][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1000342, owner: ObLog, click_count: 5, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=1000330, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:46.725424] EDIAG [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=241747][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1015548, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1bf2ec62 0x986ebd7 0x986bee6 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:46.741274] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [625][T1_L0_G0][T1][YB42AC4D0007-00063C49607B92D9-0-0] [lt=25][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1, svr="172.77.0.7:2882", version={val:1755161547738516001, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162195875625) [2025-08-14 09:03:47.724388] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [625][T1_L0_G0][T1][YB42AC4D0007-00063C49607B92D9-0-0] [lt=981625][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1, req={req_server:"172.77.0.7:2882", version:{val:1755161547738516001, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162195875625}, twrs={inited:true, tenant_id:1, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162226768075}, server_version_for_stat:{version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162226768075}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:03:46.781528] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162225767813]}) [2025-08-14 09:03:47.731622] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=273] gc stale ls task succ [2025-08-14 09:03:46.741441] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [759][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB911E-0-0] [lt=27][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1001, svr="172.77.0.7:2882", version={val:1755161719775179002, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162210868033) [2025-08-14 09:03:47.735302] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [759][T1001_L0_G0][T1001][YB42AC4D0007-00063C4960CB911E-0-0] [lt=992358][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1001, req={req_server:"172.77.0.7:2882", version:{val:1755161719775179002, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162210868033}, twrs={inited:true, tenant_id:1001, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162226740590}, server_version_for_stat:{version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162226740590}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:03:47.735826] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1952921] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:47.736072] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=1454] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:03:47.736089] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=15] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:03:46.740579] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=225] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:47.742457] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1000342][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:47.742474] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:46.725424] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=241747] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:47.742540] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=1015548] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:47.742677] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=176] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:46.758166] WDIAG pkts_wq_flush (packet_server.c:52) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:46.780955] INFO [PALF] check_parent_health (log_config_mgr.cpp:2875) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=12] re_register_parent reason(palf_id=1, self="172.77.0.7:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1755162222770967, last_first_register_time_us=1755158728414267, register_time_us=1755162222770967, curr_time_us=1755162226776627) [2025-08-14 09:03:47.744645] INFO pktc_resp_cb_on_msg (pktc_resp.h:72) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=0] PNIO resp cb not found: packet_id=560165 [2025-08-14 09:03:47.744652] INFO pktc_resp_cb_on_msg (pktc_resp.h:72) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO resp cb not found: packet_id=560157 [2025-08-14 09:03:47.744656] INFO pktc_resp_cb_on_msg (pktc_resp.h:72) [316][pnio1][T0][Y0-0000000000000000-0-0] [lt=3] PNIO resp cb not found: packet_id=560181 [2025-08-14 09:03:47.745054] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9957] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:47.736132] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=42] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623127201645, flush_io_finished_ret_:0, flush_io_finished_round_:26340, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:03:47.746751] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=680006563, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:47.746776] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:47.744517] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:47.748888] INFO [PALF] register_parent_ (log_config_mgr.cpp:2707) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=962232] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.77.0.7:2882", reason=1, leader="172.77.0.7:2882", register_time_us=1755162227744650, last_submit_register_req_time_us=1755162227744650, register_parent_reason=1) [2025-08-14 09:03:47.748909] INFO [PALF] check_parent_health (log_config_mgr.cpp:2884) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=19] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.77.0.7:2882") [2025-08-14 09:03:47.748994] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6323] get wrs ts(ls_id={id:1}, delta=478006327, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:47.749013] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:47.749635] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1001}, delta=483021439, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:47.749649] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:47.749675] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=478008070, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:47.749688] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:47.750441] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=483021439, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:47.750461] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:47.751255] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=195] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:47.751840] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=2510] get wrs ts(ls_id={id:1}, delta=507976626, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:47.751861] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:47.752396] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=507976626, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:47.752418] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:46.995162] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:47.752699] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162227752699, start=1755162226995161, delta=757538) [2025-08-14 09:03:47.752725] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=30344][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:47.753402] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=173] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:47.726697] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=943737][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1001}) [2025-08-14 09:03:47.755038] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=28297] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1001}) [2025-08-14 09:03:47.759294] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=205] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:47.753455] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=150] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:47.773620] WDIAG ev_invoke_pending (ev.c:2136) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20134][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:47.771508] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:03:47.778328] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=6808] get number(ret=-4038, service_type_=0, range=1, base_id=1755162227766136000, start_id=0, end_id=140295968341544) [2025-08-14 09:03:47.778347] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:03:47.779268] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=11] gc stale ls task succ [2025-08-14 09:03:47.781853] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=270] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:47.803272] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:47.805396] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:47.821300] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=914772] PNIO [ratelimit] time: 1755162227821298, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007639, add_bytes: 0 [2025-08-14 09:03:47.829923] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1521] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:47.832618] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=657444][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a6798, timer:0x7f9966fddee0, task:0x7f99697d29f8, task_type:"N9oceanbase5table21ObTableGroupCommitMgr19ObTableGroupOpsTaskE", scheduled_time:1755161925783083, pushed_time:0, delay:1000000}, running_task_token={this:0x7f994e1a65d8, timer:0x7f9966fddee0, task:0x7f994f1bbcc0, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161694824965, pushed_time:1755161925772608, delay:5000000}, total_delay=302049534, threashold=1000000, tenant_id=1) [2025-08-14 09:03:47.842602] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:125) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=4] try to adjust log cb pool [2025-08-14 09:03:47.842628] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=19] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:47.846756] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=680107362, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:47.846783] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:47.861450] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=363][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:47.929982] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=21] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:47.935264] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:47.946799] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=680207741, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:47.946825] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:47.961543] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:48.030066] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=22] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:48.034250] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=2073] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:03:48.034847] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:48.046824] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=680307722, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:48.046847] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:48.130147] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=20] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:48.131286] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=13] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:48.230169] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=21] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:48.231244] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:48.330202] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=21] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:48.717437] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:48.726970] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=946591] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:47.787459] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=946237] gc stale ls task succ [2025-08-14 09:03:48.730504] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=11] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:48.730641] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=935200] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:48.052881] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:48.430260] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=21] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:48.731000] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=300279] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:48.730267] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2797] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:48.732663] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=12] gc stale ls task succ [729]OB_MALLOC COST TOO MUCH TIME, cost_time=879492, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=10], seq:[0, 1] [857]OB_MALLOC COST TOO MUCH TIME, cost_time=882938, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1], seq:[0, 1] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=586791, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:03:48.733352] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=508957134, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:48.733383] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=28] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=2, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:48.733414] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119797572, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755162138745740) [2025-08-14 09:03:48.733939] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=508957134, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:48.733953] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:48.734568] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=680993797, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:48.734581] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=5, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:48.735067] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=680993797, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:48.735078] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:47.778366] INFO [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:171) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=10] handle gts request statistics(total_rt=23106, total_cnt=1, avg_rt=1.155300000000000000e+04, avg_cnt=1.000000000000000056e-01) [2025-08-14 09:03:48.736347] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=1266] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:03:48.736377] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=24] temporary file module use no blocks [2025-08-14 09:03:48.736387] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:03:48.736402] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:03:48.736419] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=17] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623130175912, flush_io_finished_ret_:0, flush_io_finished_round_:26321, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:03:47.733903] EDIAG [ELECT] prepare (election_proposer.cpp:380) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1008008, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1004338, ObMalloc#start=1, ObMalloc#end=425, ObMalloc#start=24, ObMalloc#end=73, FREE_END=3147], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0xf9d67b1 0xf9d4184 0xfa09ca2 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:48.731000] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=945907] current monitor number(seq_=-1) [2025-08-14 09:03:48.743573] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:48.731219] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=949694] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:03:48.061568] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:48.743746] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=681145][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:48.730693] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=944275] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:48.730963] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162228730962, start=1755162228052879, delta=678083) [2025-08-14 09:03:48.744406] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=583] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:48.725262] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13705] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:48.746807] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21510][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:48.751231] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:48.751431] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:48.754919] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=986726] log compressor cycles once. (ret=0, cost_time=986972, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25896497152, fast_delete_log_mode=false) [2025-08-14 09:03:48.755117] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:48.755173] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=296] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:48.739742] INFO [STORAGE] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:123) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=1330] wead read timestamp is too old(timestamp={val:1755161749741193000, v:0}, gts_scn={val:1755161762721798002, v:0}, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1755161749741193000, v:0}}) [2025-08-14 09:03:46.794762] EDIAG [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1957738, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1957731, ObMalloc#start=1, ObMalloc#end=1, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x989c577 0x989bc96 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:46.794762] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] [thread pool stat](pool_stat={pool:0x564d6e8922c0, last_check_time:1755162226794760, last_idle_time:102477764956, last_thread_count:12}, *pool={name:TimerWK, this:0x564d6e8922c0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:10, threads_idle_time:102477764956, tenant_id:500}, qsize=8, current_idle_time=102477764956) [2025-08-14 09:03:48.755471] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1393] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:48.755493] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=1957738] [thread pool stat](pool_stat={pool:0x7f99667cbcf0, last_check_time:1755162228755492, last_idle_time:34837923013, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x7f99667cbcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:34837923013, tenant_id:500}, qsize=-1, current_idle_time=34837923013) [2025-08-14 09:03:48.755513] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=19] [thread pool stat](pool_stat={pool:0x7f99667ccaf0, last_check_time:1755162228755513, last_idle_time:26612406064, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x7f99667ccaf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:26612406064, tenant_id:500}, qsize=-1, current_idle_time=26612406064) [2025-08-14 09:03:48.755526] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] [thread pool stat](pool_stat={pool:0x7f9969de56b0, last_check_time:1755162228755525, last_idle_time:102938681542, last_thread_count:17}, *pool={name:TimerWK, this:0x7f9969de56b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:16, threads_idle_time:102938681542, tenant_id:1}, qsize=10, current_idle_time=102938681542) [2025-08-14 09:03:48.755538] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] [thread pool stat](pool_stat={pool:0x7f99356b0e70, last_check_time:1755162228755537, last_idle_time:102366343177, last_thread_count:15}, *pool={name:TimerWK, this:0x7f99356b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:12, threads_idle_time:102366343177, tenant_id:1001}, qsize=12, current_idle_time=102366343177) [2025-08-14 09:03:48.755549] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] [thread pool stat](pool_stat={pool:0x7f99230b0e70, last_check_time:1755162228755549, last_idle_time:102534969158, last_thread_count:17}, *pool={name:TimerWK, this:0x7f99230b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:16, threads_idle_time:102534969158, tenant_id:1002}, qsize=15, current_idle_time=102534969158) [2025-08-14 09:03:48.755667] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17657][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:48.755813] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2608] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:03:48.755832] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:03:48.755847] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SERVER] net_in=0kB/s, net_out=0kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=1280000kB/s [2025-08-14 09:03:48.736035] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=956220][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162227755246], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:48.755999] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=19931][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162227755246], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:48.756019] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162227755246]}) [2025-08-14 09:03:48.756033] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts statistics(tenant_id=1002, gts_rpc_cnt=0, get_gts_cache_cnt=31, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:03:48.756046] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162227755246]}) [2025-08-14 09:03:48.756067] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1002}) [2025-08-14 09:03:48.756078] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1002}) [2025-08-14 09:03:48.756202] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:288) [732][T1001_ArcFetche][T1001][YB42AC4D0007-00063C495CA99C16-0-0] [lt=12817] ObArchiveFetcher is running(thread_index=0) [2025-08-14 09:03:48.756424] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=72][errcode=-4389] run this task use too much time(*p_task_={this:0x7f996f40b7f0, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40bad0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:0, time_interval:10.00s, expected_run_time:09:03:48.802, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:48.756897] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15669] get_max_applied_scn(scn={val:1755161744727824001, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:49, palf_committed_end_lsn_:{lsn:7323686009}, palf_committed_end_scn_:{val:1755161747742105001, v:0}, last_check_scn_:{val:1755161749741193000, v:0}, max_applied_cb_scn_:{val:1755161744727824001, v:0}}) [2025-08-14 09:03:48.756925] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] get_min_unreplayed_log_info(lsn={lsn:7323685240}, scn={val:1755100946796590002, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873485, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:8946764, ret_code_:0}}, next_to_submit_lsn_:{lsn:7323685240}, next_to_submit_scn_:{val:1755100946796590002, v:0}, base_lsn_:{lsn:7180210176}, base_scn_:{val:1753840803166611000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:7323685240}, end_lsn:{lsn:7323685240}, read_buf:{buf_len_:3747840, buf_:0x7f98f7a05000}, block_size:67104768, log_storage_:0x7f9901131bf0, log_storage_:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:28, read_io_size:98508800, read_disk_cost_ts:645339, start_lsn:{lsn:7180210176}}}}}}) [2025-08-14 09:03:48.757605] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=79] get wrs ts(ls_id={id:1001}, delta=484028926, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:48.757621] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:48.757639] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=213] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:48.757651] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=470535850, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=251772, succ_cluster_heartbeat_count=1502, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161006924036004, v:0}, local_cluster_delta=1221833612, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:03:48.757694] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27] get wrs ts(ls_id={id:1}, delta=479015557, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:48.757704] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:48.759262] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7977] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:48.760029] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1001}, delta=484031301, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:48.760254] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:48.761317] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=176] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:48.762007] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=215] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:48.762041] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=244] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:48.760056] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:48.768689] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=256] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:48.779500] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=24] gc stale ls task succ [2025-08-14 09:03:48.779891] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:48.787522] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=941099] gc stale ls task succ [2025-08-14 09:03:48.806253] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:48.812639] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:48.828896] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=23] PNIO [ratelimit] time: 1755162228828893, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007595, add_bytes: 0 [2025-08-14 09:03:48.830567] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=16] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:48.831063] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=69] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:48.833476] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=509058295, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=252369, succ_cluster_heartbeat_count=1905, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161003593738002, v:0}, local_cluster_delta=1225239736, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:03:48.834178] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=37] get wrs ts(ls_id={id:1}, delta=509058959, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:48.834198] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:48.835293] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=681096697, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:48.835316] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:48.843809] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:48.845309] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=940023][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc5378, timer:0x7f9970ecbf90, task:0x564d6cd25100, task_type:"N9oceanbase10rootserver23ObIngressBWAllocServiceE", scheduled_time:1755162167753034, pushed_time:0, delay:3000000}, running_task_token={this:0x7f9970fc5458, timer:0x7f9970ecbf90, task:0x564d6cd25880, task_type:"N9oceanbase10rootserver18ObSSNTAllocServiceE", scheduled_time:1755162050765815, pushed_time:1755162164753047, delay:3000000}, total_delay=61092274, threashold=1000000, tenant_id=500) [2025-08-14 09:03:48.857716] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=5833] get wrs ts(ls_id={id:1}, delta=479116113, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:48.857739] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:48.858357] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1001}, delta=484129482, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:48.858382] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:48.930626] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=19] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:48.931123] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:48.934098] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=509158175, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:48.934122] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:48.935373] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=681195897, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:48.935394] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:48.943828] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:03:48.943858] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:48.944165] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:48.957764] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=479216466, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:48.957787] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:48.958391] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] get wrs ts(ls_id={id:1001}, delta=484229835, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:48.958407] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.030688] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=20] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:49.031184] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=17] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:49.034302] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=509258413, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:49.034329] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.035403] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=681296126, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:49.035432] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.035538] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=3863] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:03:49.043866] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:49.045265] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:49.057806] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=479315639, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:49.057835] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.058457] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1001}, delta=484329008, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:49.058474] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.130747] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=20] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:49.131206] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=17] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:49.134319] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=509358141, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:49.134342] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.135418] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=681395918, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:49.135439] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.140338] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=19] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:49.143929] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:49.144089] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:49.157872] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=479416312, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:49.157894] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.158514] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27] get wrs ts(ls_id={id:1001}, delta=484429681, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:49.158533] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.230817] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=20] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:49.231262] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=21] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:49.232052] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:49.234375] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=509458440, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:49.234399] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.235553] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=681496161, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:49.235575] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.243949] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:49.256771] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=16] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:49.256794] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162229256767], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:49.256826] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162229256767], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:49.256846] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162229256767]}) [2025-08-14 09:03:49.256860] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=63, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:03:49.256872] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162229256767]}) [2025-08-14 09:03:49.256885] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=10] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:49.256902] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162229256884], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:49.256914] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162229256884]}) [2025-08-14 09:03:49.257121] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:03:49.257135] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:03:49.257149] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162229257117], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:49.257162] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162229257117]}) [2025-08-14 09:03:49.257933] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=479515805, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:49.257954] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.258583] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1001}, delta=484530235, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:49.258604] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.272302] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:49.331331] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:49.333251] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=20] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:49.334340] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=509558124, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:49.334362] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.335533] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=681596934, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:49.335559] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.342256] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=33][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a4e38, timer:0x7f9967161f10, task:0x7f994b3f4c40, task_type:"N9oceanbase7obmysql20ObConstructQueueTaskE", scheduled_time:1755162048238296, pushed_time:0, delay:500000}, running_task_token={this:0x7f994e1a4d58, timer:0x7f9967161f10, task:0x7f994b3f4bf0, task_type:"N9oceanbase7obmysql15ObEliminateTaskE", scheduled_time:1755162042245325, pushed_time:1755162047738501, delay:500000}, total_delay=181103958, threashold=1000000, tenant_id=1) [2025-08-14 09:03:49.344019] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:49.357991] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=479616576, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:49.358010] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.358526] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=484629945, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:49.358542] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.433327] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=21] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:49.435584] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=681696641, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:49.435613] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.436767] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=509661010, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:49.436787] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.437272] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=22] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:49.444103] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:49.458047] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=479716322, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:49.458065] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.458572] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1001}, delta=484729691, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:49.458583] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.534257] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=21] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:49.535631] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=681796233, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:49.535652] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.536840] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=509760629, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:49.536858] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.537343] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:49.544165] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:49.558105] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=479815926, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:49.558124] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.558643] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=484830366, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:49.558654] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.634322] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=24] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:49.635704] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=681897022, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:49.635725] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.636881] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=509861416, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:49.636904] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.642381] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=1889][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:49.644223] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:49.722324] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:49.637402] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:49.729646] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=479987847, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:49.729674] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.730280] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1001}, delta=485001216, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:49.730300] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.730091] INFO sync_wash_ (ob_tenant_ctx_allocator.cpp:442) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=36] [MEM][WASH] tenant_id: 500, ctx_id: 17, washed_size: 2031616 [2025-08-14 09:03:49.730477] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=31] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:49.732282] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:49.735490] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=18] gc stale ls task succ [2025-08-14 09:03:49.735393] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=23] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:49.737469] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=18] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:03:49.737917] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=30] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:03:49.737955] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=32] temporary file module use no blocks [2025-08-14 09:03:49.737989] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=31] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:03:49.738009] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=18] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:03:49.738030] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=21] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623131176218, flush_io_finished_ret_:0, flush_io_finished_round_:26322, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:03:49.738259] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:49.741147] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=9362] current monitor number(seq_=-1) [2025-08-14 09:03:49.741375] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161101770923, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755161572801635) [2025-08-14 09:03:49.741486] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:49.742027] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=34] get wrs ts(ls_id={id:1}, delta=682002747, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:49.742053] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=2, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.742456] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:49.742604] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=369] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:49.744825] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:49.747548] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [556][T1_OBJLockGC][T1][Y0-0000000000000000-0-0] [lt=29650][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:49.751405] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=509966084, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:49.751424] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.742490] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:49.751127] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=15621] snapshot_traffic_info once [2025-08-14 09:03:49.730537] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2376] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:49.752978] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22404][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:49.751417] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=334] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:03:49.753207] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=1782] temporary file module use no blocks [2025-08-14 09:03:49.753216] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:03:49.753245] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=28] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:03:49.753260] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623131183412, flush_io_finished_ret_:0, flush_io_finished_round_:26355, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:03:49.742591] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:49.754946] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12334][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:49.755633] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:49.757789] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=20] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:49.757956] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:49.758585] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] get wrs ts(ls_id={id:1}, delta=480000070, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:49.758606] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.759172] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1001}, delta=485013439, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:49.759186] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.759615] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=60] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:49.761768] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:49.761805] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:49.761828] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:49.761902] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:49.763132] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:49.763156] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162229763128], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:49.763183] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162229763128], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:49.763201] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162229763128]}) [2025-08-14 09:03:49.763217] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=12] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:49.763226] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162229763215], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:49.763250] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=23] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162229763215]}) [2025-08-14 09:03:49.763503] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:03:49.763518] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:03:49.763531] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162229763499], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:49.763566] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=33] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162229763499]}) [2025-08-14 09:03:49.767250] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:49.763833] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:49.779559] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=28] gc stale ls task succ [2025-08-14 09:03:49.779718] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:49.787567] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=22] gc stale ls task succ [2025-08-14 09:03:49.835880] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] get wrs ts(ls_id={id:1}, delta=682002747, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:49.835911] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.836252] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=1066] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:49.836515] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=22] PNIO [ratelimit] time: 1755162229836513, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007620, add_bytes: 0 [2025-08-14 09:03:49.838116] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=509966084, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:49.838155] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=29] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.840268] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15683] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:49.844258] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=34][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a5ed8, timer:0x7f994b3feee0, task:0x7f994b3fef68, task_type:"N9oceanbase8observer18ObTableLoadService25ObClientTaskAutoAbortTaskE", scheduled_time:1755162048762959, pushed_time:0, delay:1000000}, running_task_token={this:0x7f994e1a5df8, timer:0x7f994b3feee0, task:0x7f994b3fef50, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162023758648, pushed_time:1755162122757495, delay:1000000}, total_delay=181081297, threashold=1000000, tenant_id=1) [2025-08-14 09:03:49.844884] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:49.850254] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:49.853539] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=3423][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:49.858684] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=480000070, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:49.858708] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.859337] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1001}, delta=485013439, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:49.859359] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.911603] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a4658, timer:0x7f9966a63f10, task:0x7f9922e05038, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr12TabletGCTaskE", scheduled_time:1755162048800633, pushed_time:0, delay:20000}, running_task_token={this:0x7f991b7a4498, timer:0x7f9966a63f10, task:0x7f9922e05010, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr11TableGCTaskE", scheduled_time:1755162048800605, pushed_time:1755162048800872, delay:20000}, total_delay=181110969, threashold=1000000, tenant_id=1002) [2025-08-14 09:03:49.936084] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=682002747, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:49.936110] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.936312] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=14] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:49.938207] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=509966084, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:49.938242] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.940316] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=27] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:49.944976] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:49.958731] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=480000070, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:49.958755] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:49.959373] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1001}, delta=485013439, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:49.959400] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:50.036178] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] get wrs ts(ls_id={id:1}, delta=682002747, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:50.036205] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:50.038910] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=29] get wrs ts(ls_id={id:1}, delta=509966084, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:50.038938] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:50.040287] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=18] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:50.040380] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=19] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:50.045037] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:50.058808] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=480000070, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:50.058830] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:50.059512] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1001}, delta=485013439, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:50.059543] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:50.136251] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=682002747, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:50.136276] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:50.138946] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=509966084, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:50.138970] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:50.140416] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:50.141173] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=20] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:50.141246] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=17] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:50.145104] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:50.158867] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=480000070, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:50.158893] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:50.159485] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] get wrs ts(ls_id={id:1001}, delta=485013439, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:50.159502] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:50.236297] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=682002747, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:50.236319] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:50.239004] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] get wrs ts(ls_id={id:1}, delta=509966084, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:50.239027] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:50.240481] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=17] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:50.241305] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=18] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:50.246246] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:50.258929] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=480000070, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:50.258952] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:50.259564] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1001}, delta=485013439, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:50.259578] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:50.263884] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=45] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:50.263908] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162230263880], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:50.263930] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162230263880]}) [2025-08-14 09:03:50.263946] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=13] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:50.263954] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162230263945], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:50.263966] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162230263945]}) [2025-08-14 09:03:50.263976] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162230263945]}) [2025-08-14 09:03:50.264156] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:03:50.264167] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:03:50.340543] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=17] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:50.342253] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=14] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:50.440602] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=17] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:50.442273] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=23] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:50.542333] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=22] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:50.642391] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=21] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:50.721287] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:50.264181] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162230264152], range_size:1, sender:"172.77.0.7:2882"}) [729]OB_MALLOC COST TOO MUCH TIME, cost_time=383973, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [857]OB_MALLOC COST TOO MUCH TIME, cost_time=367185, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=390538, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [2025-08-14 09:03:50.728079] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=224] get wrs ts(ls_id={id:1}, delta=509966084, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:03:50.728101] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:50.729000] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:50.729437] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=1040] get wrs ts(ls_id={id:1001}, delta=485013439, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:50.729463] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:50.729497] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=480000070, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:50.729513] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:50.730113] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] get wrs ts(ls_id={id:1001}, delta=485013439, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:50.730128] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:50.728037] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:03:50.730474] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=1997915][errcode=-4015] clock out of order(cur_ts=1755162229764113, cur_ts_=1755162229741263, delta=22850) [2025-08-14 09:03:50.540662] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=21] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:50.730806] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=189848] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:03:50.346305] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:50.733626] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=386732][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:50.734128] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6630] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=5, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:50.734733] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=682994455, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:03:50.734744] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:50.736942] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=5904] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:50.737063] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:50.738421] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:50.739010] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9498] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:50.739342] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=35] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:03:50.739358] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] temporary file module use no blocks [2025-08-14 09:03:50.739364] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=5] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:03:50.739375] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:03:50.740635] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:50.739386] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623132177834, flush_io_finished_ret_:0, flush_io_finished_round_:26323, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:03:50.744470] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:50.724097] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=459215] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162230264152]}) [2025-08-14 09:03:50.749578] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=25439][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1002}) [2025-08-14 09:03:50.749591] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1002}) [2025-08-14 09:03:50.750364] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9765] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:03:50.750416] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=40] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:03:50.753602] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=314] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:50.753609] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=24] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:03:50.753627] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] temporary file module use no blocks [2025-08-14 09:03:50.753635] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:03:50.753648] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:03:50.753662] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623132193053, flush_io_finished_ret_:0, flush_io_finished_round_:26356, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:03:50.753838] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:50.754495] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:50.767267] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:50.753638] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=451] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:50.772939] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=19271][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:50.753799] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=3491] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:03:50.773757] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=19924] temporary file module use no blocks [2025-08-14 09:03:50.773771] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:03:50.773785] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:03:50.773799] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623132190569, flush_io_finished_ret_:0, flush_io_finished_round_:26343, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:03:50.773442] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=29537] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:03:50.773352] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=17977] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:02:12.760345] EDIAG [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1531) [871][T1002_TSnapSvc][T1001][YB42AC4D0007-00063C4962B99DC4-0-0] [lt=20][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3996744, owner: ObLog, click_count: 5, time dist:[FORMAT_END=3992545, ObMalloc#start=1, ObMalloc#end=4197, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x9ace3c3 0x9acda90 0xfad1d79 0xfad1b46 0x148e2b50 0x93d7bb4 0x9394489 0x9725f91 0x124dfd77 0x96a5a4d 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x9acea95 0x1df14b07 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:38.758890] WDIAG [STORAGE] init_and_open_iter (ob_mds_row_iterator.cpp:232) [14945][T1_TimerWK8_MergeLoop][T1][YB42AC4D0007-00063C4978199C2C-0-0] [lt=27][errcode=-4012] fail to init and open multiple scan merge(ret=-4012, scan_param={tablet_id:{id:50345}, ls_id:{id:1}, column_ids:[16, 17, 18, 19], index_id:88888, key_ranges:[{"range":"table_id:88888,group_idx:0,index_ordered_idx:0,({"TINYINT":3},MIN;{"TINYINT":3},MAX)"}], ss_key_ranges:[], range_array_pos:[], timeout:1755161926786042, scan_flag:{scan_order:0, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_bare_row_scan:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:1, is_select_follower:0, enable_lob_prefetch:0, is_bare_row_scan:0, mr_mv_scan:0, is_plain_insert_gts_opt:0, reserved:0}, sql_mode:134217760, reserved_cell_count:4, schema_version:9527, query_begin_schema_version:-1, limit_offset:{offset_:0, limit_:-1}, for_update:false, wait:-1, frozen_version:9223372036854775807, is_get:false, output_exprs:NULL, op_filters:NULL, trans_desc:NULL, snapshot:{this:0x7f97efad2ce0, valid:true, source:2, core:{version:{val:4611686018427387903, v:0}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, table_param:{table_id:88888, output_projector:[0, 1, 2, 3], aggregate_projector:[], group_by_projector:[], output_sel_mask:[true, true, true, true], pad_col_projector:[], read_param_version:1, main_read_info:{schema_column_count:4, schema_rowkey_cnt:2, rowkey_cnt:4, trans_col_index:-1, mview_old_new_col_index:-1, group_idx_col_index:-1, seq_read_column_count:2, max_col_index:5, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 4, 5], data:0x7f98c5026740, count:4, allocator:0x7f97efad31e8, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 3], data:0x7f98c5026760, count:4, allocator:0x7f97efad31e8, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=16 {type:"TINYINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"CHAR", collation:"binary", coercibility:"INVALID"} order=0, column_id=18 {type:"CHAR", collation:"binary", coercibility:"INVALID"} order=0, column_id=19 {type:"CHAR", collation:"binary", coercibility:"INVALID"} order=0], data:0x7f98c5026700, count:4, allocator:0x7f97efad31e8, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:4, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}], data:0x7f98c50267b0, count:4, allocator:0x7f97efad31e8, capacity:4, init_cnt:4}, has_all_column_group:true, need_truncate_filter:false}, use_lob_locator:false, rowid_version:0, rowid_projector:[], enable_lob_locator_v2:false, is_fts_index:false, parser_name:"", parser_properties:"", is_vec_index:false, is_column_replica_table:false, is_normal_cgs_at_the_end:false, is_mlog_table:false}, sample_info:{method:0, percent:"1.000000000000000000e+02", seed:-1, table_id:0, scope:0, force_block:false}, need_scn:false, need_switch_param:false, is_mds_query:true, fb_read_tx_uncommitted:false, external_file_format:{"TYPE":"INVALID"}, external_file_location:"", tx_seq_base:-1, auto_split_filter_type:18446744073709551615, is_tablet_spliting:false, need_update_tablet_param:false, mds_collector:{exist_uncommitted_node:false, exist_new_committed_node:false}}) [2025-08-14 09:03:50.780417] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:50.792430] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=19] gc stale ls task succ [2025-08-14 09:03:50.808254] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:50.833255] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=50] tx gc loop thread is running(MTL_ID()=1) [2025-08-14 09:03:50.833277] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=21] try gc retain ctx [2025-08-14 09:03:50.833286] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] try to retry start_working [2025-08-14 09:03:50.833688] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:03:50.833713] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:50.841789] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=164] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:03:50.841850] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:03:50.841895] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:03:50.841939] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:03:50.844309] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=23] PNIO [ratelimit] time: 1755162230844307, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007794, add_bytes: 0 [2025-08-14 09:03:50.850449] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:50.865259] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=10] LogLoopThread round_cost_time(us)(round_cost_time=13) [2025-08-14 09:03:50.933748] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:50.947789] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=746645][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:51.033846] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:51.133867] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:51.144500] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=47] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:51.221245] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=17][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:03:51.222025] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=335][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:03:51.233943] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:51.322121] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=33] ObArchiveSender is running(thread_index=0) [729]OB_MALLOC COST TOO MUCH TIME, cost_time=912387, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1], seq:[0, 1] [857]OB_MALLOC COST TOO MUCH TIME, cost_time=901456, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=904956, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=979488, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=979486], seq:[0] [2025-08-14 09:03:51.722715] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:51.775272] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=52476][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:50.781057] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=40139] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:51.775472] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=191] get wrs ts(ls_id={id:1}, delta=482025477, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:03:51.775490] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161277810819002, v:0}", last_lsn={lsn:1604088111}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:51.776049] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1001}, delta=487038846, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:03:51.776063] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:51.780806] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:51.793265] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=25] gc stale ls task succ [2025-08-14 09:03:51.851741] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=24] PNIO [ratelimit] time: 1755162231851739, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007432, add_bytes: 0 [2025-08-14 09:03:51.914205] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a5998, timer:0x7f9935f8df20, task:0x7f99200f5a78, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr15MediumCheckTaskE", scheduled_time:1755161721762022, pushed_time:0, delay:20000000}, running_task_token={this:0x7f991b7a58b8, timer:0x7f9935f8df20, task:0x7f99200f5a38, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr14MediumLoopTaskE", scheduled_time:1755161718796975, pushed_time:1755161718805054, delay:120000000}, total_delay=510152182, threashold=1000000, tenant_id=1002) [2025-08-14 09:03:51.914260] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=54][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6418, timer:0x7f994b1dfb70, task:0x7f9920110068, task_type:"N9oceanbase5share22ObIndexUsageReportTaskE", scheduled_time:1755161727099859, pushed_time:0, delay:900000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=504814400, threashold=1000000, tenant_id=1002) [2025-08-14 09:03:51.914271] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6bf8, timer:0x7f994b1dfb70, task:0x7f99201e29f8, task_type:"N9oceanbase5table21ObTableGroupCommitMgr19ObTableGroupOpsTaskE", scheduled_time:1755161732717636, pushed_time:0, delay:1000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=499196635, threashold=1000000, tenant_id=1002) [2025-08-14 09:03:51.914284] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6878, timer:0x7f994b1dfb70, task:0x7f9920110178, task_type:"N9oceanbase5share27ObIndexUsageRefreshConfTaskE", scheduled_time:1755161733727027, pushed_time:0, delay:2000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=498187257, threashold=1000000, tenant_id=1002) [2025-08-14 09:03:51.914299] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6958, timer:0x7f994b1dfb70, task:0x7f9920110810, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161734741162, pushed_time:0, delay:5000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=497173137, threashold=1000000, tenant_id=1002) [2025-08-14 09:03:51.914308] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6a38, timer:0x7f994b1dfb70, task:0x7f9920110af8, task_type:"N9oceanbase5share23ObStorageIOUsageRepoter23TenantIOUsageReportTaskE", scheduled_time:1755161734803774, pushed_time:0, delay:5000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=497110534, threashold=1000000, tenant_id=1002) [2025-08-14 09:03:51.914316] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6e98, timer:0x7f994b1dfb70, task:0x7f99201075a8, task_type:"N9oceanbase3omt11ObTenantSrs27TenantSrsUpdatePeriodicTaskE", scheduled_time:1755161734810627, pushed_time:0, delay:0}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=497103689, threashold=1000000, tenant_id=1002) [2025-08-14 09:03:51.914324] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a64f8, timer:0x7f994b1dfb70, task:0x7f992010e1b0, task_type:"N9oceanbase5table20ObTableObjectPoolMgr26ObTableSessEliminationTaskE", scheduled_time:1755161735735271, pushed_time:0, delay:5000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=496179053, threashold=1000000, tenant_id=1002) [2025-08-14 09:03:51.914338] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a65d8, timer:0x7f994b1dfb70, task:0x7f992010e1c8, task_type:"N9oceanbase5table20ObTableObjectPoolMgr27ObTableSessSysVarUpdateTaskE", scheduled_time:1755161735797729, pushed_time:0, delay:5000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=496116609, threashold=1000000, tenant_id=1002) [2025-08-14 09:03:51.914350] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a66b8, timer:0x7f994b1dfb70, task:0x7f992010e1e0, task_type:"N9oceanbase5table20ObTableObjectPoolMgr28ObTableReqResEliminationTaskE", scheduled_time:1755161735811808, pushed_time:0, delay:5000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=496102541, threashold=1000000, tenant_id=1002) [2025-08-14 09:03:51.914359] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a4ff8, timer:0x7f994b1dfb70, task:0x7f9922ff3e50, task_type:"N9oceanbase3sql3dtl23ObDTLIntermResultGCTaskE", scheduled_time:1755161742809616, pushed_time:0, delay:10000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=489104742, threashold=1000000, tenant_id=1002) [2025-08-14 09:03:51.914372] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a5b58, timer:0x7f99535c3f30, task:0x7f9922ffee90, task_type:"N9oceanbase7storage21ObTenantFreezeInfoMgr24UpdateLSResvSnapshotTaskE", scheduled_time:1755161744768148, pushed_time:0, delay:10000000}, running_task_token={this:0x7f991b7a5a78, timer:0x7f99535c3f30, task:0x7f9922ffee78, task_type:"N9oceanbase7storage21ObTenantFreezeInfoMgr10ReloadTaskE", scheduled_time:1755161737748006, pushed_time:1755161737749908, delay:3000000}, total_delay=487146224, threashold=1000000, tenant_id=1002) [2025-08-14 09:03:51.914386] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a7838, timer:0x7f994b1dfb70, task:0x7f9920110820, task_type:"N9oceanbase10rootserver16ObMViewMdsOpTaskE", scheduled_time:1755161749755625, pushed_time:0, delay:5000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=482158761, threashold=1000000, tenant_id=1002) [2025-08-14 09:03:51.914406] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a74b8, timer:0x7f994b1dfb70, task:0x7f9920110730, task_type:"N9oceanbase10rootserver23ObMViewPushSnapshotTaskE", scheduled_time:1755161804754766, pushed_time:0, delay:60000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=427159640, threashold=1000000, tenant_id=1002) [2025-08-14 09:03:51.914417] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a7218, timer:0x7f994b1dfb70, task:0x7f99201107f8, task_type:"N9oceanbase10rootserver24ObMViewCleanSnapshotTaskE", scheduled_time:1755161804755624, pushed_time:0, delay:60000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=427158793, threashold=1000000, tenant_id=1002) [2025-08-14 09:03:38.758890] EDIAG [STORAGE] init_and_open_iter (ob_mds_row_iterator.cpp:232) [14945][T1_TimerWK8_MergeLoop][T1][YB42AC4D0007-00063C4978199C2C-0-0] [lt=27][errcode=-4012] LOGGER COST TOO MUCH TIME, cost: 12001462, owner: ObLog, click_count: 5, time dist:[FORMAT_END=12001448, ObMalloc#start=2, ObMalloc#end=5, ALLOC_END=3, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1bc4ae6e 0x9766310 0x9765ae5 0x976579e 0x9764de3 0x975ab8f 0x97f7c68 0x97f78b3 0x97f7636 0x97f5260 0x97f4a4a 0x97f4859 0x97f4305 0x1aee6975 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [388]OB_MALLOC COST TOO MUCH TIME, cost_time=986800, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=986799], seq:[0] [388]OB_MALLOC COST TOO MUCH TIME, cost_time=986815, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=986811], seq:[0, 1] [2025-08-14 09:03:51.334001] EDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] LOGGER COST TOO MUCH TIME, cost: 1392019, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=6, ALLOC_BLOCK#end=3, FORMAT_END=1392004, ALLOC_END=2, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b52e837 0x996d04c 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [730]OB_MALLOC COST TOO MUCH TIME, cost_time=1946683, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=51, WASH_BLOCK_END=44, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=979505], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:50.779619] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=15][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1946715, owner: ObLog, click_count: 5, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=1946697, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:51.738462] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:52.727891] WDIAG ev_invoke_pending (ev.c:2136) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=987933][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:51.334001] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:52.728154] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1392019] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:03:52.728173] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:50.779619] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=15] gc stale ls task succ [2025-08-14 09:03:52.729301] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=1946715] gc stale ls task succ [2025-08-14 09:03:51.775344] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=992783] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:52.730889] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=954098] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:52.730901] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:52.730910] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:52.730920] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:51.914430] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6098, timer:0x7f99200ff180, task:0x7f99200ff208, task_type:"N9oceanbase8observer18ObTableLoadService25ObClientTaskAutoAbortTaskE", scheduled_time:1755162050765334, pushed_time:0, delay:1000000}, running_task_token={this:0x7f991b7a5fb8, timer:0x7f99200ff180, task:0x7f99200ff1f0, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162050755138, pushed_time:1755162084760013, delay:1000000}, total_delay=181149096, threashold=1000000, tenant_id=1002) [2025-08-14 09:03:52.736280] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=946486] blacklist_loop exec finished(cost_time=15, is_enabled=true, send_cnt=0) [2025-08-14 09:03:50.945207] EDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=36][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 1790254, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1790247, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x9757ced 0x97569df 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:50.945207] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=36][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a6a38, timer:0x7f9966fddee0, task:0x7f994f1b35a8, task_type:"N9oceanbase3omt11ObTenantSrs27TenantSrsUpdatePeriodicTaskE", scheduled_time:1755161929777851, pushed_time:0, delay:0}, running_task_token={this:0x7f994e1a65d8, timer:0x7f9966fddee0, task:0x7f994f1bbcc0, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161694824965, pushed_time:1755161925772608, delay:5000000}, total_delay=301167354, threashold=1000000, tenant_id=1) [2025-08-14 09:03:52.738226] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=985] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:52.738320] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=20072] gc stale ls task succ [2025-08-14 09:03:52.738724] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1533] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:51.759930] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=169] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:52.740055] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=978644][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:51.763650] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [497][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=49][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:51.758310] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=507698][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162231249835], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:52.742425] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=982624][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162231249835], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:52.742442] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162231249835]}) [2025-08-14 09:03:52.742454] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162231249835]}) [2025-08-14 09:03:52.742466] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:03:52.742476] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:03:52.742496] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=8] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:03:52.742506] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162232742492], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:52.756927] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8362] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:52.758592] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=233] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:52.759174] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:52.760546] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=732] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:00:48.739965] EDIAG [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [561][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] LOGGER COST TOO MUCH TIME, cost: 183714429, owner: ObLog, click_count: 6, time dist:[FORMAT_END=2528, ObMalloc#start=0, ObMalloc#end=403, ObMalloc#start=20, ObMalloc#end=1018977, FREE_END=182692501], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x973b640 0x973b132 0x973a32d 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:50.760921] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1980619, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:50.841979] EDIAG [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1899719, owner: ObLog, click_count: 8, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=25, FORMAT_END=1883365, ObMalloc#start=1, ObMalloc#end=79, ObMalloc#start=19, ObMalloc#end=53, FREE_END=16176], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1f9ebc06 0x97c9052 0x1f981a55 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:52.765289] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1980619][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:52.765320] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:52.765336] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:52.765366] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=1899719] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:03:52.765381] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:03:52.767332] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2494] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:52.772064] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=1974] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:03:52.775100] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=668] temporary file module use no blocks [2025-08-14 09:03:52.775125] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=24] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:03:52.775151] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=24] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:03:52.775172] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=20] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623134214794, flush_io_finished_ret_:0, flush_io_finished_round_:26345, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:03:52.779258] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:52.794278] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=25] gc stale ls task succ [2025-08-14 09:03:52.833601] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [571][T1_ArcSender][T1][YB42AC4D0007-00063C495BA99C16-0-0] [lt=738999][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:03:52.838145] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=822393][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a4e38, timer:0x7f99535f3df0, task:0x7f9922ff6bf0, task_type:"N9oceanbase7obmysql15ObEliminateTaskE", scheduled_time:1755161931284195, pushed_time:0, delay:500000}, running_task_token={this:0x7f991b7a4f18, timer:0x7f99535f3df0, task:0x7f9922ff6c40, task_type:"N9oceanbase7obmysql20ObConstructQueueTaskE", scheduled_time:1755161931282757, pushed_time:1755161931285404, delay:500000}, total_delay=301553948, threashold=1000000, tenant_id=1002) [2025-08-14 09:03:52.855251] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=43][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:52.859387] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=31] PNIO [ratelimit] time: 1755162232859386, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007647, add_bytes: 0 [2025-08-14 09:03:52.865583] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:03:52.865648] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:03:52.865695] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:03:52.865743] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:03:52.865786] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:03:52.865802] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:03:52.865819] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:03:53.038415] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6db8, timer:0x7f994b1dfb70, task:0x7f99201e33a0, task_type:"N9oceanbase5table20ObTableClientInfoMgr27ObTableClientInfoRetireTaskE", scheduled_time:1755161750768689, pushed_time:0, delay:60000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=482269725, threashold=1000000, tenant_id=1002) [2025-08-14 09:03:53.140345] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=1790254][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a5a78, timer:0x7f996714bf20, task:0x7f994b3fce90, task_type:"N9oceanbase7storage21ObTenantFreezeInfoMgr24UpdateLSResvSnapshotTaskE", scheduled_time:1755161931783996, pushed_time:0, delay:10000000}, running_task_token={this:0x7f994e1a5998, timer:0x7f996714bf20, task:0x7f994b3fce78, task_type:"N9oceanbase7storage21ObTenantFreezeInfoMgr10ReloadTaskE", scheduled_time:1755161924772450, pushed_time:1755161924772520, delay:3000000}, total_delay=301356348, threashold=1000000, tenant_id=1) [2025-08-14 09:03:53.147634] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=237] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:53.723473] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=219] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [729]OB_MALLOC COST TOO MUCH TIME, cost_time=1047787, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [2025-08-14 09:00:47.739202] EDIAG [SQL.PC] need_check_schema_version (ob_plan_cache_value.cpp:2334) [801][T1002_TntTransf][T1002][YB42AC4D0007-00063C496259A60F-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1981080, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1016165, ObMalloc#start=1, ObMalloc#end=964908, ALLOC_END=2, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0x15385470 0x9433023 0x92b85e2 0x92b41d6 0x92b202a 0x92d2008 0x96a777d 0x96a6679 0x96a5033 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x9a64aaf 0x10502edb 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [800]OB_MALLOC COST TOO MUCH TIME, cost_time=1015281, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1015279], seq:[0] [561]OB_MALLOC COST TOO MUCH TIME, cost_time=1018440, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1018431], seq:[0] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=1005284, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=15, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1005254, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=1005359, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1005294, WASH_BLOCK_END=65], seq:[0, 1, 2] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=1004647, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1004626, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=1004668, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1004659, WASH_BLOCK_END=9], seq:[0, 1, 2] [389]OB_MALLOC COST TOO MUCH TIME, cost_time=1003135, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1003114, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [389]OB_MALLOC COST TOO MUCH TIME, cost_time=1003151, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1003143, WASH_BLOCK_END=8], seq:[0, 1, 2] [383]OB_MALLOC COST TOO MUCH TIME, cost_time=1001883, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1001857, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [383]OB_MALLOC COST TOO MUCH TIME, cost_time=1001899, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1001891, WASH_BLOCK_END=8], seq:[0, 1, 2] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=1001935, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1001916, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=1001952, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1001943, WASH_BLOCK_END=9], seq:[0, 1, 2] [386]OB_MALLOC COST TOO MUCH TIME, cost_time=1001976, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1001956, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [386]OB_MALLOC COST TOO MUCH TIME, cost_time=1001992, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1001984, WASH_BLOCK_END=8], seq:[0, 1, 2] [388]OB_MALLOC COST TOO MUCH TIME, cost_time=1002038, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1002019, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [388]OB_MALLOC COST TOO MUCH TIME, cost_time=1002063, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1002054, WASH_BLOCK_END=8], seq:[0, 1, 2] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=1002036, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1002014, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=1002053, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1002045, WASH_BLOCK_END=8], seq:[0, 1, 2] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=1001201, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1001178, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=1001216, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1001209, WASH_BLOCK_END=7], seq:[0, 1, 2] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=998370, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=998347, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=998390, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=998382, WASH_BLOCK_END=8], seq:[0, 1, 2] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=997724, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=997705, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=997740, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=997732, WASH_BLOCK_END=8], seq:[0, 1, 2] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=996863, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=996843, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=996879, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=996871, WASH_BLOCK_END=8], seq:[0, 1, 2] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=996850, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=996831, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=996867, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=996859, WASH_BLOCK_END=7], seq:[0, 1, 2] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=996913, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=996893, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=996929, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=996921, WASH_BLOCK_END=8], seq:[0, 1, 2] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=996490, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=996469, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=996506, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=996498, WASH_BLOCK_END=7], seq:[0, 1, 2] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=995798, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=995778, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=995815, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=995807, WASH_BLOCK_END=8], seq:[0, 1, 2] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=995860, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=995841, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=995876, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=995868, WASH_BLOCK_END=8], seq:[0, 1, 2] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=993335, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=993314, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=993353, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=993344, WASH_BLOCK_END=8], seq:[0, 1, 2] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=989554, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=989531, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=989570, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=989562, WASH_BLOCK_END=8], seq:[0, 1, 2] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=989522, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=989500, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=989539, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=989531, WASH_BLOCK_END=8], seq:[0, 1, 2] [864]OB_MALLOC COST TOO MUCH TIME, cost_time=983729, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=983706, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [864]OB_MALLOC COST TOO MUCH TIME, cost_time=983805, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=983748, WASH_BLOCK_END=57], seq:[0, 1, 2] [2025-08-14 09:03:53.753556] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3398][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1003382, owner: ObLog, click_count: 6, time dist:[FORMAT_END=6, ObMalloc#start=0, ObMalloc#end=1002059, ObMalloc#start=1, ObMalloc#end=49, FREE_END=1267], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:53.753422] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1003805, owner: ObLog, click_count: 6, time dist:[FORMAT_END=6, ObMalloc#start=0, ObMalloc#end=1001999, ObMalloc#start=1, ObMalloc#end=48, FREE_END=1750], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:53.753367] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1004002, owner: ObLog, click_count: 6, time dist:[FORMAT_END=9, ObMalloc#start=0, ObMalloc#end=1001958, ObMalloc#start=1, ObMalloc#end=49, FREE_END=1985], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:53.753312] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5779][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1004180, owner: ObLog, click_count: 6, time dist:[FORMAT_END=21, ObMalloc#start=1, ObMalloc#end=1001906, ObMalloc#start=1, ObMalloc#end=49, FREE_END=2202], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:53.751977] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1005652, owner: ObLog, click_count: 6, time dist:[FORMAT_END=7, ObMalloc#start=1, ObMalloc#end=1003157, ObMalloc#start=1, ObMalloc#end=50, FREE_END=2436], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:53.750364] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=1003][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1007396, owner: ObLog, click_count: 6, time dist:[FORMAT_END=7, ObMalloc#start=0, ObMalloc#end=1004676, ObMalloc#start=0, ObMalloc#end=48, FREE_END=2665], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:53.754481] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=231][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1003685, owner: ObLog, click_count: 6, time dist:[FORMAT_END=8, ObMalloc#start=0, ObMalloc#end=1001224, ObMalloc#start=0, ObMalloc#end=57, FREE_END=2396], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:53.757416] EDIAG [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1001066, owner: ObLog, click_count: 6, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=998396, ObMalloc#start=33, ObMalloc#end=42, FREE_END=2584], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a1f867 0x27a135c3 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:53.749569] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2813][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1009032, owner: ObLog, click_count: 6, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=1005367, ObMalloc#start=5, ObMalloc#end=44, FREE_END=3605], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [477]OB_MALLOC COST TOO MUCH TIME, cost_time=979488, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=979462, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=979538, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=979510, WASH_BLOCK_END=27], seq:[0, 1, 2] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=960871, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=960844, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=892241, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=892217, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=892267, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=892258, WASH_BLOCK_END=8], seq:[0, 1, 2] [513]OB_MALLOC COST TOO MUCH TIME, cost_time=605860, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=605837, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [513]OB_MALLOC COST TOO MUCH TIME, cost_time=605877, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=605869, WASH_BLOCK_END=8], seq:[0, 1, 2] [808]OB_MALLOC COST TOO MUCH TIME, cost_time=805919, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=805895, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [808]OB_MALLOC COST TOO MUCH TIME, cost_time=805938, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=805929, WASH_BLOCK_END=9], seq:[0, 1, 2] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=960893, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=960882, WASH_BLOCK_END=10], seq:[0, 1, 2] [2025-08-14 09:03:53.750343] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=762][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1009185, owner: ObLog, click_count: 6, time dist:[FORMAT_END=983993, ObMalloc#start=1, ObMalloc#end=24991, ObMalloc#start=1, ObMalloc#end=51, FREE_END=148], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:52.772670] EDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=321][errcode=-4015] LOGGER COST TOO MUCH TIME, cost: 1986020, owner: ObLog, click_count: 8, time dist:[FORMAT_END=1968994, ObMalloc#start=1, ObMalloc#end=16287, ObMalloc#start=26, ObMalloc#end=64, ObMalloc#start=197, ObMalloc#end=98, FREE_END=282], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x279e59be 0x971b1c1 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [398]OB_MALLOC COST TOO MUCH TIME, cost_time=980691, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=980520, ALLOC_CHUNK_END=153], seq:[0, 1, 2, 3] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=980728, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=980705, WASH_BLOCK_END=23], seq:[0, 1, 2] [381]OB_MALLOC COST TOO MUCH TIME, cost_time=1995863, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1995861], seq:[0] [381]OB_MALLOC COST TOO MUCH TIME, cost_time=1996109, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=64, WASH_BLOCK_END=145, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1995896], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:53.729774] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1996526, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=7, ALLOC_BLOCK#end=12, FORMAT_END=1996504, ALLOC_END=0, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [561]OB_MALLOC COST TOO MUCH TIME, cost_time=1990958, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1018460], seq:[0, 1] [2025-08-14 09:03:53.736297] EDIAG [DETECT] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [561][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=183714429][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1991011, owner: ObLog, click_count: 5, time dist:[FORMAT_END=24, ObMalloc#start=1, ObMalloc#end=1990980, ALLOC_END=2, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ea7c4 0x9818f9c 0x973b153 0x973a32d 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [800]OB_MALLOC COST TOO MUCH TIME, cost_time=1997662, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1015298], seq:[0, 1] [2025-08-14 09:03:53.729700] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1999042, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1999038, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:52.755021] EDIAG [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2972316, owner: ObLog, click_count: 6, time dist:[FORMAT_END=2971755, ObMalloc#start=1, ObMalloc#end=71, ObMalloc#start=1, ObMalloc#end=49, FREE_END=438], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1c02313f 0x97a45d8 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:51.749936] EDIAG [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=292][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3977439, owner: ObLog, click_count: 6, time dist:[FORMAT_END=3974417, ObMalloc#start=1, ObMalloc#end=74, ObMalloc#start=22, ObMalloc#end=52, FREE_END=2873], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1bdd73e2 0x1bdd6e36 0x986bc6f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:52.748670] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6150][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162232742492], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:03:53.729774] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=7] gc stale ls task succ [2025-08-14 09:03:55.729341] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=1996526] gc stale ls task succ [2025-08-14 09:03:53.736297] WDIAG [DETECT] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [561][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=183714429][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:52.730930] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:54.768667] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=25923] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:03:51.739482] INFO [SQL.PC] need_check_schema_version (ob_plan_cache_value.cpp:2334) [578][T1_Occam][T1][YB42AC4D0007-00063C4966299E20-0-0] [lt=23] need check schema(new_schema_version=1753073979411448, cached_tenant_schema_version=1753073979411448, is_contain_synonym()=false, contain_sys_name_table_=true, is_contain_tmp_tbl()=false, is_contain_sys_pl_object()=false, need_check=true, constructed_sql_=select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = ?) [2025-08-14 09:03:53.729700] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:55.731800] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:55.731815] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:55.734870] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [726][T1001_ObXAHbWor][T1001][Y0-0000000000000000-0-0] [lt=988710] XA scheduler heartbeat task statistics(avg_time=332655) [2025-08-14 09:03:54.732273] EDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 1001310, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1001303, ObMalloc#start=0, ObMalloc#end=2, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1b53d414 0x97ad9aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:53.760216] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1972407, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1972400, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:54.732268] EDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 1001887, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1001883, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1b53d414 0x97ad9aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:53.749650] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=994317][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1983062, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1983056, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:54.732273] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162232744700, start=1755162231752270, delta=992430) [2025-08-14 09:03:53.760216] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:55.735642] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=0] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:54.732268] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162232744596, start=1755162231808867, delta=935729) [2025-08-14 09:03:53.749650] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=994317] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:52.748670] EDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6150][errcode=-4038] LOGGER COST TOO MUCH TIME, cost: 2976083, owner: ObLog, click_count: 5, time dist:[FORMAT_END=979947, ObMalloc#start=1, ObMalloc#end=1996127, ALLOC_END=2, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1b5043ed 0x1b501579 0x1b5009d1 0x1b50387c 0x1b8905de 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:55.735792] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=2976083] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162232742492]}) [2025-08-14 09:03:53.731498] EDIAG [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2001611, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1994850, ObMalloc#start=1, ObMalloc#end=73, ObMalloc#start=23, ObMalloc#end=45, FREE_END=6619], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1bdd73e2 0x1bdd6e36 0x986bc6f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:52.828217] EDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] LOGGER COST TOO MUCH TIME, cost: 2903558, owner: ObLog, click_count: 11, time dist:[ALLOC_BLOCK#start=9, ALLOC_BLOCK#end=3, FORMAT_END=2896931, ObMalloc#start=0, ObMalloc#end=62, ObMalloc#start=26, ObMalloc#end=61, ObMalloc#start=6459, ObMalloc#end=2, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b52e837 0x996d04c 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:52.828217] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:55.736191] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=2903558] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:03:52.730930] EDIAG [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2995796, owner: ObLog, click_count: 5, time dist:[FORMAT_END=998096, ObMalloc#start=1, ObMalloc#end=1997693, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x99721a1 0x997a72e 0x2006ed17 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:55.737057] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=2995796] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:55.737077] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=20] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:55.739352] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1638] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:55.739396] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1498] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:55.739425] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1948] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:52.862654] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2874564, owner: ObLog, click_count: 11, time dist:[ALLOC_BLOCK#start=7, ALLOC_BLOCK#end=2, FORMAT_END=2864470, ObMalloc#start=0, ObMalloc#end=52, ObMalloc#start=21, ObMalloc#end=53, ObMalloc#start=9943, ObMalloc#end=10, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:52.862654] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:55.741604] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162235741603, start=1755162232862652, delta=2878951) [2025-08-14 09:03:51.756084] EDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3980698, owner: ObLog, click_count: 5, time dist:[FORMAT_END=3980681, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=8, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0xfc0e73d 0x993f560 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:51.756084] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1001, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:8571658, max_lsn:7323687369, max_scn:{val:1755161749741193002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:7278610047}, reuse_lsn:{lsn:7323686131}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:7281744329}}, last_submit_lsn:{lsn:7323686131}, last_submit_end_lsn:{lsn:7323687369}, last_submit_log_id:8571658, last_submit_log_pid:49, max_flushed_lsn:{lsn:7323686009}, max_flushed_end_lsn:{lsn:7323686131}, max_flushed_log_pid:49, committed_end_lsn:{lsn:7323686009}, last_slide_log_id:8571656, last_slide_scn:{val:1755161747742105001, v:0}, last_slide_lsn:{lsn:7323685887}, last_slide_end_lsn:{lsn:7323686009}, last_slide_log_pid:49, last_slide_log_accum_checksum:2505458373, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f990112a530}, [StateMgr]={this:0x7f990112e9b0, self:"172.77.0.7:2882", palf_id:1001, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"PENDING", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:49}, leader:"0.0.0.0:0", leader_epoch:-1, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:7323687369}, scan_disk_log_finished:true, last_check_start_id:8571657, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161767744317, palf_role_change_cb:0x7f9901135470, allow_vote:true}, [ConfigMgr]={palf_id:1001, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:49, prev:{version:1, config:{config_version:{proposal_id:48, config_seq:47}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:49, config_seq:48}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_mode_pid:49}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_end_lsn:{lsn:7323685240}, prev_mode_pid:49}, reconfig_barrier:{prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_end_lsn:{lsn:7323685240}, prev_mode_pid:49}, persistent_config_version:{proposal_id:49, config_seq:48}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755161739755312, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755161739755312, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f990112bc70}, [ModeMgr]={palf_id:1001, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, accepted_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1001, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1753069671378640001, v:0}, base_lsn_for_block_gc:{lsn:7247314944}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:49}, log_config_meta:{version:3, proposal_id:49, prev:{version:1, config:{config_version:{proposal_id:48, config_seq:47}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:49, config_seq:48}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_mode_pid:49}, log_snapshot_meta:{version:2, base_lsn:{lsn:7247314944}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1753069671378640000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}}, log_meta_storage:{log_tail:{lsn:765952}, readable_log_tail:{lsn:765952}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7381524480}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/meta", dir_fd:232, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66338816, block_header_serialize_buf_:0x7f9901130bb8, flashback_version:0}, log_storage:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, palf_epoch:2, last_purge_throttling_ts:1755161769777679, this:0x7f990112f0f0}, [Reconfirm]={palf_id:1001, self:"172.77.0.7:2882", state:"WAITING_LOG_FLUSHED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:1755161769777679, this:0x7f990112eb30}) [2025-08-14 09:03:54.763181] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15846] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:03:55.745128] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2733] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:55.748427] INFO [PALF] register_parent_ (log_config_mgr.cpp:2707) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=23998] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.77.0.7:2882", reason=1, leader="172.77.0.7:2882", register_time_us=1755162235733516, last_submit_register_req_time_us=1755162235733516, register_parent_reason=1) [2025-08-14 09:03:55.748454] INFO [PALF] check_parent_health (log_config_mgr.cpp:2884) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=24] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.77.0.7:2882") [2025-08-14 09:03:55.751365] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:55.751568] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=400] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:55.752062] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=248] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:55.754378] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=584] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:55.755422] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:55.758302] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=207] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:55.758714] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=984002] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:03:55.759229] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=162] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:55.759711] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=997292] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:53.753453] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=663][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1004834, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:55.759780] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=997978] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:55.765488] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:55.735812] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] gts statistics(tenant_id=1001, gts_rpc_cnt=0, get_gts_cache_cnt=71, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:03:55.784298] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=48410] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162232742492]}) [2025-08-14 09:03:55.784323] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1001}) [2025-08-14 09:03:55.760123] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:55.784421] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24260][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:55.764314] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=828] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:55.787267] WDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22917][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:55.764920] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:55.765657] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:55.765770] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:55.765852] WDIAG ev_invoke_pending (ev.c:2136) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1004834][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:55.767001] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=239] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:55.787493] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20461][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:55.787497] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21597] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:55.787520] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:55.787714] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20423][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:55.736217] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:03:55.775915] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:55.805270] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=203] gc stale ls task succ [2025-08-14 09:03:55.841684] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:55.864936] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=44][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:55.882208] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=242] PNIO [ratelimit] time: 1755162235882206, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007615, add_bytes: 0 [2025-08-14 09:03:55.896481] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=4789347][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:56.165116] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=186] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:56.322212] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=21][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:03:56.322306] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=24][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:03:56.720245] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=433] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [856]OB_MALLOC COST TOO MUCH TIME, cost_time=935088, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 10, time dist:[ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=934632, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 10, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=993442, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=46, WASH_BLOCK_END=55, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=32418], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:55.851422] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:12632581, max_lsn:8461098908, max_scn:{val:1755161719775179002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:8214679957}, reuse_lsn:{lsn:8461098908}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:8419155868}}, last_submit_lsn:{lsn:8461098786}, last_submit_end_lsn:{lsn:8461098908}, last_submit_log_id:12632581, last_submit_log_pid:45, max_flushed_lsn:{lsn:8461098786}, max_flushed_end_lsn:{lsn:8461098908}, max_flushed_log_pid:45, committed_end_lsn:{lsn:8461098908}, last_slide_log_id:12632581, last_slide_scn:{val:1755161719775179002, v:0}, last_slide_lsn:{lsn:8461098786}, last_slide_end_lsn:{lsn:8461098908}, last_slide_log_pid:45, last_slide_log_accum_checksum:1091335677, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f992979c530}, [StateMgr]={this:0x7f99297a09b0, self:"172.77.0.7:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:46}, leader:"0.0.0.0:0", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:8461098908}, scan_disk_log_finished:true, last_check_start_id:12632582, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161899748977, palf_role_change_cb:0x7f99297a7470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:45, prev:{version:1, config:{config_version:{proposal_id:43, config_seq:41}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:45, config_seq:42}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:41, prev_lsn:{lsn:8461098218}, prev_mode_pid:45}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:41, prev_lsn:{lsn:8461098218}, prev_end_lsn:{lsn:8461098377}, prev_mode_pid:45}, reconfig_barrier:{prev_log_proposal_id:41, prev_lsn:{lsn:8461098218}, prev_end_lsn:{lsn:8461098377}, prev_mode_pid:45}, persistent_config_version:{proposal_id:45, config_seq:42}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755162235733516, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755162235733516, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f992979dc70}, [ModeMgr]={palf_id:1, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:45, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:46, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:46, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:1753719139394335001, v:0}, min_block_id:93, min_block_min_scn:{val:1753712284262753001, v:0}, base_lsn_for_block_gc:{lsn:8186781696}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:46}, log_config_meta:{version:3, proposal_id:45, prev:{version:1, config:{config_version:{proposal_id:43, config_seq:41}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:45, config_seq:42}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:41, prev_lsn:{lsn:8461098218}, prev_mode_pid:45}, log_snapshot_meta:{version:2, base_lsn:{lsn:8186781696}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:46, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:655360}, readable_log_tail:{lsn:655360}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8522305536}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/meta", dir_fd:214, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66449408, block_header_serialize_buf_:0x7f99297a2bb8, flashback_version:0}, log_storage:{log_tail:{lsn:8461098908}, readable_log_tail:{lsn:8461098908}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8455200768}, min_scn:{val:1755160384851871002, v:0}, curr_block_id:126, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:215, min_block_id:93, max_block_id:127, curr_writable_block_id:126}, logical_block_size_:67104768, curr_block_writable_size_:61206628, block_header_serialize_buf_:0x7f99297a48b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1755161900739661, this:0x7f99297a10f0}, [Reconfirm]={palf_id:1, self:"172.77.0.7:2882", state:"FETCH_MAX_LOG_LSN", new_proposal_id:46, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"172.77.0.7:2882", majority_max_accept_pid:45, majority_max_lsn:{lsn:8461098908}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:1755161954782466, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f99297a0b30}) [2025-08-14 09:03:56.745716] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=892929] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161956777957, last_flushed_end_lsn:{lsn:8461098908}}]) [2025-08-14 09:03:56.745750] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=29] current log_task status(palf_id=1, self="172.77.0.7:2882", log_id=12632582, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161956777957, last_flushed_end_lsn:{lsn:8461098908}}]) [2025-08-14 09:03:55.784336] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1001}) [2025-08-14 09:03:55.751434] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=1120] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:56.746412] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=993473][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:56.729403] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=8] gc stale ls task succ [2025-08-14 09:03:56.733997] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:56.746894] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12876][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:55.840271] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:56.746928] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162236746927, start=1755162235840269, delta=906658) [2025-08-14 09:03:55.835247] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:56.746991] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] timer task use too much time(end=1755162236746990, start=1755162235835245, delta=911745) [2025-08-14 09:03:56.742304] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=1983062] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:55.791267] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=54963][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:55.784345] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:03:56.746981] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=3980698] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [2025-08-14 09:03:56.747307] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=323] current log_task status(palf_id=1001, self="172.77.0.7:2882", log_id=8571657, log_task={header:{begin_lsn:{lsn:7323686009}, end_lsn:{lsn:7323686131}, log_id:8571657, min_scn:{val:1755161749741193000, v:0}, max_scn:{val:1755161749741193000, v:0}, data_len:66, proposal_id:49, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:7323686009}, data_checksum:2222604892, accum_checksum:1531236791, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1755161749745909, freeze_ts:1755161749745914, submit_ts:1755161749745940, flushed_ts:1755161785758706, gen_to_freeze cost time:5, gen_to_submit cost time:31, submit_to_flush cost time:36012766}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [2025-08-14 09:03:56.747626] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26483] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:03:56.754257] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:56.754600] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:56.755610] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:55.791874] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=15261] temporary file module use no blocks [2025-08-14 09:03:56.738259] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:56.757883] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19593][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:56.746260] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=1999] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:03:56.758202] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11920] temporary file module use no blocks [2025-08-14 09:03:56.740255] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:56.758485] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=17] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:56.746515] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:56.758562] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12027][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:56.759260] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=40] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:56.760288] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=20] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:56.756313] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:56.761271] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:55.757336] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1005776, owner: ObLog, click_count: 5, time dist:[FORMAT_END=986503, ObMalloc#start=1, ObMalloc#end=19248, ALLOC_END=21, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:56.517664] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=398][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:56.764519] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] timer task use too much time(end=1755162236764519, start=1755162236517662, delta=246857) [2025-08-14 09:03:55.757336] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:54.760355] EDIAG [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2001606, owner: ObLog, click_count: 9, time dist:[ALLOC_BLOCK#start=969813, ALLOC_BLOCK#end=96, ALLOC_BLOCK#start=10189, ALLOC_BLOCK#end=2, FORMAT_END=1021497, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1af3d850 0x1af3c294 0x20071394 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:54.760355] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] [COMPACTION DAG STATUS] (start_time=1755162221765193, end_time=1755162234737808, dag_status={}) [2025-08-14 09:03:55.731879] EDIAG [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=2972316][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1053620, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1051761, ObMalloc#start=9, ObMalloc#end=1673, ObMalloc#start=18, ObMalloc#end=30, FREE_END=129], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x9c6174d 0xfd347d6 0xfd344b7 0xfd34254 0x1c029398 0x97a1ad0 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:55.749063] EDIAG [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=978502][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1036498, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1034628, ObMalloc#start=0, ObMalloc#end=1718, ObMalloc#start=18, ObMalloc#end=43, FREE_END=91], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x9c6174d 0xfd347d6 0xfd344b7 0xfd34254 0x1c029398 0x97a1ad0 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:53.775609] EDIAG [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=279][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3007034, owner: ObLog, click_count: 6, time dist:[FORMAT_END=3006047, ObMalloc#start=1, ObMalloc#end=856, ObMalloc#start=20, ObMalloc#end=37, FREE_END=73], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x9c6174d 0xfd347d6 0xfd344b7 0xfd34254 0x1c029398 0x97a1ad0 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [398]OB_MALLOC COST TOO MUCH TIME, cost_time=966720, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=38330], seq:[0, 1] [2025-08-14 09:03:55.791764] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11324][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:55.791764] EDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11324][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1920303, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=934649, FORMAT_END=18895, ObMalloc#start=0, ObMalloc#end=966749, ALLOC_END=1, APPEND_END=6], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:57.715013] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1920303] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:56.758212] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [864]OB_MALLOC COST TOO MUCH TIME, cost_time=955454, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=955453], seq:[0] [2025-08-14 09:03:57.716004] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1388] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:57.716026] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:57.718774] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=970077][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:57.718796] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:57.721638] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:57.729777] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=17364] gc stale ls task succ [2025-08-14 09:03:57.716043] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=17] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:56.756604] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=25] log compressor cycles once. (ret=0, cost_time=4029974, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25897234432, fast_delete_log_mode=false) [2025-08-14 09:03:57.732628] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=33] LogLoopThread round_cost_time(us)(round_cost_time=1881076) [864]OB_MALLOC COST TOO MUCH TIME, cost_time=972196, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=955469], seq:[0, 1] [2025-08-14 09:03:56.758999] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:03:57.731336] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15268] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:57.732734] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1395] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:57.732745] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:03:57.732757] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:03:57.732767] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:03:57.738261] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:57.738491] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:57.740252] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=29] LogLoopThread round_cost_time(us)(round_cost_time=6888867) [2025-08-14 09:03:57.740292] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:56.760653] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:03:57.740397] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6692] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:57.749529] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=4867] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:56.783899] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [412][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=17287] [sql nio session](*s={this:0x7f98f733a030, session_id:3221590255, trace_id:YB42AC4D0007-00063C49651A3232-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:257}, err:0, last_decode_time:1755161952736711, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1716, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:03:56.747218] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=9] [sql nio session](*s={this:0x7f98f72b3030, session_id:3221562808, trace_id:YB42AC4D0007-00063C4965DA1F2E-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:267}, err:0, last_decode_time:1755161748766816, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1890, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:03:56.747218] EDIAG [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1000918, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1000909, ObMalloc#start=0, ObMalloc#end=2, ALLOC_END=0, APPEND_END=7], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a5bbf2 0x94da542 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:57.749677] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=1000918] [sql nio session](*s={this:0x7f98f72ab030, session_id:3221589435, trace_id:YB42AC4D0007-00063C4965DA1F35-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:246}, err:0, last_decode_time:1755161762721798, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:2456, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [393]OB_MALLOC COST TOO MUCH TIME, cost_time=962016, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=962015], seq:[0] [2025-08-14 09:03:56.746155] EDIAG [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1002838, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1002813, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=21], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a5bbf2 0x94da542 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:56.745480] EDIAG [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1003894, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1003581, ObMalloc#start=1, ObMalloc#end=70, ObMalloc#start=21, ObMalloc#end=38, FREE_END=183], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a5bbf2 0x94da542 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:56.746155] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=11] [sql nio session](*s={this:0x7f98f733f030, session_id:3221589417, trace_id:YB42AC4D0007-00063C496359B268-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:252}, err:0, last_decode_time:1755161813752329, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1778, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:03:57.750533] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=1002838] [sql nio session](*s={this:0x7f98f733e030, session_id:3221589980, trace_id:YB42AC4D0007-00063C496359B261-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:251}, err:0, last_decode_time:1755161748778070, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1681, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:03:57.750559] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=26] [sql nio session](*s={this:0x7f98f7332030, session_id:3221590132, trace_id:YB42AC4D0007-00063C496359B265-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:266}, err:0, last_decode_time:1755161768783802, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:2125, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:03:57.751264] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:57.751485] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:57.751519] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=4377] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:57.754258] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:57.758249] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=19] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:57.749698] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=20] [sql nio session](*s={this:0x7f98f72a9030, session_id:3221590528, trace_id:YB42AC4D0007-00063C4965DA1F34-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:250}, err:0, last_decode_time:1755161762715415, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:205, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:03:57.761695] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:57.769537] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1121] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:57.773988] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:57.774366] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=990147, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=32, WASH_BLOCK_END=2307, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=962026], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:56.767249] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:57.773539] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:03:57.776875] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:57.779129] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:57.787466] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=3803] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:03:57.787912] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:57.788710] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=188] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:57.792125] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=162] current monitor number(seq_=-1) [2025-08-14 09:03:57.805401] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=222] gc stale ls task succ [2025-08-14 09:03:57.818877] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:57.848518] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=368][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:57.849654] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:57.897483] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=217] PNIO [ratelimit] time: 1755162237897481, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007637, add_bytes: 0 [2025-08-14 09:03:57.918937] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:03:57.807703] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:57.933717] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [571][T1_ArcSender][T1][YB42AC4D0007-00063C495BA99C16-0-0] [lt=25][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:03:57.776314] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=2762] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:8607757, max_lsn:1604090331, max_scn:{val:1755161756730847001, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:1556737504}, reuse_lsn:{lsn:1604090331}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:1562147291}}, last_submit_lsn:{lsn:1604090203}, last_submit_end_lsn:{lsn:1604090331}, last_submit_log_id:8607757, last_submit_log_pid:50, max_flushed_lsn:{lsn:1604090203}, max_flushed_end_lsn:{lsn:1604090331}, max_flushed_log_pid:50, committed_end_lsn:{lsn:1604090090}, last_slide_log_id:8607755, last_slide_scn:{val:1755161749741193000, v:0}, last_slide_lsn:{lsn:1604089968}, last_slide_end_lsn:{lsn:1604090090}, last_slide_log_pid:50, last_slide_log_accum_checksum:1446576759, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f990bfe4530}, [StateMgr]={this:0x7f990bfe89b0, self:"172.77.0.7:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"PENDING", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:51}, leader:"0.0.0.0:0", leader_epoch:-1, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:1604090331}, scan_disk_log_finished:true, last_check_start_id:8607756, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161901828039, palf_role_change_cb:0x7f990bfef470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:50, prev:{version:1, config:{config_version:{proposal_id:49, config_seq:45}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:50, config_seq:46}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_mode_pid:50}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_end_lsn:{lsn:1604088821}, prev_mode_pid:50}, reconfig_barrier:{prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_end_lsn:{lsn:1604088821}, prev_mode_pid:50}, persistent_config_version:{proposal_id:50, config_seq:46}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755161738854289, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755161738854289, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f990bfe5c70}, [ModeMgr]={palf_id:1, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:50, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:1543409664}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:51}, log_config_meta:{version:3, proposal_id:50, prev:{version:1, config:{config_version:{proposal_id:49, config_seq:45}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:50, config_seq:46}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_mode_pid:50}, log_snapshot_meta:{version:2, base_lsn:{lsn:1543409664}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:712704}, readable_log_tail:{lsn:712704}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:1610514432}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/meta", dir_fd:225, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66392064, block_header_serialize_buf_:0x7f990bfeabb8, flashback_version:0}, log_storage:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1755161901828095, this:0x7f990bfe90f0}, [Reconfirm]={palf_id:1, self:"172.77.0.7:2882", state:"RECONFIRM_MODE_META", new_proposal_id:51, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"172.77.0.7:2882", majority_max_accept_pid:50, majority_max_lsn:{lsn:1604090331}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:1755161901828373, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f990bfe8b30}) [2025-08-14 09:03:57.973637] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=977116, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=977114], seq:[0] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=977214, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=35, WASH_BLOCK_END=39, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=977133], seq:[0, 1, 2, 3, 4] [413]OB_MALLOC COST TOO MUCH TIME, cost_time=960083, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=960074, ALLOC_CHUNK_END=2], seq:[0, 1] [2025-08-14 09:03:57.739268] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:56.747610] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=1364][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1973930, owner: ObLog, click_count: 8, time dist:[ALLOC_BLOCK#start=8990, ALLOC_BLOCK#end=4, FORMAT_END=1964703, ObMalloc#start=2, ObMalloc#end=69, ObMalloc#start=24, ObMalloc#end=74, FREE_END=64], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:56.771316] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=985185][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1950303, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1950033, ObMalloc#start=1, ObMalloc#end=75, ObMalloc#start=34, ObMalloc#end=64, FREE_END=96], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:55.751411] EDIAG [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=3977439][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2979668, owner: ObLog, click_count: 6, time dist:[FORMAT_END=2978508, ObMalloc#start=1, ObMalloc#end=76, ObMalloc#start=24, ObMalloc#end=48, FREE_END=1011], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b51df0f 0x986bcde 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [856]OB_MALLOC COST TOO MUCH TIME, cost_time=741405, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=30, WASH_BLOCK_END=741375], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=589043, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=44, WASH_BLOCK_END=588974, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=25], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:57.758595] EDIAG [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=956112][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1004993, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1004322, ObMalloc#start=1, ObMalloc#end=35, ObMalloc#start=28, ObMalloc#end=35, FREE_END=572], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0xfd250b6 0xfd247d3 0xfd244ea 0xfd242a2 0x1d19629b 0x1d17baa3 0x1d17ad94 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:56.759902] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=32181][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2002376, owner: ObLog, click_count: 10, time dist:[ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=72, ALLOC_BLOCK#start=11352, ALLOC_BLOCK#end=48, FORMAT_END=1990525, ObMalloc#start=1, ObMalloc#end=63, ObMalloc#start=22, ObMalloc#end=45, FREE_END=243], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:58.773394] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=974449] log compressor cycles once. (ret=0, cost_time=70035, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25897234432, fast_delete_log_mode=false) [2025-08-14 09:03:57.758432] EDIAG [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [413][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=8720][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1014828, owner: ObLog, click_count: 8, time dist:[ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=960094, FORMAT_END=40, ObMalloc#start=1, ObMalloc#end=33967, ObMalloc#start=32, ObMalloc#end=20632, FREE_END=59], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a5bbf2 0x94da542 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:57.751561] EDIAG [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=958016][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1021722, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1000670, ObMalloc#start=1, ObMalloc#end=20846, ObMalloc#start=25, ObMalloc#end=65, FREE_END=115], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0xfd250b6 0xfd247d3 0xfd244ea 0xfd242a2 0x1d19629b 0x1d17baa3 0x1d17ad94 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:56.758329] EDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18046][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2013850, owner: ObLog, click_count: 9, time dist:[ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=79, ALLOC_BLOCK#start=22796, ALLOC_BLOCK#end=48, FORMAT_END=1990916, ObMalloc#start=0, ObMalloc#end=3, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:56.900136] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=2794048][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1872787, owner: ObLog, click_count: 8, time dist:[FORMAT_END=1872484, ObMalloc#start=1, ObMalloc#end=52, ObMalloc#start=18, ObMalloc#end=54, ObMalloc#start=74, ObMalloc#end=43, FREE_END=61], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:56.995586] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1777522, owner: ObLog, click_count: 8, time dist:[FORMAT_END=1777000, ObMalloc#start=1, ObMalloc#end=76, ObMalloc#start=33, ObMalloc#end=48, ObMalloc#start=156, ObMalloc#end=54, FREE_END=154], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:56.758329] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18046][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:58.775267] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2013850] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:58.775282] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:58.779947] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:58.784435] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=1693521, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=53, WASH_BLOCK_END=699821, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=51788], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:58.019248] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1693584, owner: ObLog, click_count: 5, time dist:[FORMAT_END=26, ObMalloc#start=0, ObMalloc#end=1693551, ALLOC_END=2, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:58.019248] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:59.715424] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1693584][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=927394, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=31, WASH_BLOCK_END=927363], seq:[0, 1, 2] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=926777, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=54, WASH_BLOCK_END=926723], seq:[0, 1, 2] [388]OB_MALLOC COST TOO MUCH TIME, cost_time=926148, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=38, WASH_BLOCK_END=926109], seq:[0, 1, 2] [2025-08-14 09:03:58.788255] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:59.715888] WDIAG ev_invoke_pending (ev.c:2136) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=926230][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=925905, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=30, WASH_BLOCK_END=925875], seq:[0, 1, 2] [2025-08-14 09:03:58.788652] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=924701, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=51, WASH_BLOCK_END=924650], seq:[0, 1, 2] [2025-08-14 09:03:58.790190] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=905628, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=33, WASH_BLOCK_END=905594], seq:[0, 1, 2] [656]OB_MALLOC COST TOO MUCH TIME, cost_time=865986, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=56, WASH_BLOCK_END=865930], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=861331, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=43, WASH_BLOCK_END=861288], seq:[0, 1, 2] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=852857, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=34, WASH_BLOCK_END=852823], seq:[0, 1, 2] [466]OB_MALLOC COST TOO MUCH TIME, cost_time=827301, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=55, WASH_BLOCK_END=827246], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=810574, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=43, WASH_BLOCK_END=810531], seq:[0, 1, 2] [808]OB_MALLOC COST TOO MUCH TIME, cost_time=744933, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=43, WASH_BLOCK_END=744889], seq:[0, 1, 2] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=735642, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=51, WASH_BLOCK_END=735591], seq:[0, 1, 2] [2025-08-14 09:03:58.809444] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=696409, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=36, WASH_BLOCK_END=696373], seq:[0, 1, 2] [2025-08-14 09:03:58.980248] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=1872787][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:58.970854] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:59.722415] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=243] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [567]OB_MALLOC COST TOO MUCH TIME, cost_time=977446, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=977445], seq:[0] [2025-08-14 09:03:56.767249] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1006968, owner: ObLog, click_count: 5, time dist:[FORMAT_END=16796, ObMalloc#start=1, ObMalloc#end=990164, ALLOC_END=2, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [857]OB_MALLOC COST TOO MUCH TIME, cost_time=969407, tenant_id=1002, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1], seq:[0, 1] [729]OB_MALLOC COST TOO MUCH TIME, cost_time=967208, tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=996707, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=50, WASH_BLOCK_END=7580, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=977462], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:57.775624] EDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3007034][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1970258, owner: ObLog, click_count: 5, time dist:[FORMAT_END=973526, ObMalloc#start=1, ObMalloc#end=996722, ALLOC_END=2, APPEND_END=7], seq:[0, 1, 2, 3, 4], BACKTRACE: 0xfa06846 0xfa05f62 0xfa05c84 0xfa05a50 0x1c020b34 0x97a4183 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:57.775624] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3007034] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161101770923, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755161572801635) [2025-08-14 09:03:59.745728] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=263] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:59.750752] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=707] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:59.750798] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=509] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:59.751601] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=758] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:59.754262] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4926] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:57.958513] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1795196, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=6, ALLOC_BLOCK#end=4, FORMAT_END=1795177, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:57.958513] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:58.740405] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1014785, owner: ObLog, click_count: 9, time dist:[ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=17991, ALLOC_BLOCK#start=17539, ALLOC_BLOCK#end=83, ALLOC_BLOCK#start=10067, ALLOC_BLOCK#end=53, FORMAT_END=969044, ALLOC_END=0, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:57.751098] EDIAG [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=954827][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1012539, owner: ObLog, click_count: 8, time dist:[FORMAT_END=1011680, ObMalloc#start=1, ObMalloc#end=56, ObMalloc#start=24, ObMalloc#end=36, FREE_END=742, ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=10620], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0xfd250b6 0xfd247d3 0xfd244ea 0xfd242a2 0x1d19629b 0x1d17baa3 0x1d17ad94 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:55.750530] EDIAG [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=2001611][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2986731, owner: ObLog, click_count: 10, time dist:[FORMAT_END=2985259, ObMalloc#start=1, ObMalloc#end=63, ObMalloc#start=23, ObMalloc#end=46, FREE_END=1339, ALLOC_BLOCK#start=18445, ALLOC_BLOCK#end=74, ALLOC_BLOCK#start=15868, ALLOC_BLOCK#end=113], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b51df0f 0x986bcde 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [477]OB_MALLOC COST TOO MUCH TIME, cost_time=971987, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=39, WASH_BLOCK_END=927376, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40004], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:56.962989] EDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=606823][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 2793046, , BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x9757ced 0x97569df 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:59.737086] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [719][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:58.740405] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:59.182820] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:59.756746] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1014785][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:59.756764] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:58.805464] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=25] gc stale ls task succ [2025-08-14 09:03:59.739429] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=236] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:59.760715] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21252][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:52.747269] EDIAG [ELECT] operator() (election_proposer.cpp:232) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 7003215, owner: ObLog, click_count: 5, time dist:[FORMAT_END=6993035, ObMalloc#start=1, ObMalloc#end=10174, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xfa07f77 0xfa07a14 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:58.715869] EDIAG [RS] skip_check_member_list_change_ (ob_ls_recovery_stat_handler.cpp:109) [777][T1002_TeRec][T1002][YB42AC4D0007-00063C495DDA0064-0-0] [lt=925365][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1043986, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=6, ALLOC_BLOCK#end=35, FORMAT_END=1039038, ObMalloc#start=1, ObMalloc#end=4882, ALLOC_END=21, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x1067fb2c 0x1067ecc2 0x1067e03d 0x1002f2a3 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:59.758231] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=996720][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:59.760890] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2643] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:59.761128] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:58.751291] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1009017, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:58.745142] EDIAG [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=25959][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1015788, , BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0xfb85bcd 0xfb858a5 0x993f72e 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:58.719964] EDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=979213][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1041637, owner: ObLog, click_count: 8, time dist:[FORMAT_END=1034941, ObMalloc#start=1, ObMalloc#end=4826, ObMalloc#start=21, ObMalloc#end=57, ObMalloc#start=201, ObMalloc#end=215, FREE_END=1374], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:58.759743] EDIAG [RS.COMPACTION] try_idle (ob_major_merge_scheduler.cpp:226) [1275][T1001_MergeSche][T1001][YB42AC4D0007-00063C4963E99CBE-0-0] [lt=13][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1966992, owner: ObLog, click_count: 8, time dist:[ALLOC_BLOCK#start=16531, ALLOC_BLOCK#end=93, FORMAT_END=1950015, ObMalloc#start=1, ObMalloc#end=105, ObMalloc#start=31, ObMalloc#end=60, FREE_END=156], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1125d502 0x1125cf81 0x1125a4c3 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:58.774849] EDIAG [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=1021722][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1962310, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1961998, ObMalloc#start=0, ObMalloc#end=69, ObMalloc#start=0, ObMalloc#end=57, FREE_END=186], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1d17bb52 0x1d17ada0 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:58.765147] EDIAG [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=1004993][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1972080, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1971826, ObMalloc#start=1, ObMalloc#end=70, ObMalloc#start=0, ObMalloc#end=65, FREE_END=118], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1d17bb52 0x1d17ada0 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:59.761576] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=152] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:59.761693] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=986837] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:03:59.762444] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=994418][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:59.762514] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1015788][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:59.762526] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:03:59.762712] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=442] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:03:59.762805] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:03:59.767386] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:00.361609] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=493835] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:03:59.763402] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=887] finish to resize log external storage handler(current_ts=1755162239763397, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-08-14 09:04:00.722021] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:00.731454] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=953587] snapshot_traffic_info once [2025-08-14 09:04:00.740109] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:00.741898] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:00.750254] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:58.754546] EDIAG [RS.COMPACTION] try_idle (ob_major_merge_scheduler.cpp:226) [1057][T1_MergeSchedul][T1][YB42AC4D0007-00063C4961599CC3-0-0] [lt=24][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1993103, owner: ObLog, click_count: 9, time dist:[ALLOC_BLOCK#start=28192, ALLOC_BLOCK#end=72, ALLOC_BLOCK#start=10037, ALLOC_BLOCK#end=71, FORMAT_END=1954722, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1125d502 0x1125cf81 0x1125a4c3 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:00.750908] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=13] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:04:00.751312] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1260] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:00.753264] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:00.759101] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=1860] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:00.759576] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:00.759957] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [968]OB_MALLOC COST TOO MUCH TIME, cost_time=996154, tenant_id=1002, label=SqlExecutor, ctx_id=0, prio=0, size=245760, [2025-08-14 09:03:59.764734] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=2979668] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:04:00.760707] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=994461] tx gc loop thread is running(MTL_ID()=1001) [2025-08-14 09:04:00.760720] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=11] try gc retain ctx [2025-08-14 09:04:00.760718] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=64] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:00.760729] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=9] try to retry start_working [2025-08-14 09:04:00.760757] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=357] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:00.762292] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=345] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:00.762806] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=10] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:04:00.764458] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=731] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:00.765723] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=992] gc stale ls task succ [2025-08-14 09:04:00.747802] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:04:00.766342] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18498] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:04:00.767556] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:04:00.767592] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=30] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:04:00.768178] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:00.746353] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=23736] try to retry start_working [2025-08-14 09:04:00.733483] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=21907] [COMPACTION DAG STATUS] (start_time=1755162222727883, end_time=1755162240733482, dag_status={}) [2025-08-14 09:04:00.739402] WDIAG [SQL] open_plan (ob_result_set.cpp:104) [801][T1002_TntTransf][T1002][YB42AC4D0007-00063C496259A60F-0-0] [lt=1981080][errcode=-4012] query is timeout(ret=-4012, timeout_ts=1755162058713703, start_time=1755162028750810) [2025-08-14 09:04:00.770674] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:00.770716] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=16483] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:04:00.771419] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=41689] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:04:00.771508] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=38979] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:04:00.771554] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=1973930] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:04:00.776257] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=203] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:00.779254] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=222] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:00.789280] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=251] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:00.792605] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=214] current monitor number(seq_=-1) [2025-08-14 09:04:00.798872] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:00.801524] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:00.806318] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=234] gc stale ls task succ [2025-08-14 09:04:00.815826] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=409][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:00.818001] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:00.818414] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:00.819820] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:00.835215] INFO [PALF] check_parent_health (log_config_mgr.cpp:2875) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=23] re_register_parent reason(palf_id=1, self="172.77.0.7:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1755162235733516, last_first_register_time_us=1755158728414267, register_time_us=1755162235733516, curr_time_us=1755162240835209) [2025-08-14 09:04:00.915869] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:00.920288] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=235] PNIO [ratelimit] time: 1755162240920286, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007618, add_bytes: 0 [2025-08-14 09:04:00.920313] INFO pn_print_diag_info (group.c:775) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=24] PNIO client_send:0/0, client_queue_time=0, cnt=0, server_send:0/0, server_queue_time=0, cnt=0 [2025-08-14 09:04:00.991459] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:01.012849] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:01.015930] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:01.115990] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:01.216049] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:01.219391] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:01.316108] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:01.362279] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=27] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:04:01.416180] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:01.422411] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=36][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:04:01.423245] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=22][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:04:01.516261] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:01.522500] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=25] ObArchiveSender is running(thread_index=0) [2025-08-14 09:04:01.616322] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:01.699569] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=75025][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:01.716385] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:01.720944] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:00.770763] INFO [STORAGE.TRANS] refresh_elr_tenant_config_ (ob_tx_elr_util.cpp:42) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=9] refresh tenant config success(tenant_id=1001, *this={last_refresh_ts:1755162240770532, can_tenant_elr:true}) [2025-08-14 09:04:01.726501] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=954280] get_max_applied_scn(scn={val:1755161719775179002, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:45, palf_committed_end_lsn_:{lsn:8461098908}, palf_committed_end_scn_:{val:1755161719775179002, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161719775179002, v:0}}) [2025-08-14 09:04:01.727292] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=5858] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:04:01.726200] WDIAG [SQL] open (ob_result_set.cpp:137) [801][T1002_TntTransf][T1002][YB42AC4D0007-00063C496259A60F-0-0] [lt=30590][errcode=-4012] execute plan failed(ret=-4012) [2025-08-14 09:04:01.737890] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=50] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:01.739844] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:01.741724] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=785] gc stale ls task succ [2025-08-14 09:04:01.751537] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:04:01.751614] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:01.754300] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:01.754335] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:01.759462] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=353] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:01.759509] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=53] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:01.759672] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=1053620] get_max_applied_scn(scn={val:1755161749741193000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:50, palf_committed_end_lsn_:{lsn:1604090090}, palf_committed_end_scn_:{val:1755161749741193000, v:0}, last_check_scn_:{val:1755161749741193001, v:0}, max_applied_cb_scn_:{val:1755161749741193000, v:0}}) [2025-08-14 09:04:01.738455] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=510] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:01.759763] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21274][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:01.758586] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [801][T1002_TntTransf][T1002][YB42AC4D0007-00063C496259A60F-0-0] [lt=8694][errcode=-4012] open result set failed(ret=-4012) [2025-08-14 09:04:01.760085] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:01.760755] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:01.760771] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:01.762916] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=31] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:04:01.764393] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:01.764666] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:01.766383] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:04:01.767043] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:01.771313] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=16] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:04:01.775690] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:01.776140] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:01.764316] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=1972080] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:01.762550] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=983804] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:01.734478] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=36255] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:04:01.788290] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=53728] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:04:01.789324] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:01.792705] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=22] current monitor number(seq_=-1) [2025-08-14 09:04:01.799420] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:01.805815] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:01.806415] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=21] gc stale ls task succ [705]OB_MALLOC COST TOO MUCH TIME, cost_time=924621, tenant_id=1001, label=OB_LOG_REGISTER_PARENT_REQ, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [808]OB_MALLOC COST TOO MUCH TIME, cost_time=899638, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=46, WASH_BLOCK_END=899591], seq:[0, 1, 2] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=854234, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=58, WASH_BLOCK_END=854176], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=791049, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=52, WASH_BLOCK_END=790997], seq:[0, 1, 2] [400]OB_MALLOC COST TOO MUCH TIME, cost_time=757754, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=43, WASH_BLOCK_END=757711], seq:[0, 1, 2] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=722078, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=41, WASH_BLOCK_END=722037], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=356855, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=49, WASH_BLOCK_END=356806], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=629816, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=60, WASH_BLOCK_END=629756], seq:[0, 1, 2] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=499750, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=53, WASH_BLOCK_END=499697], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=911587, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=33, WASH_BLOCK_END=900346, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=11207], seq:[0, 1, 2, 3, 4] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=942044, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=51, WASH_BLOCK_END=932587, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=9406], seq:[0, 1, 2, 3, 4] [717]OB_MALLOC COST TOO MUCH TIME, cost_time=953270, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=953268], seq:[0] [466]OB_MALLOC COST TOO MUCH TIME, cost_time=915307, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=49, WASH_BLOCK_END=900068, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=9186], seq:[0, 1, 2, 3, 4] [2025-08-14 09:04:01.818577] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:02.737916] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=52] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:02.738812] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=977338] get wrs ts(ls_id={id:1}, delta=522962819, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:04:01.780222] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:02.739037] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=957365][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:02.740334] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:02.748285] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=987087] get wrs ts(ls_id={id:1}, delta=492996805, log_service_ts=1755161749741193, min_tx_service_ts=4611686018427387, timestamp={val:1755161749741193000, v:0}) [2025-08-14 09:04:02.740360] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=15964] temporary file module use no blocks [2025-08-14 09:04:01.734312] EDIAG [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1013381, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1013371, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=1, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xff22be4 0x97a9cd7 0x97a9238 0x97a9307 0x97a8fe2 0x986bc46 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:01.734312] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=21] get_min_unreplayed_log_info(lsn={lsn:8461098908}, scn={val:1755100947600130001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161920778323, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:23084275, ret_code_:0}}, next_to_submit_lsn_:{lsn:8461098908}, next_to_submit_scn_:{val:1755100947600130001, v:0}, base_lsn_:{lsn:7515734016}, base_scn_:{val:1753840867171901045, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6e05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:8461098908}, end_lsn:{lsn:8461098908}, read_buf:{buf_len_:3747840, buf_:0x7f98f6e05000}, block_size:67104768, log_storage_:0x7f99297a3bf0, log_storage_:{log_tail:{lsn:8461098908}, readable_log_tail:{lsn:8461098908}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8455200768}, min_scn:{val:1755160384851871002, v:0}, curr_block_id:126, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:215, min_block_id:93, max_block_id:127, curr_writable_block_id:126}, logical_block_size_:67104768, curr_block_writable_size_:61206628, block_header_serialize_buf_:0x7f99297a48b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:198, read_io_size:699822080, read_disk_cost_ts:4178035, start_lsn:{lsn:7515734016}}}}}}) [2025-08-14 09:04:02.754396] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [717]OB_MALLOC COST TOO MUCH TIME, cost_time=975648, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=57, WASH_BLOCK_END=63, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=953284], seq:[0, 1, 2, 3, 4] [2025-08-14 09:04:01.764371] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=1962310] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:02.755491] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=989616] temporary file module use no blocks [2025-08-14 09:04:02.755837] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=22346] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:04:02.755851] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:04:02.759109] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=32] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:02.760359] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:01.727764] EDIAG [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=23398][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1032098, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1032090, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xff22be4 0x97a9cd7 0x97a9238 0x97a9307 0x97a8fe2 0x986bc46 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:01.727764] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=23398] get_min_unreplayed_log_info(lsn={lsn:5282737655}, scn={val:1755100947825033001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161624848121, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:13063726, ret_code_:0}}, next_to_submit_lsn_:{lsn:5282737655}, next_to_submit_scn_:{val:1755100947825033001, v:0}, base_lsn_:{lsn:4831543296}, base_scn_:{val:1755063445733337001, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7605000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:5282737655}, end_lsn:{lsn:5282737655}, read_buf:{buf_len_:3747840, buf_:0x7f98f7605000}, block_size:67104768, log_storage_:0x7f993e529bf0, log_storage_:{log_tail:{lsn:5282737655}, readable_log_tail:{lsn:5282737655}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:5234171904}, min_scn:{val:1755152244802535004, v:0}, curr_block_id:78, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:206, min_block_id:42, max_block_id:79, curr_writable_block_id:78}, logical_block_size_:67104768, curr_block_writable_size_:18539017, block_header_serialize_buf_:0x7f993e52a8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:85, read_io_size:301367296, read_disk_cost_ts:2364604, start_lsn:{lsn:4831543296}}}}}}) [2025-08-14 09:04:02.739063] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14170] temporary file module use no blocks [2025-08-14 09:04:02.762229] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:02.762264] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:02.763218] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=20] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:04:02.763266] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=29] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:02.764666] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=53] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:02.766443] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=115] gc stale ls task succ [2025-08-14 09:04:02.768125] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:02.759255] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:02.776382] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:02.779149] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:02.779940] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:02.789341] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:02.802729] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:02.806468] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=21] gc stale ls task succ [2025-08-14 09:04:02.834412] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:02.935526] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=802276] PNIO [ratelimit] time: 1755162242935523, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007623, add_bytes: 0 [382]OB_MALLOC COST TOO MUCH TIME, cost_time=974168, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=974166], seq:[0] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=496964, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=39, WASH_BLOCK_END=496925], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=296941, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=45, WASH_BLOCK_END=296869, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=26], seq:[0, 1, 2, 3, 4] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=632403, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=41, WASH_BLOCK_END=632331, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=30], seq:[0, 1, 2, 3, 4] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=983207, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=54, WASH_BLOCK_END=48, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=974192], seq:[0, 1, 2, 3, 4] [2025-08-14 09:04:02.753686] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:03.738561] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=983387][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:03.732824] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=975844] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:04:03.738658] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=5825] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623143179967, flush_io_finished_ret_:0, flush_io_finished_round_:26349, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:03.741413] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [822]OB_MALLOC COST TOO MUCH TIME, cost_time=969242, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=969241], seq:[0] [822]OB_MALLOC COST TOO MUCH TIME, cost_time=971951, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=62, WASH_BLOCK_END=2624, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=969261], seq:[0, 1, 2, 3, 4] [2025-08-14 09:04:01.755691] EDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1984458, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1012482, ObMalloc#start=1, ObMalloc#end=971964, ALLOC_END=5, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0xfc0e73d 0x993f560 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:03.732811] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=969812] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:04:03.741912] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9087] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623143180009, flush_io_finished_ret_:0, flush_io_finished_round_:26329, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:01.755691] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1001, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:8571658, max_lsn:7323687369, max_scn:{val:1755161749741193002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:7278610047}, reuse_lsn:{lsn:7323686131}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:7281744329}}, last_submit_lsn:{lsn:7323686131}, last_submit_end_lsn:{lsn:7323687369}, last_submit_log_id:8571658, last_submit_log_pid:49, max_flushed_lsn:{lsn:7323686009}, max_flushed_end_lsn:{lsn:7323686131}, max_flushed_log_pid:49, committed_end_lsn:{lsn:7323686009}, last_slide_log_id:8571656, last_slide_scn:{val:1755161747742105001, v:0}, last_slide_lsn:{lsn:7323685887}, last_slide_end_lsn:{lsn:7323686009}, last_slide_log_pid:49, last_slide_log_accum_checksum:2505458373, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f990112a530}, [StateMgr]={this:0x7f990112e9b0, self:"172.77.0.7:2882", palf_id:1001, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"PENDING", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:49}, leader:"0.0.0.0:0", leader_epoch:-1, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:7323687369}, scan_disk_log_finished:true, last_check_start_id:8571657, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161767744317, palf_role_change_cb:0x7f9901135470, allow_vote:true}, [ConfigMgr]={palf_id:1001, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:49, prev:{version:1, config:{config_version:{proposal_id:48, config_seq:47}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:49, config_seq:48}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_mode_pid:49}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_end_lsn:{lsn:7323685240}, prev_mode_pid:49}, reconfig_barrier:{prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_end_lsn:{lsn:7323685240}, prev_mode_pid:49}, persistent_config_version:{proposal_id:49, config_seq:48}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755161739755312, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755161739755312, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f990112bc70}, [ModeMgr]={palf_id:1001, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, accepted_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1001, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1753069671378640001, v:0}, base_lsn_for_block_gc:{lsn:7247314944}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:49}, log_config_meta:{version:3, proposal_id:49, prev:{version:1, config:{config_version:{proposal_id:48, config_seq:47}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:49, config_seq:48}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_mode_pid:49}, log_snapshot_meta:{version:2, base_lsn:{lsn:7247314944}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1753069671378640000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}}, log_meta_storage:{log_tail:{lsn:765952}, readable_log_tail:{lsn:765952}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7381524480}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/meta", dir_fd:232, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66338816, block_header_serialize_buf_:0x7f9901130bb8, flashback_version:0}, log_storage:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, palf_epoch:2, last_purge_throttling_ts:1755161769777679, this:0x7f990112f0f0}, [Reconfirm]={palf_id:1001, self:"172.77.0.7:2882", state:"WAITING_LOG_FLUSHED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:1755161769777679, this:0x7f990112eb30}) [2025-08-14 09:04:03.743182] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1984458] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [2025-08-14 09:04:03.743201] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15] current log_task status(palf_id=1001, self="172.77.0.7:2882", log_id=8571657, log_task={header:{begin_lsn:{lsn:7323686009}, end_lsn:{lsn:7323686131}, log_id:8571657, min_scn:{val:1755161749741193000, v:0}, max_scn:{val:1755161749741193000, v:0}, data_len:66, proposal_id:49, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:7323686009}, data_checksum:2222604892, accum_checksum:1531236791, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1755161749745909, freeze_ts:1755161749745914, submit_ts:1755161749745940, flushed_ts:1755161785758706, gen_to_freeze cost time:5, gen_to_submit cost time:31, submit_to_flush cost time:36012766}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [2025-08-14 09:04:01.816438] EDIAG [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1925162, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=7, ALLOC_BLOCK#end=3, FORMAT_END=1925148, ObMalloc#start=0, ObMalloc#end=1, ALLOC_END=0, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1b52e1b7 0x996d441 0x996cfd8 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:02.769640] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10368][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:01.816438] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:04:03.744534] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1925162] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:04:03.744547] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:03.744576] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:03.751281] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:03.751316] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=449] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:03.020412] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=735702][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:03.754259] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1525] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [660]OB_MALLOC COST TOO MUCH TIME, cost_time=732127, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=42, WASH_BLOCK_END=707070, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2456], seq:[0, 1, 2, 3, 4] [2025-08-14 09:04:03.022090] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:01.821851] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1930666, owner: ObLog, click_count: 9, time dist:[FORMAT_END=1920148, ObMalloc#start=0, ObMalloc#end=50, ObMalloc#start=18, ObMalloc#end=44, ObMalloc#start=10400, ObMalloc#end=2, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:01.821851] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:02.753956] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1000056, owner: ObLog, click_count: 6, time dist:[FORMAT_END=989145, ObMalloc#start=1, ObMalloc#end=62, ObMalloc#start=19, ObMalloc#end=45, FREE_END=10784], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:01.818999] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1933644, owner: ObLog, click_count: 9, time dist:[FORMAT_END=1922850, ObMalloc#start=0, ObMalloc#end=53, ObMalloc#start=18, ObMalloc#end=52, ObMalloc#start=10666, ObMalloc#end=2, ALLOC_END=0, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:01.818999] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:02.729328] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1008) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:03.756637] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=26] LogLoopThread round_cost_time(us)(round_cost_time=2875631) [2025-08-14 09:04:03.760339] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:03.244647] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:03.762227] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=1096] gc stale ls task succ [2025-08-14 09:04:03.762297] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=985133] blacklist_loop exec finished(cost_time=22, is_enabled=true, send_cnt=0) [2025-08-14 09:04:03.762357] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:03.762741] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=41] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:03.763420] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=32] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:04:03.763955] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=869][errcode=-4015] clock out of order(cur_ts=1755162243763954, cur_ts_=1755162243738749, delta=25205) [2025-08-14 09:04:02.720287] EDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:801) [801][T1002_TntTransf][T1002][YB42AC4D0007-00063C496259A60F-0-0] [lt=1390][errcode=-4012] LOGGER COST TOO MUCH TIME, cost: 1073135, owner: ObLog, click_count: 8, time dist:[FORMAT_END=1072737, ObMalloc#start=0, ObMalloc#end=38, ObMalloc#start=25, ObMalloc#end=36, ObMalloc#start=209, ObMalloc#end=52, FREE_END=38], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x124de942 0x96a686c 0x96a5033 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x9a64aaf 0x10502edb 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:02.749465] EDIAG [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=970345][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1048256, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1044502, ObMalloc#start=1, ObMalloc#end=3082, ObMalloc#start=26, ObMalloc#end=37, FREE_END=608], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0xfa06846 0xfa05f62 0xfa05c84 0xfa05a50 0x1b51bd85 0x1b51b9c4 0x97a8d82 0x97a88d2 0x97a7481 0x97a6a43 0x97a42c5 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:02.755822] EDIAG [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1041950, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1041089, ObMalloc#start=1, ObMalloc#end=52, ObMalloc#start=23, ObMalloc#end=30, FREE_END=755], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0xfa06846 0xfa05f62 0xfa05c84 0xfa05a50 0x1b51bd85 0x1b51b9c4 0x97a8d82 0x97a88d2 0x97a7481 0x97a6a43 0x97a42c5 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:02.748314] EDIAG [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1049666, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1049471, ObMalloc#start=1, ObMalloc#end=42, ObMalloc#start=20, ObMalloc#end=37, FREE_END=95], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0xfa06846 0xfa05f62 0xfa05c84 0xfa05a50 0x1b51bd85 0x1b51b9c4 0x97a8d82 0x97a88d2 0x97a7481 0x97a6a43 0x97a42c5 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [397]OB_MALLOC COST TOO MUCH TIME, cost_time=958745, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=958737], seq:[0] [2025-08-14 09:04:03.765220] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=973361] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [397]OB_MALLOC COST TOO MUCH TIME, cost_time=969328, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=61, WASH_BLOCK_END=84, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=958777], seq:[0, 1, 2, 3, 4] [2025-08-14 09:04:04.720212] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=65048] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:04:03.763010] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:04.733861] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=969383][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:04.730329] WARN [LIB] alloc (ob_malloc_allocator.cpp:924) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=995986][errcode=-4013] [OOPS]: alloc failed reason is that server memory has reached the upper limit(server_hold: 4294156288, server_limit: 4294967296, alloc_size: 2097152) [2025-08-14 09:04:04.735320] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=12959][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:04.738267] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:04.736507] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=2800] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:04:04.751397] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:04.751408] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=8] temporary file module use no blocks [2025-08-14 09:04:04.751414] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=5] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:04:04.751422] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=7] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:04:04.751431] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623146181707, flush_io_finished_ret_:0, flush_io_finished_round_:26330, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:03.775875] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=449] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:04:04.752406] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:04.753467] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=446][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:04.744599] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1041950] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=524969397, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=252386, succ_cluster_heartbeat_count=1905, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161003593738002, v:0}, local_cluster_delta=1241150838, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:04:04.748033] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1048256] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=697009515, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=253241, succ_cluster_heartbeat_count=2213, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161000606550000, v:0}, local_cluster_delta=1244141481, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:04:04.759077] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10998] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161101770923, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755161572801635) [2025-08-14 09:04:04.759112] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] get_max_applied_scn(scn={val:1755161547738516001, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:50, palf_committed_end_lsn_:{lsn:5282737655}, palf_committed_end_scn_:{val:1755161547738516001, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161547738516001, v:0}}) [2025-08-14 09:04:04.759732] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=697018594, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:04:04.759749] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:04:04.764692] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=24] gc stale ls task succ [2025-08-14 09:04:04.764782] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2409] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:04.764906] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:04:04.766501] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:04.766538] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=285] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:04:04.766614] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:04.766767] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:04.766916] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14268] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623146182826, flush_io_finished_ret_:0, flush_io_finished_round_:26362, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:04.738211] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=486] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:04:04.752498] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:946) [801][T1002_TntTransf][T1002][YB42AC4D0007-00063C496259A60F-0-0] [lt=1073135][errcode=-4012] execute failed(ret=-4012, tenant_id=1002, executor={ObIExecutor:, sql:"SELECT task_id, status FROM __all_transfer_task"}, retry_cnt=12, local_sys_schema_version=1753073979411448, local_tenant_schema_version=1755156894100160) [2025-08-14 09:04:04.753189] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=508] get wrs ts(ls_id={id:1001}, delta=500024331, log_service_ts=1755161744727824, min_tx_service_ts=4611686018427387, timestamp={val:1755161744727824001, v:0}) [2025-08-14 09:04:04.768014] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14797] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:04:04.752999] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=984224][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:04.768147] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15110] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:04.754055] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=240] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:04.770362] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16282][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=978436, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=978395], seq:[0, 1, 2, 3, 4, 5, 6] [659]OB_MALLOC COST TOO MUCH TIME, cost_time=976890, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=976843, ALLOC_CHUNK_END=13], seq:[0, 1, 2, 3, 4, 5, 6, 7] [659]OB_MALLOC COST TOO MUCH TIME, cost_time=976939, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=976902, WASH_BLOCK_END=36], seq:[0, 1, 2] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=973965, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=973930, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=974004, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=973977, WASH_BLOCK_END=27], seq:[0, 1, 2] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=970371, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=970333, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3, 4, 5, 6, 7] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=970394, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=970386, WASH_BLOCK_END=8], seq:[0, 1, 2] [2025-08-14 09:04:05.736857] WDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=974269][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=971507, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=971470], seq:[0, 1, 2, 3, 4, 5, 6] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=971530, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=971529], seq:[0, 1] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=960213, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=960184, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [2025-08-14 09:04:04.764343] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=54] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:05.737368] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=971554][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=960244, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=960224, WASH_BLOCK_END=20], seq:[0, 1, 2] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=976995, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=976953, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=957556, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=957529, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=957579, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=957570, WASH_BLOCK_END=8], seq:[0, 1, 2] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=977040, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=977013, WASH_BLOCK_END=26], seq:[0, 1, 2] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=962459, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=962427, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=962479, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=962470, WASH_BLOCK_END=9], seq:[0, 1, 2] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=961461, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=961430, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=961481, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=961471, WASH_BLOCK_END=9], seq:[0, 1, 2] [388]OB_MALLOC COST TOO MUCH TIME, cost_time=949226, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=949194, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [388]OB_MALLOC COST TOO MUCH TIME, cost_time=949246, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=949237, WASH_BLOCK_END=9], seq:[0, 1, 2] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=930772, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=930740, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=931272, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=931253, WASH_BLOCK_END=18], seq:[0, 1, 2] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=917740, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=917699, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=872060, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=872033, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=872101, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=872077, WASH_BLOCK_END=24], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=880365, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=880323, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=880387, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=880375, WASH_BLOCK_END=12], seq:[0, 1, 2] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=919678, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=24, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=919631, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=919707, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=919695, WASH_BLOCK_END=12], seq:[0, 1, 2] [466]OB_MALLOC COST TOO MUCH TIME, cost_time=872143, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=872110, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [466]OB_MALLOC COST TOO MUCH TIME, cost_time=872159, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=872152, WASH_BLOCK_END=7], seq:[0, 1, 2] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=711172, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=711144, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [400]OB_MALLOC COST TOO MUCH TIME, cost_time=691463, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=691431, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=711191, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=711182, WASH_BLOCK_END=9], seq:[0, 1, 2] [400]OB_MALLOC COST TOO MUCH TIME, cost_time=691497, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=691481, WASH_BLOCK_END=16], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=476878, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=476844, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [2025-08-14 09:04:05.028015] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:05.047778] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=973][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=504337, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=504303, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=476899, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=476888, WASH_BLOCK_END=11], seq:[0, 1, 2] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=504357, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=504347, WASH_BLOCK_END=10], seq:[0, 1, 2] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=487109, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=487068, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3, 4, 5] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=487137, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=487120, WASH_BLOCK_END=17], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=281877, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=21, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=281830, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3, 4, 5] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=281910, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=281888, WASH_BLOCK_END=22], seq:[0, 1, 2] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=917761, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=917750, WASH_BLOCK_END=11], seq:[0, 1, 2] [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=945005, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=19, ALLOC_CHUNK_END=944958, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=945032, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=945017, WASH_BLOCK_END=15], seq:[0, 1, 2] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=901213, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=901182, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=901241, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=901227, WASH_BLOCK_END=14], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=789065, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=789037, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=789089, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=789075, WASH_BLOCK_END=14], seq:[0, 1, 2] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=982411, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=982370, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=982431, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=982422, WASH_BLOCK_END=9], seq:[0, 1, 2] [2025-08-14 09:04:01.758539] EDIAG [ELECT] advance_ballot_number_and_reset_related_states_ (election_proposer.cpp:210) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3976705, owner: ObLog, click_count: 6, time dist:[FORMAT_END=2994959, ObMalloc#start=0, ObMalloc#end=976948, ObMalloc#start=29, ObMalloc#end=4458, FREE_END=311], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9d4e62 0xf9cf0e7 0xf9d3ec1 0xfa09ca2 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [770]OB_MALLOC COST TOO MUCH TIME, cost_time=916099, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=916069, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=916119, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=916110, WASH_BLOCK_END=9], seq:[0, 1, 2] [729]OB_MALLOC COST TOO MUCH TIME, cost_time=982215, tenant_id=500, label=TenantCtxAlloca, ctx_id=0, prio=0, size=2944, [729]OB_MALLOC COST TOO MUCH TIME, cost_time=982334, tenant_id=500, label=CLIENT_REQ_CHUNK, ctx_id=17, prio=0, size=16480, owner: ALLOC_BLOCK, click_count: 19, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ObMalloc#end=982226, ALLOC_CHUNK_END=18, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=6], seq:[0, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 17, 18] [729]OB_MALLOC COST TOO MUCH TIME, cost_time=982370, tenant_id=500, label=CLIENT_REQ_CHUNK, ctx_id=17, prio=0, size=16288, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=982348, WASH_BLOCK_END=22], seq:[0, 1, 2] [2025-08-14 09:04:05.740424] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=3736] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:04:05.235287] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:04.838321] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] get wrs ts(ls_id={id:1}, delta=697098116, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:04:05.740988] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=901297] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:04:04.823778] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:05.741542] WDIAG ev_invoke_pending (ev.c:2136) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=950874][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:05.741567] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=963377][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:05.736626] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [720][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:05.740366] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3936] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [736]OB_MALLOC COST TOO MUCH TIME, cost_time=993568, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=978468], seq:[0, 1] [2025-08-14 09:04:04.756384] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=1000056] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:04:05.751528] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=19] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [477]OB_MALLOC COST TOO MUCH TIME, cost_time=984950, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=984949], seq:[0] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=984969, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=984965], seq:[0, 1] [2025-08-14 09:04:05.757930] INFO [PALF] register_parent_ (log_config_mgr.cpp:2707) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=185] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.77.0.7:2882", reason=1, leader="172.77.0.7:2882", register_time_us=1755162245747627, last_submit_register_req_time_us=1755162245747627, register_parent_reason=1) [2025-08-14 09:04:05.757950] INFO [PALF] check_parent_health (log_config_mgr.cpp:2884) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=18] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.77.0.7:2882") [2025-08-14 09:04:05.757543] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:05.769245] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=11] gc stale ls task succ [2025-08-14 09:04:05.775501] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:05.747430] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [564][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=28327] XA scheduler heartbeat task statistics(avg_time=283080) [2025-08-14 09:04:05.776549] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=488] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:04:05.780257] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=964382] gc stale ls task succ [717]OB_MALLOC COST TOO MUCH TIME, cost_time=976023, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=49, WASH_BLOCK_END=2940, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=12203], seq:[0, 1, 2, 3, 4] [2025-08-14 09:04:05.751231] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=2398] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:06.736001] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=983273] temporary file module use no blocks [2025-08-14 09:04:06.736015] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=14] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:04:06.736033] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=16] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:04:06.736049] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623147187085, flush_io_finished_ret_:0, flush_io_finished_round_:26351, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:05.903534] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=962654, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=962647], seq:[0] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=962675, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=962670], seq:[0, 1] [2025-08-14 09:04:05.780297] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1784] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:06.744454] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=962698][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [857]OB_MALLOC COST TOO MUCH TIME, cost_time=963782, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=963745], seq:[0, 1, 2, 3, 4, 5, 6] [857]OB_MALLOC COST TOO MUCH TIME, cost_time=963794, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=963793], seq:[0, 1] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=963602, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=963574, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [388]OB_MALLOC COST TOO MUCH TIME, cost_time=957042, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=957011, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [388]OB_MALLOC COST TOO MUCH TIME, cost_time=957065, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=957052, WASH_BLOCK_END=13], seq:[0, 1, 2] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=937415, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=15, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=937371, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=937432, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=937424, WASH_BLOCK_END=8], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=983524, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=983522], seq:[0] [2025-08-14 09:04:05.754583] INFO [STORAGE.TRANS] amend (ob_tenant_weak_read_server_version_mgr.cpp:190) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [WRS] tenant has no valid partition on current server, generate weak read snapshot version based on current timestamp(tenant_id=1002, epoch_tstamp_=1755162241759655, total_part_count_=2, valid_inner_part_count_=0, valid_user_part_count_=0, version_={val:1755161758221798000, v:0}, base_version_when_no_valid_partition={val:1755161758221798000, v:0}) [2025-08-14 09:04:06.745547] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [857][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=989460] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=488523746, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=251789, succ_cluster_heartbeat_count=1502, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161006924036004, v:0}, local_cluster_delta=1239821508, force_self_check=false, weak_read_refresh_interval=100000) [374]OB_MALLOC COST TOO MUCH TIME, cost_time=924542, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=924513], seq:[0, 1, 2, 3, 4] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=924551, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=924550], seq:[0, 1] [2025-08-14 09:04:05.820498] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:06.746339] WDIAG ev_invoke_pending (ev.c:2136) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=957262][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=902456, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=902428], seq:[0, 1, 2, 3, 4] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=902465, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=902464], seq:[0, 1] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=883085, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=883056, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=883103, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=883093, WASH_BLOCK_END=10], seq:[0, 1, 2] [2025-08-14 09:04:05.842773] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3108] get wrs ts(ls_id={id:1}, delta=698103182, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:04:06.746634] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=902491] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [472]OB_MALLOC COST TOO MUCH TIME, cost_time=803759, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=803732], seq:[0, 1, 2, 3, 4] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=803806, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=803799], seq:[0, 1] [2025-08-14 09:04:05.941769] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=787323, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=787286], seq:[0, 1, 2, 3, 4] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=787333, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=787332], seq:[0, 1] [2025-08-14 09:04:05.958392] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=789664] PNIO [ratelimit] time: 1755162245958391, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007644, add_bytes: 0 [2025-08-14 09:04:06.747002] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:04:06.747017] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [660]OB_MALLOC COST TOO MUCH TIME, cost_time=716713, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=716685], seq:[0, 1, 2, 3, 4] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=716722, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=716721], seq:[0, 1] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=669726, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=669698, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=669746, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=669735, WASH_BLOCK_END=11], seq:[0, 1, 2] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=505394, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=505363, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=505423, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=505408, WASH_BLOCK_END=14], seq:[0, 1, 2] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=488482, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=488454, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=488500, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=488491, WASH_BLOCK_END=9], seq:[0, 1, 2] [2025-08-14 09:04:06.029217] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=478573, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=478546, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=478592, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=478583, WASH_BLOCK_END=9], seq:[0, 1, 2] [861]OB_MALLOC COST TOO MUCH TIME, cost_time=224871, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=224838, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [861]OB_MALLOC COST TOO MUCH TIME, cost_time=224898, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=224881, WASH_BLOCK_END=17], seq:[0, 1, 2] [733]OB_MALLOC COST TOO MUCH TIME, cost_time=223274, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=223245, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [733]OB_MALLOC COST TOO MUCH TIME, cost_time=223292, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=223284, WASH_BLOCK_END=8], seq:[0, 1, 2] [836]OB_MALLOC COST TOO MUCH TIME, cost_time=234338, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=234295, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [836]OB_MALLOC COST TOO MUCH TIME, cost_time=234360, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=234351, WASH_BLOCK_END=9], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=206051, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=206021, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=206070, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=206062, WASH_BLOCK_END=8], seq:[0, 1, 2] [2025-08-14 09:04:06.740511] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=22] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:04:06.741034] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=945210] current monitor number(seq_=-1) [2025-08-14 09:04:06.748570] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27640][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:06.513315] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:06.750844] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=836] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:06.751596] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=993612] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:04:06.752277] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4853] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [770]OB_MALLOC COST TOO MUCH TIME, cost_time=990814, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=983530], seq:[0, 1] [2025-08-14 09:04:05.741350] EDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 1010765, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=10730, FORMAT_END=9198, ObMalloc#start=2, ObMalloc#end=990827, ALLOC_END=0, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1b53d414 0x97ad9aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:06.753271] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=207] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:05.741350] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162245741349, start=1755162244823776, delta=917573) [2025-08-14 09:04:06.753666] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1010765][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:06.753865] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] get_min_unreplayed_log_info(lsn={lsn:5282737655}, scn={val:1755100947825033001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161624848121, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:13063726, ret_code_:0}}, next_to_submit_lsn_:{lsn:5282737655}, next_to_submit_scn_:{val:1755100947825033001, v:0}, base_lsn_:{lsn:4831543296}, base_scn_:{val:1755063445733337001, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7605000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:5282737655}, end_lsn:{lsn:5282737655}, read_buf:{buf_len_:3747840, buf_:0x7f98f7605000}, block_size:67104768, log_storage_:0x7f993e529bf0, log_storage_:{log_tail:{lsn:5282737655}, readable_log_tail:{lsn:5282737655}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:5234171904}, min_scn:{val:1755152244802535004, v:0}, curr_block_id:78, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:206, min_block_id:42, max_block_id:79, curr_writable_block_id:78}, logical_block_size_:67104768, curr_block_writable_size_:18539017, block_header_serialize_buf_:0x7f993e52a8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:85, read_io_size:301367296, read_disk_cost_ts:2364604, start_lsn:{lsn:4831543296}}}}}}) [2025-08-14 09:04:06.754254] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8032] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:06.754429] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=75] get wrs ts(ls_id={id:1}, delta=699015411, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:04:06.754447] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [776]OB_MALLOC COST TOO MUCH TIME, cost_time=972307, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=963611, WASH_BLOCK_END=62, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=8630], seq:[0, 1, 2, 3, 4] [2025-08-14 09:04:05.751419] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [776][T1002_StandbySc][T1002][YB42AC4D0007-00063C495DFA0361-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1001791, owner: ObLog, click_count: 7, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=29433, ObMalloc#start=24, ObMalloc#end=972317, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:06.753617] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=984024] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:04:06.755430] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=1764] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:04:06.755482] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:04:06.755528] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:04:06.759956] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1228] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:06.760971] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:06.761699] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=224] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:06.762373] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=9747] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:04:06.762519] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=251] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:06.764945] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=251] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:05.775869] EDIAG [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1005120, , BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1d189030 0x1d18285e 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:05.775869] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:05.841106] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:06.740355] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12930] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:06.784258] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=43835][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:06.784490] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22163][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:05.763282] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11713] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:04:05.763282] EDIAG [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11713][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1020033, , BACKTRACE: 0x26a1806d 0x9826e08 0x98267df 0x982650b 0x98262e0 0x1d188ec1 0x1d17b8ec 0x1d17ad88 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:06.765406] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=15] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:04:06.787644] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=229] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:06.832318] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:06.847039] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:06.850716] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=699112050, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:04:06.850744] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:04:06.864254] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:06.882819] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:06.946998] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:06.947088] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:06.965983] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=787358] PNIO [ratelimit] time: 1755162246965981, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007590, add_bytes: 0 [2025-08-14 09:04:07.275975] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:07.493364] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:07.512976] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:07.562531] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=206780] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:04:07.663798] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:07.720886] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=423] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:06.810245] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=937620] gc stale ls task succ [2025-08-14 09:04:07.249109] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=772053, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=9], seq:[0, 1] [2025-08-14 09:04:07.724059] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] get wrs ts(ls_id={id:1}, delta=699984815, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:04:07.724083] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:04:07.724124] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161101770923, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755161572801635) [2025-08-14 09:04:07.047152] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:05.753024] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1076) [801][T1002_TntTransf][T1002][YB42AC4D0007-00063C496259A60F-0-0] [lt=14825][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:12, local_retry_times:12, err_:-4012, err_:"OB_TIMEOUT", retry_type:0, client_ret:-4012}, need_retry=false) [2025-08-14 09:04:05.762588] EDIAG [ELECT] prepare (election_proposer.cpp:354) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=9613][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1959657, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1957794, ObMalloc#start=0, ObMalloc#end=1858, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0xf9d5ea1 0xf9d3eea 0xfa09ca2 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:07.726187] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=1020033] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:04:07.726221] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=33] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623147191219, flush_io_finished_ret_:0, flush_io_finished_round_:26331, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:07.726577] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=298] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:04:07.726595] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=17] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:04:07.726607] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623147206712, flush_io_finished_ret_:0, flush_io_finished_round_:26363, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:07.727555] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=630] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:04:03.799494] EDIAG [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=968211][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3933939, owner: ObLog, click_count: 6, time dist:[FORMAT_END=3932076, ObMalloc#start=1, ObMalloc#end=1682, ObMalloc#start=18, ObMalloc#end=39, FREE_END=123], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b51df0f 0x986bcde 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:05.938244] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=10] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:12632581, max_lsn:8461098908, max_scn:{val:1755161719775179002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:8214679957}, reuse_lsn:{lsn:8461098908}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:8419155868}}, last_submit_lsn:{lsn:8461098786}, last_submit_end_lsn:{lsn:8461098908}, last_submit_log_id:12632581, last_submit_log_pid:45, max_flushed_lsn:{lsn:8461098786}, max_flushed_end_lsn:{lsn:8461098908}, max_flushed_log_pid:45, committed_end_lsn:{lsn:8461098908}, last_slide_log_id:12632581, last_slide_scn:{val:1755161719775179002, v:0}, last_slide_lsn:{lsn:8461098786}, last_slide_end_lsn:{lsn:8461098908}, last_slide_log_pid:45, last_slide_log_accum_checksum:1091335677, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f992979c530}, [StateMgr]={this:0x7f99297a09b0, self:"172.77.0.7:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:46}, leader:"0.0.0.0:0", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:8461098908}, scan_disk_log_finished:true, last_check_start_id:12632582, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161899748977, palf_role_change_cb:0x7f99297a7470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:45, prev:{version:1, config:{config_version:{proposal_id:43, config_seq:41}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:45, config_seq:42}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:41, prev_lsn:{lsn:8461098218}, prev_mode_pid:45}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:41, prev_lsn:{lsn:8461098218}, prev_end_lsn:{lsn:8461098377}, prev_mode_pid:45}, reconfig_barrier:{prev_log_proposal_id:41, prev_lsn:{lsn:8461098218}, prev_end_lsn:{lsn:8461098377}, prev_mode_pid:45}, persistent_config_version:{proposal_id:45, config_seq:42}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755162245747627, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755162245747627, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f992979dc70}, [ModeMgr]={palf_id:1, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:45, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:46, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:46, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:1753719139394335001, v:0}, min_block_id:93, min_block_min_scn:{val:1753712284262753001, v:0}, base_lsn_for_block_gc:{lsn:8186781696}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:46}, log_config_meta:{version:3, proposal_id:45, prev:{version:1, config:{config_version:{proposal_id:43, config_seq:41}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:45, config_seq:42}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:41, prev_lsn:{lsn:8461098218}, prev_mode_pid:45}, log_snapshot_meta:{version:2, base_lsn:{lsn:8186781696}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:46, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:655360}, readable_log_tail:{lsn:655360}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8522305536}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/meta", dir_fd:214, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66449408, block_header_serialize_buf_:0x7f99297a2bb8, flashback_version:0}, log_storage:{log_tail:{lsn:8461098908}, readable_log_tail:{lsn:8461098908}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8455200768}, min_scn:{val:1755160384851871002, v:0}, curr_block_id:126, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:215, min_block_id:93, max_block_id:127, curr_writable_block_id:126}, logical_block_size_:67104768, curr_block_writable_size_:61206628, block_header_serialize_buf_:0x7f99297a48b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1755161900739661, this:0x7f99297a10f0}, [Reconfirm]={palf_id:1, self:"172.77.0.7:2882", state:"FETCH_MAX_LOG_LSN", new_proposal_id:46, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"172.77.0.7:2882", majority_max_accept_pid:45, majority_max_lsn:{lsn:8461098908}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:1755161954782466, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f99297a0b30}) [2025-08-14 09:04:05.938244] EDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1814597, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1783869, ObMalloc#start=1, ObMalloc#end=1714, ALLOC_END=12883, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0xfc0e73d 0x993f560 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:07.737337] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:07.755725] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=18355] temporary file module use no blocks [2025-08-14 09:04:07.755733] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:00:49.723306] EDIAG [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:613) [807][T1002_TimerWK3_TenantMetaMemMg][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] LOGGER COST TOO MUCH TIME, cost: 9002723, owner: ObLog, click_count: 3, time dist:[FORMAT_END=9002722, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x9ace3c3 0x9acda90 0xfad1d79 0xfad1b46 0x1b43a4df 0x96efabc 0x96ebbf3 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:07.755751] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=17] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:04:07.755761] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623149175846, flush_io_finished_ret_:0, flush_io_finished_round_:26352, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:07.755843] WDIAG alarm_if_necessary (ob_timer_service.cpp:143) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=9002723][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:07.757711] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=77] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:03.794789] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=980419] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136829134139001, v:0}, clog_base_lsn:{lsn:8186781696}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1753840867372367034, v:0}, all_id_meta:{id_meta:[{limited_id:1755161729775179000, latest_log_ts:{val:1755161719775179001, v:0}}, {limited_id:18000001, latest_log_ts:{val:1755157440846556001, v:0}}, {limited_id:60000001, latest_log_ts:{val:1755135657266496001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:46, palf_env_:0x7f99321fa030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-08-14 09:04:03.794789] EDIAG [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=980419][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3956978, owner: ObLog, click_count: 5, time dist:[FORMAT_END=3925786, ObMalloc#start=1, ObMalloc#end=31189, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b51df0f 0x986bcde 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:05.753024] EDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1076) [801][T1002_TntTransf][T1002][YB42AC4D0007-00063C496259A60F-0-0] [lt=14825][errcode=-4012] LOGGER COST TOO MUCH TIME, cost: 1969065, owner: ObLog, click_count: 7, time dist:[FORMAT_END=1969058, ObMalloc#start=0, ObMalloc#end=3, ALLOC_END=0, APPEND_END=4, ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=77], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x12ead58e 0x981b5d1 0x124ddd52 0x96a5a41 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x9a64aaf 0x10502edb 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:07.724874] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=676694][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:07.758914] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=1959657] get_min_unreplayed_log_info(lsn={lsn:8461098908}, scn={val:1755100947600130001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161920778323, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:23084275, ret_code_:0}}, next_to_submit_lsn_:{lsn:8461098908}, next_to_submit_scn_:{val:1755100947600130001, v:0}, base_lsn_:{lsn:7515734016}, base_scn_:{val:1753840867171901045, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6e05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:8461098908}, end_lsn:{lsn:8461098908}, read_buf:{buf_len_:3747840, buf_:0x7f98f6e05000}, block_size:67104768, log_storage_:0x7f99297a3bf0, log_storage_:{log_tail:{lsn:8461098908}, readable_log_tail:{lsn:8461098908}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8455200768}, min_scn:{val:1755160384851871002, v:0}, curr_block_id:126, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:215, min_block_id:93, max_block_id:127, curr_writable_block_id:126}, logical_block_size_:67104768, curr_block_writable_size_:61206628, block_header_serialize_buf_:0x7f99297a48b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:198, read_io_size:699822080, read_disk_cost_ts:4178035, start_lsn:{lsn:7515734016}}}}}}) [2025-08-14 09:04:07.759016] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1421) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=84] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-08-14 09:04:07.759258] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:07.759845] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:07.788367] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=3956978] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:04:07.788399] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=21] tx gc loop thread is running(MTL_ID()=1001) [2025-08-14 09:04:07.788412] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=13] try gc retain ctx [2025-08-14 09:04:07.788421] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=8] try to retry start_working [2025-08-14 09:04:07.788441] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=8] get_max_applied_scn(scn={val:1755161719775179002, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:45, palf_committed_end_lsn_:{lsn:8461098908}, palf_committed_end_scn_:{val:1755161719775179002, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161719775179002, v:0}}) [2025-08-14 09:04:07.788467] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=19] finish update min start scn(max_decided_scn={val:1755161719775179002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1755161287778691002, v:0}, keep_alive_scn_:{val:1755161287778691002, v:0}, update_ts_:1755162247741032}) [2025-08-14 09:04:07.789279] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=1814597] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161956777957, last_flushed_end_lsn:{lsn:8461098908}}]) [2025-08-14 09:04:07.789313] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=29] current log_task status(palf_id=1, self="172.77.0.7:2882", log_id=12632582, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161956777957, last_flushed_end_lsn:{lsn:8461098908}}]) [2025-08-14 09:04:07.762996] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:07.796296] WDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=33248][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:07.796330] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28855][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=909935, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [801]OB_MALLOC COST TOO MUCH TIME, cost_time=949544, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=949535], seq:[0] [801]OB_MALLOC COST TOO MUCH TIME, cost_time=949672, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=51, WASH_BLOCK_END=46, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=949568], seq:[0, 1, 2, 3, 4] [2025-08-14 09:04:07.783147] WDIAG [SQL] do_close_plan (ob_result_set.cpp:844) [801][T1002_TntTransf][T1002][YB42AC4D0007-00063C496259A60F-0-0] [lt=1969065][errcode=-4006] exec result is null(ret=-4006) [2025-08-14 09:04:08.747863] WDIAG [SQL] do_close (ob_result_set.cpp:935) [801][T1002_TntTransf][T1002][YB42AC4D0007-00063C496259A60F-0-0] [lt=963253][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:07.767299] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:08.749158] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=980374][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [575]OB_MALLOC COST TOO MUCH TIME, cost_time=953240, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=953205, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=925390, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=925324, ALLOC_CHUNK_END=15], seq:[0, 1, 2, 3, 4, 5, 6, 7] [466]OB_MALLOC COST TOO MUCH TIME, cost_time=902111, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=902070, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=891266, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=891219, ALLOC_CHUNK_END=14], seq:[0, 1, 2, 3, 4, 5, 6, 7] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=777083, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=16, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=18, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=777020, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=180326, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=18, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=180274, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=939132, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=925402, WASH_BLOCK_END=13730], seq:[0, 1, 2] [466]OB_MALLOC COST TOO MUCH TIME, cost_time=915883, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=902134, WASH_BLOCK_END=13749], seq:[0, 1, 2] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=905063, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=891277, WASH_BLOCK_END=13786], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=790914, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=777099, WASH_BLOCK_END=13815], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=195820, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=180336, WASH_BLOCK_END=15484], seq:[0, 1, 2] [2025-08-14 09:00:56.773758] EDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [339][EventTimer][T0][Y0-0000000000000000-0-0] [lt=1001353][errcode=0] LOGGER COST TOO MUCH TIME, cost: 85874434, owner: ObLog, click_count: 5, time dist:[FORMAT_END=85856006, ObMalloc#start=1, ObMalloc#end=18424, ALLOC_END=3, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ed35c 0x97aead0 0x97ad969 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [567]OB_MALLOC COST TOO MUCH TIME, cost_time=987889, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=987887], seq:[0] [575]OB_MALLOC COST TOO MUCH TIME, cost_time=988016, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=988014], seq:[0] [575]OB_MALLOC COST TOO MUCH TIME, cost_time=1953861, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=953260, WASH_BLOCK_END=12561, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=988036], seq:[0, 1, 2, 3, 4] [2025-08-14 09:04:07.796830] EDIAG [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=3933939][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1953888, owner: ObLog, click_count: 5, time dist:[FORMAT_END=9, ObMalloc#start=0, ObMalloc#end=1953873, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1bf2ec62 0x986ebd7 0x986bee6 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:07.796830] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=3933939] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:04:09.753709] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1953888] tx gc loop thread is running(MTL_ID()=1) [2025-08-14 09:04:09.753722] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] try gc retain ctx [2025-08-14 09:04:09.753800] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=78] try to retry start_working [2025-08-14 09:04:09.753829] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] finish update min start scn(max_decided_scn={val:1755161547738516001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1755161493768447002, v:0}, keep_alive_scn_:{val:1755161493768447002, v:0}, update_ts_:1755162249740156}) [2025-08-14 09:04:07.812311] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=911186][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1940295, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1940288, ObMalloc#start=0, ObMalloc#end=2, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:07.812311] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=911186] gc stale ls task succ [2025-08-14 09:04:09.755565] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=1940295] gc stale ls task succ [2025-08-14 09:04:07.753496] EDIAG [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=7697][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2000277, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1f832f91 0x97c36fe 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:07.753651] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2000182, owner: ObLog, click_count: 10, time dist:[ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=125, ALLOC_BLOCK#start=36956, ALLOC_BLOCK#end=85, FORMAT_END=1961745, ObMalloc#start=1, ObMalloc#end=61, ObMalloc#start=22, ObMalloc#end=57, FREE_END=1127], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:09.756881] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2000182][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:09.756911] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:09.756931] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [567]OB_MALLOC COST TOO MUCH TIME, cost_time=1005348, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=987907], seq:[0, 1] [2025-08-14 09:04:08.750069] EDIAG [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1005383, owner: ObLog, click_count: 5, time dist:[FORMAT_END=21, ObMalloc#start=1, ObMalloc#end=1005356, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1c03a517 0x97a8d73 0x97a88d2 0x97a7481 0x97a6a43 0x97a42c5 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:08.750069] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] get wrs ts(ls_id={id:1}, delta=700002516, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:04:09.756998] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1005383] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:04:09.757051] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=734] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:09.734785] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=953608][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [387]OB_MALLOC COST TOO MUCH TIME, cost_time=994476, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=53, WASH_BLOCK_END=54, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=9326], seq:[0, 1, 2, 3, 4] [2025-08-14 09:04:09.726273] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1026659, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:09.726273] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:10.754515] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1026659][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:10.754537] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [399]OB_MALLOC COST TOO MUCH TIME, cost_time=1961170, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1961168], seq:[0] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=1961202, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1961197], seq:[0, 1] [2025-08-14 09:04:09.759011] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9741][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1961228, owner: ObLog, click_count: 5, time dist:[FORMAT_END=8, ObMalloc#start=0, ObMalloc#end=1961212, ALLOC_END=1, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:09.759011] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9741] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:11.723257] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1961228][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:11.723283] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [717]OB_MALLOC COST TOO MUCH TIME, cost_time=954824, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=24, WASH_BLOCK_END=954799], seq:[0, 1, 2] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=927035, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=20, WASH_BLOCK_END=927015], seq:[0, 1, 2] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=867377, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=30, WASH_BLOCK_END=867347], seq:[0, 1, 2] [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=988427, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=28, WASH_BLOCK_END=988399], seq:[0, 1, 2] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=987792, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=23, WASH_BLOCK_END=987769], seq:[0, 1, 2] [964]OB_MALLOC COST TOO MUCH TIME, cost_time=987421, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=30, WASH_BLOCK_END=987391], seq:[0, 1, 2] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=987968, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=24, WASH_BLOCK_END=987944], seq:[0, 1, 2] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=986230, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=29, WASH_BLOCK_END=986201], seq:[0, 1, 2] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=981194, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=58, WASH_BLOCK_END=981136], seq:[0, 1, 2] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=979142, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=29, WASH_BLOCK_END=979113], seq:[0, 1, 2] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=979132, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=25, WASH_BLOCK_END=979107], seq:[0, 1, 2] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=979126, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=26, WASH_BLOCK_END=979100], seq:[0, 1, 2] [552]OB_MALLOC COST TOO MUCH TIME, cost_time=975704, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=39, WASH_BLOCK_END=975665], seq:[0, 1, 2] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=971724, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=39, WASH_BLOCK_END=971684], seq:[0, 1, 2] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=971105, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=28, WASH_BLOCK_END=971077], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=1760583, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=27, WASH_BLOCK_END=1760556], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=1652876, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=27, WASH_BLOCK_END=1652849], seq:[0, 1, 2] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=1444900, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=29, WASH_BLOCK_END=1444871], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=1073615, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=32, WASH_BLOCK_END=1073583], seq:[0, 1, 2] [1057]OB_MALLOC COST TOO MUCH TIME, cost_time=1024760, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=34, WASH_BLOCK_END=1024725], seq:[0, 1, 2] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=1983452, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=28, WASH_BLOCK_END=1983424], seq:[0, 1, 2] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=1978003, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=32, WASH_BLOCK_END=1977971], seq:[0, 1, 2] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=1977807, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=31, WASH_BLOCK_END=1977776], seq:[0, 1, 2] [864]OB_MALLOC COST TOO MUCH TIME, cost_time=1977819, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=34, WASH_BLOCK_END=1977785], seq:[0, 1, 2] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=1977770, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=31, WASH_BLOCK_END=1977739], seq:[0, 1, 2] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=1976914, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=39, WASH_BLOCK_END=1976874], seq:[0, 1, 2] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=1972565, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=29, WASH_BLOCK_END=1972536], seq:[0, 1, 2] [832]OB_MALLOC COST TOO MUCH TIME, cost_time=1969919, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=30, WASH_BLOCK_END=1969889], seq:[0, 1, 2] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=1968229, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=30, WASH_BLOCK_END=1968199], seq:[0, 1, 2] [388]OB_MALLOC COST TOO MUCH TIME, cost_time=1959761, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=42, WASH_BLOCK_END=1959719], seq:[0, 1, 2] [387]OB_MALLOC COST TOO MUCH TIME, cost_time=1029052, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=33, WASH_BLOCK_END=1029019], seq:[0, 1, 2] [466]OB_MALLOC COST TOO MUCH TIME, cost_time=1903770, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=24, WASH_BLOCK_END=1903745], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=1778387, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=29, WASH_BLOCK_END=1778358], seq:[0, 1, 2] [556]OB_MALLOC COST TOO MUCH TIME, cost_time=1051382, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=38, WASH_BLOCK_END=1051344], seq:[0, 1, 2] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=2979203, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=2979194], seq:[0] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=3932672, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=23, WASH_BLOCK_END=953416, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2979229], seq:[0, 1, 2, 3, 4] [2025-08-14 09:00:56.773806] EDIAG [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [14862][T1001_TimerWK5_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=5008711][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3972104, owner: ObLog, click_count: 5, time dist:[FORMAT_END=3972096, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1bd99231 0x1bd992e2 0x1bf31b90 0x9aaab88 0x1c3fa49f 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [717]OB_MALLOC COST TOO MUCH TIME, cost_time=2996650, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=2996629], seq:[0, 1, 2] [717]OB_MALLOC COST TOO MUCH TIME, cost_time=2996671, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2996669], seq:[0, 1] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=2996638, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=2996621, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=2996675, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2996648, WASH_BLOCK_END=27], seq:[0, 1, 2] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=2996636, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=2996618, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=2996654, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2996644, WASH_BLOCK_END=10], seq:[0, 1, 2] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=2990061, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=2990028, ALLOC_CHUNK_END=13], seq:[0, 1, 2, 3] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=2990080, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2990070, WASH_BLOCK_END=10], seq:[0, 1, 2] [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=2989958, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=17, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=2989925, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=2990004, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2989978, WASH_BLOCK_END=26], seq:[0, 1, 2] [383]OB_MALLOC COST TOO MUCH TIME, cost_time=2989942, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=2989923, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [383]OB_MALLOC COST TOO MUCH TIME, cost_time=2989962, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=2989951, WASH_BLOCK_END=10], seq:[0, 1, 2] [386]OB_MALLOC COST TOO MUCH TIME, cost_time=2990013, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=2989993, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [386]OB_MALLOC COST TOO MUCH TIME, cost_time=2990030, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2990022, WASH_BLOCK_END=8], seq:[0, 1, 2] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=2990059, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=2990035, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=2990079, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2990071, WASH_BLOCK_END=8], seq:[0, 1, 2] [369]OB_MALLOC COST TOO MUCH TIME, cost_time=2989395, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=2989365, ALLOC_CHUNK_END=12], seq:[0, 1, 2, 3] [369]OB_MALLOC COST TOO MUCH TIME, cost_time=2989410, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2989402, WASH_BLOCK_END=8], seq:[0, 1, 2] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=2987327, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=2987306, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=2987344, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2987335, WASH_BLOCK_END=9], seq:[0, 1, 2] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=2987314, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=2987296, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=2987330, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2987322, WASH_BLOCK_END=8], seq:[0, 1, 2] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=2987298, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=30, ALLOC_CHUNK_END=2987249, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=2987315, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2987306, WASH_BLOCK_END=9], seq:[0, 1, 2] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=2987244, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=2987216, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=2987260, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2987252, WASH_BLOCK_END=8], seq:[0, 1, 2] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=2987232, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=2987208, ALLOC_CHUNK_END=11], seq:[0, 1, 2, 3] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=2987248, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2987239, WASH_BLOCK_END=9], seq:[0, 1, 2] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=2987222, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=2987204, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=2987239, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2987230, WASH_BLOCK_END=8], seq:[0, 1, 2] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=2986326, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=2986295, ALLOC_CHUNK_END=12], seq:[0, 1, 2, 3] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=2986346, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2986338, WASH_BLOCK_END=8], seq:[0, 1, 2] [389]OB_MALLOC COST TOO MUCH TIME, cost_time=2986354, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=2986336, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [389]OB_MALLOC COST TOO MUCH TIME, cost_time=2986381, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2986372, WASH_BLOCK_END=9], seq:[0, 1, 2] [552]OB_MALLOC COST TOO MUCH TIME, cost_time=2984535, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=2984513, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [552]OB_MALLOC COST TOO MUCH TIME, cost_time=2984555, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2984543, WASH_BLOCK_END=12], seq:[0, 1, 2] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=2984509, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=2984491, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=2984526, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2984517, WASH_BLOCK_END=9], seq:[0, 1, 2] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=2984504, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=2984480, ALLOC_CHUNK_END=12], seq:[0, 1, 2, 3] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=2984522, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2984512, WASH_BLOCK_END=10], seq:[0, 1, 2] [822]OB_MALLOC COST TOO MUCH TIME, cost_time=2983593, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=2983571, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [822]OB_MALLOC COST TOO MUCH TIME, cost_time=2983611, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2983601, WASH_BLOCK_END=10], seq:[0, 1, 2] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=2983541, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=2983519, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=2983559, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2983551, WASH_BLOCK_END=8], seq:[0, 1, 2] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=2977691, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=2977667, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=2977709, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2977701, WASH_BLOCK_END=8], seq:[0, 1, 2] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=4946851, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=1991972, ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=2954849, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=4946874, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=4946865, WASH_BLOCK_END=9], seq:[0, 1, 2] [733]OB_MALLOC COST TOO MUCH TIME, cost_time=2889450, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=2889419, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [733]OB_MALLOC COST TOO MUCH TIME, cost_time=2889468, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2889458, WASH_BLOCK_END=10], seq:[0, 1, 2] [861]OB_MALLOC COST TOO MUCH TIME, cost_time=2879410, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=2879378, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [861]OB_MALLOC COST TOO MUCH TIME, cost_time=2879427, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2879419, WASH_BLOCK_END=8], seq:[0, 1, 2] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=1995372, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1995338, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=1995400, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1995392, WASH_BLOCK_END=8], seq:[0, 1, 2] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=3943892, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=1952668, ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=1991177, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=3943911, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=3943902, WASH_BLOCK_END=9], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=1989792, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=1989754, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=1989813, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1989803, WASH_BLOCK_END=9], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=1989760, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1989720, ALLOC_CHUNK_END=11], seq:[0, 1, 2, 3, 4, 5, 6, 7] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=1989778, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1989770, WASH_BLOCK_END=8], seq:[0, 1, 2] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=1989729, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1989695, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=1989747, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1989738, WASH_BLOCK_END=9], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=1989719, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1989679, ALLOC_CHUNK_END=11], seq:[0, 1, 2, 3, 4, 5, 6, 7] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=1989737, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1989728, WASH_BLOCK_END=9], seq:[0, 1, 2] [1057]OB_MALLOC COST TOO MUCH TIME, cost_time=1989695, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1989659, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [1057]OB_MALLOC COST TOO MUCH TIME, cost_time=1989713, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1989705, WASH_BLOCK_END=8], seq:[0, 1, 2] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=1989697, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1989653, ALLOC_CHUNK_END=12], seq:[0, 1, 2, 3, 4, 5, 6, 7] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=3981183, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=33, WASH_BLOCK_END=1991443, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1989705], seq:[0, 1, 2, 3, 4] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=1989299, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1989259, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=1989317, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1989309, WASH_BLOCK_END=8], seq:[0, 1, 2] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=1989275, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1989232, ALLOC_CHUNK_END=11], seq:[0, 1, 2, 3, 4, 5, 6, 7] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=1989292, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1989284, WASH_BLOCK_END=8], seq:[0, 1, 2] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=1989251, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1989216, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=1989269, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1989260, WASH_BLOCK_END=9], seq:[0, 1, 2] [864]OB_MALLOC COST TOO MUCH TIME, cost_time=1989239, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1989203, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [864]OB_MALLOC COST TOO MUCH TIME, cost_time=1989269, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1989255, WASH_BLOCK_END=14], seq:[0, 1, 2] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=1989234, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1989198, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=1989263, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1989253, WASH_BLOCK_END=10], seq:[0, 1, 2] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=1989242, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1989206, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=1989262, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1989252, WASH_BLOCK_END=10], seq:[0, 1, 2] [832]OB_MALLOC COST TOO MUCH TIME, cost_time=1989229, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1989198, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [832]OB_MALLOC COST TOO MUCH TIME, cost_time=1989257, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1989247, WASH_BLOCK_END=10], seq:[0, 1, 2] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=1989223, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=1989192, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=1989242, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1989232, WASH_BLOCK_END=10], seq:[0, 1, 2] [387]OB_MALLOC COST TOO MUCH TIME, cost_time=1989157, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1989124, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [387]OB_MALLOC COST TOO MUCH TIME, cost_time=1989177, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1989169, WASH_BLOCK_END=8], seq:[0, 1, 2] [964]OB_MALLOC COST TOO MUCH TIME, cost_time=1975858, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1975813, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [964]OB_MALLOC COST TOO MUCH TIME, cost_time=1975875, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1975867, WASH_BLOCK_END=8], seq:[0, 1, 2] [404]OB_MALLOC COST TOO MUCH TIME, cost_time=1964053, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=1964013, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [404]OB_MALLOC COST TOO MUCH TIME, cost_time=1964074, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1964065, WASH_BLOCK_END=8], seq:[0, 1, 2] [466]OB_MALLOC COST TOO MUCH TIME, cost_time=1964121, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1964080, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [466]OB_MALLOC COST TOO MUCH TIME, cost_time=1964140, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1964132, WASH_BLOCK_END=8], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=1964080, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1964038, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=1964098, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1964090, WASH_BLOCK_END=8], seq:[0, 1, 2] [556]OB_MALLOC COST TOO MUCH TIME, cost_time=1964028, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1963992, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [556]OB_MALLOC COST TOO MUCH TIME, cost_time=1964046, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1964037, WASH_BLOCK_END=9], seq:[0, 1, 2] [400]OB_MALLOC COST TOO MUCH TIME, cost_time=1843440, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1843383, ALLOC_CHUNK_END=12], seq:[0, 1, 2, 3, 4, 5, 6, 7] [400]OB_MALLOC COST TOO MUCH TIME, cost_time=1843458, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1843450, WASH_BLOCK_END=8], seq:[0, 1, 2] [764]OB_MALLOC COST TOO MUCH TIME, cost_time=1061662, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=1061625, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [764]OB_MALLOC COST TOO MUCH TIME, cost_time=1061680, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1061671, WASH_BLOCK_END=9], seq:[0, 1, 2] [339]OB_MALLOC COST TOO MUCH TIME, cost_time=1000830, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1000790, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [339]OB_MALLOC COST TOO MUCH TIME, cost_time=1000853, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1000843, WASH_BLOCK_END=10], seq:[0, 1, 2] [726]OB_MALLOC COST TOO MUCH TIME, cost_time=991373, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=991332, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [726]OB_MALLOC COST TOO MUCH TIME, cost_time=991395, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=991385, WASH_BLOCK_END=10], seq:[0, 1, 2] [659]OB_MALLOC COST TOO MUCH TIME, cost_time=989751, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=24, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=989690, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [659]OB_MALLOC COST TOO MUCH TIME, cost_time=989772, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=989762, WASH_BLOCK_END=10], seq:[0, 1, 2] [402]OB_MALLOC COST TOO MUCH TIME, cost_time=769922, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=769870, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3, 4, 5, 6, 7] [402]OB_MALLOC COST TOO MUCH TIME, cost_time=769940, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=769933, WASH_BLOCK_END=7], seq:[0, 1, 2] [376]OB_MALLOC COST TOO MUCH TIME, cost_time=966902, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=966861, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [376]OB_MALLOC COST TOO MUCH TIME, cost_time=966922, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=966913, WASH_BLOCK_END=9], seq:[0, 1, 2] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=1990384, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1990350, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=1990402, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1990394, WASH_BLOCK_END=8], seq:[0, 1, 2] [571]OB_MALLOC COST TOO MUCH TIME, cost_time=708506, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=708468, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [571]OB_MALLOC COST TOO MUCH TIME, cost_time=708528, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=708518, WASH_BLOCK_END=10], seq:[0, 1, 2] [388]OB_MALLOC COST TOO MUCH TIME, cost_time=1990183, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1990151, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [388]OB_MALLOC COST TOO MUCH TIME, cost_time=1990201, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1990193, WASH_BLOCK_END=8], seq:[0, 1, 2] [2025-08-14 09:04:10.782374] EDIAG [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=182][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3951559, owner: ObLog, click_count: 7, time dist:[FORMAT_END=10, ObMalloc#start=0, ObMalloc#end=954842, ObMalloc#start=25, ObMalloc#end=2996679, ALLOC_END=2, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1d189030 0x1d18285e 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:10.776085] EDIAG [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=180][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3963508, owner: ObLog, click_count: 6, time dist:[FORMAT_END=13, ObMalloc#start=1, ObMalloc#end=975722, ObMalloc#start=31, ObMalloc#end=2984561, FREE_END=3180], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1d189030 0x1d18285e 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:10.770988] EDIAG do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=312][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3969529, owner: ObLog, click_count: 6, time dist:[FORMAT_END=11, ObMalloc#start=0, ObMalloc#end=1977779, ObMalloc#start=19, ObMalloc#end=1990409, FREE_END=1311], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x202c530b 0x202c6888 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:10.770545] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [663][T1001_StandbySc][T1001][YB42AC4D0007-00063C495C7A035E-0-0] [lt=276][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3970618, owner: ObLog, click_count: 6, time dist:[FORMAT_END=15, ObMalloc#start=1, ObMalloc#end=1978012, ObMalloc#start=21, ObMalloc#end=1989298, FREE_END=3271], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:10.996448] EDIAG eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=184][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3745091, owner: ObLog, click_count: 6, time dist:[FORMAT_END=15, ObMalloc#start=1, ObMalloc#end=1778396, ObMalloc#start=19, ObMalloc#end=1964104, FREE_END=2556], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x27a8a8c8 0x97b7e1a 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:11.675609] EDIAG try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=204][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3067033, owner: ObLog, click_count: 6, time dist:[FORMAT_END=15, ObMalloc#start=2, ObMalloc#end=1073623, ObMalloc#start=16, ObMalloc#end=1989743, FREE_END=3634], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x26499505 0x97074a3 0x26498fa9 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:10.770818] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [776][T1002_StandbySc][T1002][YB42AC4D0007-00063C495DFA0365-0-0] [lt=2104][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3967427, owner: ObLog, click_count: 6, time dist:[FORMAT_END=8, ObMalloc#start=0, ObMalloc#end=1977815, ObMalloc#start=21, ObMalloc#end=1989275, FREE_END=308], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:10.770863] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=275][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3970876, owner: ObLog, click_count: 6, time dist:[FORMAT_END=19, ObMalloc#start=1, ObMalloc#end=1977827, ObMalloc#start=19, ObMalloc#end=1989275, FREE_END=3735], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:11.747847] EDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=316][errcode=-4015] LOGGER COST TOO MUCH TIME, cost: 2989453, owner: ObLog, click_count: 5, time dist:[FORMAT_END=14, ObMalloc#start=0, ObMalloc#end=2989416, ALLOC_END=22, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x279e59be 0x971b1c1 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:09.788548] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [477][T1_StandbySchem][T1][YB42AC4D0007-00063C495B6A0373-0-0] [lt=218][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4946925, owner: ObLog, click_count: 5, time dist:[FORMAT_END=13, ObMalloc#start=1, ObMalloc#end=4946881, ALLOC_END=29, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:09.780593] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [556][T1_OBJLockGC][T1][Y0-0000000000000000-0-0] [lt=430][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4960131, owner: ObLog, click_count: 8, time dist:[FORMAT_END=1941136, ObMalloc#start=1, ObMalloc#end=1051391, ObMalloc#start=20, ObMalloc#end=1964052, ObMalloc#start=644, ObMalloc#end=50, FREE_END=2837], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:10.780805] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=211][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3957543, owner: ObLog, click_count: 6, time dist:[FORMAT_END=11, ObMalloc#start=0, ObMalloc#end=1968239, ObMalloc#start=18, ObMalloc#end=1989248, FREE_END=27], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:10.987320] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3755290, owner: ObLog, click_count: 8, time dist:[FORMAT_END=35, ObMalloc#start=1, ObMalloc#end=1760613, ObMalloc#start=32, ObMalloc#end=1989820, ObMalloc#start=1901, ObMalloc#end=55, FREE_END=2833], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:10.779053] EDIAG [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=220][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3959308, owner: ObLog, click_count: 6, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=1969927, ObMalloc#start=18, ObMalloc#end=1989263, FREE_END=89], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1d189030 0x1d18285e 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:11.303660] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=2][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3439499, owner: ObLog, click_count: 8, time dist:[FORMAT_END=51, ObMalloc#start=3, ObMalloc#end=1444907, ObMalloc#start=19, ObMalloc#end=1989756, ObMalloc#start=681, ObMalloc#end=57, FREE_END=4025], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:10.870737] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3872354, owner: ObLog, click_count: 8, time dist:[FORMAT_END=35, ObMalloc#start=1, ObMalloc#end=1903794, ObMalloc#start=28, ObMalloc#end=1964147, ObMalloc#start=625, ObMalloc#end=71, FREE_END=3653], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:11.095298] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3648180, owner: ObLog, click_count: 8, time dist:[FORMAT_END=46, ObMalloc#start=1, ObMalloc#end=1652886, ObMalloc#start=19, ObMalloc#end=1989784, ObMalloc#start=2902, ObMalloc#end=61, FREE_END=2481], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:10.870082] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3873195, owner: ObLog, click_count: 8, time dist:[FORMAT_END=29, ObMalloc#start=1, ObMalloc#end=867409, ObMalloc#start=18, ObMalloc#end=2996662, ObMalloc#start=3867, ObMalloc#end=4259, FREE_END=950], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:11.849149] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=224374][errcode=-4018] LOGGER COST TOO MUCH TIME, cost: 2897546, owner: ObLog, click_count: 8, time dist:[FORMAT_END=18, ObMalloc#start=1, ObMalloc#end=2889473, ObMalloc#start=18, ObMalloc#end=50, ObMalloc#start=5524, ObMalloc#end=53, FREE_END=2409], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3b4b 0x113d34aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:10.793444] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3951686, owner: ObLog, click_count: 8, time dist:[FORMAT_END=47, ObMalloc#start=0, ObMalloc#end=3943917, ObMalloc#start=24, ObMalloc#end=49, ObMalloc#start=5318, ObMalloc#end=49, FREE_END=2282], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:12.898458] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=691549][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1850136, owner: ObLog, click_count: 8, time dist:[FORMAT_END=28, ObMalloc#start=1, ObMalloc#end=1843464, ObMalloc#start=17, ObMalloc#end=52, ObMalloc#start=4067, ObMalloc#end=55, FREE_END=2451], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:11.859310] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=226100][errcode=-4018] LOGGER COST TOO MUCH TIME, cost: 2887933, owner: ObLog, click_count: 8, time dist:[FORMAT_END=25, ObMalloc#start=1, ObMalloc#end=2879434, ObMalloc#start=24, ObMalloc#end=59, ObMalloc#start=5974, ObMalloc#end=73, FREE_END=2343], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3b4b 0x113d34aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:10.725042] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [339][EventTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4020803, owner: ObLog, click_count: 8, time dist:[FORMAT_END=3012910, ObMalloc#start=2, ObMalloc#end=1000860, ObMalloc#start=27, ObMalloc#end=60, ObMalloc#start=4308, ObMalloc#end=47, FREE_END=2589], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:10.725568] EDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] LOGGER COST TOO MUCH TIME, cost: 4020338, owner: ObLog, click_count: 8, time dist:[FORMAT_END=3023603, ObMalloc#start=2, ObMalloc#end=989779, ObMalloc#start=27, ObMalloc#end=45, ObMalloc#start=3881, ObMalloc#end=78, FREE_END=2923], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0xfd2ef08 0xfd2bd1d 0xf9d16a8 0xf9d4150 0xfa09ca2 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:09.225032] EDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] LOGGER COST TOO MUCH TIME, cost: 5519063, , BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x13b06ffb 0x958a39e 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:10.787891] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=20457][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3962580, owner: ObLog, click_count: 6, time dist:[FORMAT_END=3959292, ObMalloc#start=1, ObMalloc#end=99, ObMalloc#start=39, ObMalloc#end=80, FREE_END=3069], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:10.789025] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=18811][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4921858, owner: ObLog, click_count: 6, time dist:[FORMAT_END=4920617, ObMalloc#start=2, ObMalloc#end=126, ObMalloc#start=43, ObMalloc#end=59, FREE_END=1011], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:10.771912] EDIAG [RS] skip_check_member_list_change_ (ob_ls_recovery_stat_handler.cpp:109) [777][T1002_TeRec][T1002][YB42AC4D0007-00063C495DDA006D-0-0] [lt=265][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4945058, owner: ObLog, click_count: 6, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=1976921, ObMalloc#start=21, ObMalloc#end=1989271, FREE_END=978835], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x1067fb2c 0x1067ecc2 0x1067e03d 0x1002f2a3 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:10.782524] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=18385][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4951069, owner: ObLog, click_count: 6, time dist:[FORMAT_END=4950251, ObMalloc#start=2, ObMalloc#end=66, ObMalloc#start=24, ObMalloc#end=54, FREE_END=672], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:13.681515] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=315][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1068293, owner: ObLog, click_count: 8, time dist:[FORMAT_END=50, ObMalloc#start=1, ObMalloc#end=1061686, ObMalloc#start=25, ObMalloc#end=3466, ObMalloc#start=1040, ObMalloc#end=53, FREE_END=1972], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:23.763108] EDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=1294490][errcode=0] LOGGER COST TOO MUCH TIME, cost: 110797881, owner: ObLog, click_count: 8, time dist:[FORMAT_END=8, ObMalloc#start=0, ObMalloc#end=996013, ObMalloc#start=109794597, ObMalloc#end=76, ObMalloc#start=1738, ObMalloc#end=113, FREE_END=5336], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ed35c 0x97aead0 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:10.762298] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9013][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3975711, owner: ObLog, click_count: 6, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=986240, ObMalloc#start=20, ObMalloc#end=2987336, FREE_END=2104], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:10.780174] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=938][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3958098, owner: ObLog, click_count: 6, time dist:[FORMAT_END=8, ObMalloc#start=0, ObMalloc#end=971733, ObMalloc#start=19, ObMalloc#end=2984533, FREE_END=1805], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:10.810287] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3928436, owner: ObLog, click_count: 8, time dist:[FORMAT_END=26, ObMalloc#start=1, ObMalloc#end=927043, ObMalloc#start=20, ObMalloc#end=2996683, ObMalloc#start=3996, ObMalloc#end=77, FREE_END=590], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:10.760437] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1300][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3977817, owner: ObLog, click_count: 6, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=987986, ObMalloc#start=75, ObMalloc#end=2987350, FREE_END=2395], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:10.757726] EDIAG [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=218][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3980518, owner: ObLog, click_count: 6, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=987813, ObMalloc#start=98, ObMalloc#end=2990085, FREE_END=2509], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a1f867 0x27a135c3 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:13.750584] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=23] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:04:10.782374] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=182] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:11.747847] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=316][errcode=-4015] clock out of order(cur_ts=1755162251747846, cur_ts_=1755162251719926, delta=27920) [2025-08-14 09:04:11.751347] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=627] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:15.744889] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3975711][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:13.754242] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [721][T1001_OBJLockGC][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:10.780953] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=504] gc stale ls task succ [2025-08-14 09:04:10.765000] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=406][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3973118, owner: ObLog, click_count: 6, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=1983473, ObMalloc#start=27, ObMalloc#end=1989324, FREE_END=283], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:10.769556] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3968563, owner: ObLog, click_count: 8, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=979153, ObMalloc#start=21, ObMalloc#end=2987266, FREE_END=2112, ALLOC_BLOCK#start=1006203, ALLOC_BLOCK#end=2], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:10.769690] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=247][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3968424, owner: ObLog, click_count: 6, time dist:[FORMAT_END=6, ObMalloc#start=0, ObMalloc#end=979134, ObMalloc#start=25, ObMalloc#end=2987245, FREE_END=2014], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:11.751347] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=627][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2986400, owner: ObLog, click_count: 5, time dist:[FORMAT_END=10, ObMalloc#start=0, ObMalloc#end=2986388, ALLOC_END=2, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:11.760564] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2984108, owner: ObLog, click_count: 6, time dist:[FORMAT_END=11, ObMalloc#start=0, ObMalloc#end=2977715, ObMalloc#start=28, ObMalloc#end=52, FREE_END=6302], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:11.745477] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=199][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2994652, owner: ObLog, click_count: 6, time dist:[FORMAT_END=13, ObMalloc#start=2, ObMalloc#end=2990086, ObMalloc#start=2, ObMalloc#end=62, FREE_END=4487], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:15.751820] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3973118][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:14.748261] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [556][T1_OBJLockGC][T1][Y0-0000000000000000-0-0] [lt=4960131][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:14.751968] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [339][EventTimer][T0][Y0-0000000000000000-0-0] [lt=4020803][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:11.724498] EDIAG [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [726][T1001_ObXAHbWor][T1001][Y0-0000000000000000-0-0] [lt=650][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3019608, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b64d8ba 0x1b64d154 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:11.754589] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=469][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2989497, owner: ObLog, click_count: 6, time dist:[FORMAT_END=12, ObMalloc#start=0, ObMalloc#end=2983566, ObMalloc#start=1, ObMalloc#end=66, FREE_END=5852], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:11.747042] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=436][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2996711, owner: ObLog, click_count: 6, time dist:[FORMAT_END=10, ObMalloc#start=0, ObMalloc#end=2989968, ObMalloc#start=2, ObMalloc#end=57, FREE_END=6674], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:11.747081] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=192][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2996666, owner: ObLog, click_count: 6, time dist:[FORMAT_END=7, ObMalloc#start=0, ObMalloc#end=2990037, ObMalloc#start=1, ObMalloc#end=56, FREE_END=6565], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:10.789350] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [388][RpcIO][T0][Y0-0000000000000000-0-0] [lt=213][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3951986, owner: ObLog, click_count: 6, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=1959769, ObMalloc#start=20, ObMalloc#end=1990207, FREE_END=1978], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:10.769636] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8286][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3969822, owner: ObLog, click_count: 6, time dist:[FORMAT_END=7, ObMalloc#start=0, ObMalloc#end=979140, ObMalloc#start=19, ObMalloc#end=2987254, FREE_END=3401], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:11.721533] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3019294, owner: ObLog, click_count: 6, time dist:[FORMAT_END=16, ObMalloc#start=2, ObMalloc#end=1029060, ObMalloc#start=19, ObMalloc#end=1989183, FREE_END=1014], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:10.756977] EDIAG [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=2000277][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3985032, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1f832f91 0x97c36fe 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [397]OB_MALLOC COST TOO MUCH TIME, cost_time=982662, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=44, WASH_BLOCK_END=982618], seq:[0, 1, 2] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=993073, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=46, WASH_BLOCK_END=993027], seq:[0, 1, 2] [2025-08-14 09:04:10.756507] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4959829, owner: ObLog, click_count: 6, time dist:[FORMAT_END=12, ObMalloc#start=0, ObMalloc#end=3981189, ObMalloc#start=18, ObMalloc#end=50, FREE_END=978560], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [392]OB_MALLOC COST TOO MUCH TIME, cost_time=1008652, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=42, WASH_BLOCK_END=1008610], seq:[0, 1, 2] [2025-08-14 09:04:10.780848] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=648][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4928451, owner: ObLog, click_count: 6, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=971113, ObMalloc#start=17, ObMalloc#end=2984528, FREE_END=972781], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:15.750487] EDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3977817][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1011526, owner: ObLog, click_count: 8, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=69, ObMalloc#start=1, ObMalloc#end=51, ObMalloc#start=1555, ObMalloc#end=1008673, FREE_END=1167], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:09.770188] EDIAG [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4971635, , BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1fce4e91 0x1fce455d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [383]OB_MALLOC COST TOO MUCH TIME, cost_time=1001969, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1001961], seq:[0] [386]OB_MALLOC COST TOO MUCH TIME, cost_time=1002869, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1002841, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [386]OB_MALLOC COST TOO MUCH TIME, cost_time=1002947, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1002882, WASH_BLOCK_END=65], seq:[0, 1, 2] [388]OB_MALLOC COST TOO MUCH TIME, cost_time=1002975, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1002943, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [388]OB_MALLOC COST TOO MUCH TIME, cost_time=1002993, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1002984, WASH_BLOCK_END=8], seq:[0, 1, 2] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=1002951, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1002928, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=1002967, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1002960, WASH_BLOCK_END=7], seq:[0, 1, 2] [2025-08-14 09:04:16.721490] EDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2996666][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1027831, owner: ObLog, click_count: 8, time dist:[FORMAT_END=8, ObMalloc#start=0, ObMalloc#end=24595, ObMalloc#start=1, ObMalloc#end=1002954, ObMalloc#start=155, ObMalloc#end=39, FREE_END=79], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [382]OB_MALLOC COST TOO MUCH TIME, cost_time=1008429, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1008399, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=1008485, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1008450, WASH_BLOCK_END=35], seq:[0, 1, 2] [683]OB_MALLOC COST TOO MUCH TIME, cost_time=1008373, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1008340, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [683]OB_MALLOC COST TOO MUCH TIME, cost_time=1008392, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1008383, WASH_BLOCK_END=9], seq:[0, 1, 2] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=1008338, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1008312, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=1008357, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1008348, WASH_BLOCK_END=9], seq:[0, 1, 2] [389]OB_MALLOC COST TOO MUCH TIME, cost_time=1008323, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1008296, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [389]OB_MALLOC COST TOO MUCH TIME, cost_time=1008340, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1008332, WASH_BLOCK_END=8], seq:[0, 1, 2] [410]OB_MALLOC COST TOO MUCH TIME, cost_time=1008362, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1008336, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [410]OB_MALLOC COST TOO MUCH TIME, cost_time=1008381, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1008371, WASH_BLOCK_END=9], seq:[0, 1, 2] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=1006930, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1006901, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=1006949, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1006940, WASH_BLOCK_END=9], seq:[0, 1, 2] [413]OB_MALLOC COST TOO MUCH TIME, cost_time=1006945, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1006919, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [413]OB_MALLOC COST TOO MUCH TIME, cost_time=1006968, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1006959, WASH_BLOCK_END=9], seq:[0, 1, 2] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=993178, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=993145, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=993196, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=993187, WASH_BLOCK_END=9], seq:[0, 1, 2] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=993155, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=993126, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=993172, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=993164, WASH_BLOCK_END=8], seq:[0, 1, 2] [528]OB_MALLOC COST TOO MUCH TIME, cost_time=993144, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=993109, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [528]OB_MALLOC COST TOO MUCH TIME, cost_time=993162, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=993153, WASH_BLOCK_END=9], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=991905, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=991873, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=991922, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=991913, WASH_BLOCK_END=9], seq:[0, 1, 2] [2025-08-14 09:04:16.720547] EDIAG eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=3745091][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1034984, owner: ObLog, click_count: 6, time dist:[FORMAT_END=17, ObMalloc#start=1, ObMalloc#end=42956, ObMalloc#start=21, ObMalloc#end=991928, FREE_END=61], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x27a8a8c8 0x97b7e1a 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:12.773258] EDIAG [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4976512, owner: ObLog, click_count: 6, time dist:[FORMAT_END=3945396, ObMalloc#start=1, ObMalloc#end=37622, ObMalloc#start=20, ObMalloc#end=993168, FREE_END=305], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x12377491 0x1236e8e9 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:15.752024] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [477][T1_StandbySchem][T1][YB42AC4D0007-00063C495B6A0374-0-0] [lt=4946925][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2002368, owner: ObLog, click_count: 6, time dist:[FORMAT_END=974799, ObMalloc#start=1, ObMalloc#end=19956, ObMalloc#start=2, ObMalloc#end=1006956, FREE_END=654], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:16.722064] EDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3968424][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1033835, owner: ObLog, click_count: 8, time dist:[FORMAT_END=10, ObMalloc#start=0, ObMalloc#end=40021, ObMalloc#start=39, ObMalloc#end=993202, ObMalloc#start=351, ObMalloc#end=41, FREE_END=171], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:16.722129] EDIAG ev_invoke_pending (ev.c:2136) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=3980518][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1033791, owner: ObLog, click_count: 8, time dist:[FORMAT_END=8, ObMalloc#start=0, ObMalloc#end=40043, ObMalloc#start=21, ObMalloc#end=993178, ObMalloc#start=97, ObMalloc#end=49, FREE_END=395], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:15.752028] EDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2994652][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2002454, owner: ObLog, click_count: 8, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=993092, ObMalloc#start=3, ObMalloc#end=1008363, ObMalloc#start=851, ObMalloc#end=40, FREE_END=95], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:16.721954] EDIAG ev_invoke_pending (ev.c:2136) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2986400][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1034013, owner: ObLog, click_count: 8, time dist:[FORMAT_END=19, ObMalloc#start=1, ObMalloc#end=24701, ObMalloc#start=1, ObMalloc#end=1008346, ObMalloc#start=741, ObMalloc#end=41, FREE_END=163], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [398]OB_MALLOC COST TOO MUCH TIME, cost_time=993298, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=993264, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=993352, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=993316, WASH_BLOCK_END=36], seq:[0, 1, 2] [466]OB_MALLOC COST TOO MUCH TIME, cost_time=895509, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=895480, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [466]OB_MALLOC COST TOO MUCH TIME, cost_time=895538, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=895520, WASH_BLOCK_END=18], seq:[0, 1, 2] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=895715, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=895689, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=895758, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=895729, WASH_BLOCK_END=29], seq:[0, 1, 2] [402]OB_MALLOC COST TOO MUCH TIME, cost_time=696126, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=17, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=696086, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [402]OB_MALLOC COST TOO MUCH TIME, cost_time=696159, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=696144, WASH_BLOCK_END=15], seq:[0, 1, 2] [2025-08-14 09:04:15.745552] EDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3958098][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2013396, owner: ObLog, click_count: 10, time dist:[FORMAT_END=997376, ObMalloc#start=1, ObMalloc#end=21106, ObMalloc#start=27, ObMalloc#end=993363, ObMalloc#start=1022, ObMalloc#end=84, FREE_END=417, ALLOC_BLOCK#start=7, ALLOC_BLOCK#end=1], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:16.746312] EDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3968563][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1018954, owner: ObLog, click_count: 8, time dist:[FORMAT_END=12, ObMalloc#start=0, ObMalloc#end=1008494, ObMalloc#start=24, ObMalloc#end=45, ObMalloc#start=1136, ObMalloc#end=49, FREE_END=9194], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:16.776471] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1003634, owner: ObLog, click_count: 8, time dist:[FORMAT_END=1000867, ObMalloc#start=2, ObMalloc#end=77, ObMalloc#start=26, ObMalloc#end=47, ObMalloc#start=2440, ObMalloc#end=75, FREE_END=100], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:16.761655] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1022150, owner: ObLog, click_count: 8, time dist:[FORMAT_END=1018608, ObMalloc#start=1, ObMalloc#end=56, ObMalloc#start=20, ObMalloc#end=47, ObMalloc#start=72, ObMalloc#end=53, FREE_END=3293], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [776]OB_MALLOC COST TOO MUCH TIME, cost_time=2009934, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=41, WASH_BLOCK_END=24697, ALLOC_BLOCK#start=1985132, ALLOC_BLOCK#end=64], seq:[0, 1, 2, 3, 4] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=2968508, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=43, WASH_BLOCK_END=982981, ALLOC_BLOCK#start=1985444, ALLOC_BLOCK#end=39], seq:[0, 1, 2, 3, 4] [2025-08-14 09:04:15.752009] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [776][T1002_StandbySc][T1002][YB42AC4D0007-00063C495DFA0366-0-0] [lt=3967427][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2979114, owner: ObLog, click_count: 6, time dist:[FORMAT_END=967843, ObMalloc#start=1, ObMalloc#end=2009961, ObMalloc#start=30, ObMalloc#end=123, FREE_END=1156], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:15.761589] EDIAG ev_invoke_pending (ev.c:2136) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2989497][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2971137, owner: ObLog, click_count: 8, time dist:[FORMAT_END=7, ObMalloc#start=0, ObMalloc#end=2968524, ObMalloc#start=22, ObMalloc#end=76, ObMalloc#start=976, ObMalloc#end=91, FREE_END=1441], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:15.762050] EDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3969822][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1985986, owner: ObLog, click_count: 8, time dist:[FORMAT_END=7, ObMalloc#start=0, ObMalloc#end=982697, ObMalloc#start=2, ObMalloc#end=1002976, ObMalloc#start=94, ObMalloc#end=37, FREE_END=173], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:12.766388] EDIAG [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=202][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4983580, owner: ObLog, click_count: 6, time dist:[FORMAT_END=3974342, ObMalloc#start=0, ObMalloc#end=1008387, ObMalloc#start=23, ObMalloc#end=42, FREE_END=786], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a5bbf2 0x94da542 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [663]OB_MALLOC COST TOO MUCH TIME, cost_time=2067974, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=20, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=2067935], seq:[0, 1, 2, 3, 4] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=2067998, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2067994], seq:[0, 1] [2025-08-14 09:04:15.751989] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [663][T1001_StandbySc][T1001][YB42AC4D0007-00063C495C7A035F-0-0] [lt=3970618][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3035926, owner: ObLog, click_count: 5, time dist:[FORMAT_END=967913, ObMalloc#start=2, ObMalloc#end=2068005, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [383]OB_MALLOC COST TOO MUCH TIME, cost_time=2043527, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1001997], seq:[0, 1] [2025-08-14 09:04:16.721429] EDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2996711][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2068203, owner: ObLog, click_count: 7, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=24638, ObMalloc#start=3, ObMalloc#end=2043545, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:11.887377] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2859733, , BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:10.780953] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=504][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4959803, owner: ObLog, click_count: 9, time dist:[FORMAT_END=8, ObMalloc#start=0, ObMalloc#end=3932685, ALLOC_END=1027108, APPEND_END=1, ALLOC_BLOCK#start=6, ALLOC_BLOCK#end=69, ALLOC_BLOCK#start=11897, ALLOC_BLOCK#end=22], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:16.745646] EDIAG do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=3969529][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2045782, owner: ObLog, click_count: 6, time dist:[FORMAT_END=2045294, ObMalloc#start=1, ObMalloc#end=73, ObMalloc#start=23, ObMalloc#end=44, FREE_END=347], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x202c530b 0x202c6888 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:16.763537] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2030586, owner: ObLog, click_count: 8, time dist:[FORMAT_END=2030139, ObMalloc#start=1, ObMalloc#end=65, ObMalloc#start=28, ObMalloc#end=45, ObMalloc#start=156, ObMalloc#end=59, FREE_END=93], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:11.754396] EDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=154][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2992605, , BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0xfb622cd 0xfb619cb 0xfb892d3 0x994150a 0x993eecd 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:16.721429] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2996711][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:17.769648] EDIAG [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=3985032][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1029717, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1023287, ObMalloc#start=1, ObMalloc#end=55, ObMalloc#start=20, ObMalloc#end=48, FREE_END=6306], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1f832f91 0x97c36fe 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:16.761472] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2036418, owner: ObLog, click_count: 9, time dist:[FORMAT_END=2036152, ObMalloc#start=0, ObMalloc#end=48, ObMalloc#start=20, ObMalloc#end=44, ObMalloc#start=149, ObMalloc#end=1, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:16.761472] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:18.800989] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=2036418][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:09.867771] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 8919769, , BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:09.867771] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:18.801048] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=8919769][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:18.801059] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task use too much time(end=1755162258801058, start=1755162249867769, delta=8933289) [2025-08-14 09:04:16.598245] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2202292, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2202278, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=10], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:16.598245] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:18.803899] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=2202292][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:18.806593] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:11.247290] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 7551682, , BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:10.870713] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=332][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3874398, , BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:11.247290] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:18.810419] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=7551682][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:18.810860] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=4959829] gc stale ls task succ [705]OB_MALLOC COST TOO MUCH TIME, cost_time=1032759, tenant_id=1001, label=OB_LOG_REGISTER_PARENT_REQ, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [2025-08-14 09:04:18.814753] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=4928451] gc stale ls task succ [2025-08-14 09:04:19.017983] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:19.648454] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:10.776338] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=347][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3961998, owner: ObLog, click_count: 8, time dist:[FORMAT_END=7, ObMalloc#start=1, ObMalloc#end=1972573, ObMalloc#start=20, ObMalloc#end=1989275, FREE_END=122, ALLOC_BLOCK#start=1006271, ALLOC_BLOCK#end=18], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:10.767407] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=471][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3972167, owner: ObLog, click_count: 6, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=981202, ObMalloc#start=21, ObMalloc#end=2987321, FREE_END=3613], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:11.751273] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=2092][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2992801, owner: ObLog, click_count: 8, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=2986361, ObMalloc#start=0, ObMalloc#end=49, FREE_END=6381, ALLOC_BLOCK#start=1011538, ALLOC_BLOCK#end=10], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:11.724594] EDIAG [RS.COMPACTION] try_idle (ob_major_merge_scheduler.cpp:226) [1057][T1_MergeSchedul][T1][YB42AC4D0007-00063C4961599CC3-0-0] [lt=1993103][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3014978, , BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1125d502 0x1125cf81 0x1125a4c3 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:19.071048] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=452][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:09.739340] EDIAG print_info (ob_kvcache_hazard_domain.cpp:138) [964][T1002_L0_G0][T1002][YB42AC4D0007-00063C4962EA9002-0-0] [lt=32][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4998568, , BACKTRACE: 0x26a1d7e0 0x26a180ba 0x99daaa6 0x99da47c 0xf9845d6 0x1fc9de13 0x1fc8ea3b 0x1fc7a380 0x96f94a6 0x96f8efa 0x9312121 0x930ecab 0x930dc16 0x930b653 0x92d4d44 0x92d1397 0x9292579 0x9285f13 0x927efad 0x926c154 0x12c5a0c9 0x2668bbb2 [2025-08-14 09:04:09.878581] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=510][errcode=0] LOGGER COST TOO MUCH TIME, cost: 9877609, , BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:09.878581] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=510][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:19.771170] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=9877609][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:19.772533] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2971137] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:19.772562] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:19.773526] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1985986] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:19.773557] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:19.789135] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:19.798116] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:19.800985] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=1029717] current monitor number(seq_=-1) [2025-08-14 09:04:19.801655] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:19.801698] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:19.810923] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=12] gc stale ls task succ [2025-08-14 09:04:19.814806] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=54] gc stale ls task succ [2025-08-14 09:04:19.864984] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:19.886489] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:19.995598] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:20.025718] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:20.095046] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=2859733][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:20.096174] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=918184, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=58, WASH_BLOCK_END=918120, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=5], seq:[0, 1, 2, 3, 4] [808]OB_MALLOC COST TOO MUCH TIME, cost_time=938687, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=41, WASH_BLOCK_END=938645], seq:[0, 1, 2] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=938812, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=43, WASH_BLOCK_END=938769], seq:[0, 1, 2] [2025-08-14 09:04:19.794325] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=6794] gc stale ls task succ [2025-08-14 09:04:20.714463] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=939272][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:08.803525] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] [thread pool stat](pool_stat={pool:0x564d6e8922c0, last_check_time:1755162248803523, last_idle_time:102477764956, last_thread_count:12}, *pool={name:TimerWK, this:0x564d6e8922c0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:10, threads_idle_time:102477764956, tenant_id:500}, qsize=10, current_idle_time=102477764956) [2025-08-14 09:04:09.737639] EDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [801][T1002_TntTransf][T1002][YB42AC4D0007-00063C496259A60F-0-0] [lt=51][errcode=-4012] LOGGER COST TOO MUCH TIME, cost: 11007305, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x124fe4b5 0x972619d 0x124dfd77 0x96a5a4d 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x9a64aaf 0x10502edb 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:09.737639] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [801][T1002_TntTransf][T1002][YB42AC4D0007-00063C496259A60F-0-0] [lt=51][errcode=-4012] result set close failed(ret=-4012) [2025-08-14 09:04:08.803525] EDIAG [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 11893955, owner: ObLog, click_count: 5, time dist:[FORMAT_END=11893947, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x989c577 0x989bc96 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:20.714507] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=30] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:20.770356] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3972167][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=996479, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=996444, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=996501, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=996494, WASH_BLOCK_END=7], seq:[0, 1, 2] [2025-08-14 09:04:18.759332] EDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=5519063][errcode=-4016] LOGGER COST TOO MUCH TIME, cost: 2009981, owner: ObLog, click_count: 8, time dist:[FORMAT_END=2009300, ObMalloc#start=0, ObMalloc#end=62, ObMalloc#start=23, ObMalloc#end=47, ObMalloc#start=368, ObMalloc#end=54, FREE_END=126], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x13b06ffb 0x958a39e 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:15.745114] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10134] [COMPACTION DAG STATUS] (start_time=1755162240733482, end_time=1755162255745111, dag_status={}) [2025-08-14 09:04:20.772541] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=466] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:20.772396] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=2009981][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:20.773552] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:20.773988] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=2045782] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:04:20.774262] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [403]OB_MALLOC COST TOO MUCH TIME, cost_time=956607, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=956583, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=943696, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=56, WASH_BLOCK_END=943640], seq:[0, 1, 2] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=943657, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=35, WASH_BLOCK_END=943622], seq:[0, 1, 2] [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=937100, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=937060], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=935854, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=28, WASH_BLOCK_END=935826], seq:[0, 1, 2] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=935940, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=59, WASH_BLOCK_END=935881], seq:[0, 1, 2] [2025-08-14 09:04:19.755436] EDIAG [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:233) [529][T1_RFLWorker][T1][YB42AC4D0007-00063C495B199C16-0-0] [lt=8027][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2018968, owner: ObLog, click_count: 6, time dist:[FORMAT_END=2018771, ObMalloc#start=1, ObMalloc#end=39, ObMalloc#start=21, ObMalloc#end=28, FREE_END=107], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x11359281 0x98b739c 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:18.758328] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=4951069][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3017371, owner: ObLog, click_count: 6, time dist:[FORMAT_END=3016397, ObMalloc#start=1, ObMalloc#end=58, ObMalloc#start=24, ObMalloc#end=28, FREE_END=863], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:18.758395] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=3962580][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3017586, owner: ObLog, click_count: 6, time dist:[FORMAT_END=3016484, ObMalloc#start=1, ObMalloc#end=22, ObMalloc#start=19, ObMalloc#end=29, FREE_END=1031], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:18.750884] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=4921858][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3025115, owner: ObLog, click_count: 6, time dist:[FORMAT_END=3023870, ObMalloc#start=1, ObMalloc#end=57, ObMalloc#start=24, ObMalloc#end=23, FREE_END=1140], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [730]OB_MALLOC COST TOO MUCH TIME, cost_time=1901975, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=26, WASH_BLOCK_END=1901918, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=30], seq:[0, 1, 2, 3, 4] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=1920882, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=1920854, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=1920939, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1920902, WASH_BLOCK_END=37], seq:[0, 1, 2] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=1901971, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=33, WASH_BLOCK_END=1901938], seq:[0, 1, 2] [656]OB_MALLOC COST TOO MUCH TIME, cost_time=1848137, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=34, WASH_BLOCK_END=1848103], seq:[0, 1, 2] [404]OB_MALLOC COST TOO MUCH TIME, cost_time=1624751, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=46, WASH_BLOCK_END=1624705], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=1508773, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=42, WASH_BLOCK_END=1508731], seq:[0, 1, 2] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=1475144, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=38, WASH_BLOCK_END=1475106], seq:[0, 1, 2] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=1445078, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=37, WASH_BLOCK_END=1445041], seq:[0, 1, 2] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=2945871, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=46, WASH_BLOCK_END=2945825], seq:[0, 1, 2] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=2946113, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=2946073], seq:[0, 1, 2] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=2951306, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=47, WASH_BLOCK_END=2951258], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=2952475, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=52, WASH_BLOCK_END=2952423], seq:[0, 1, 2] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=2954658, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=71, WASH_BLOCK_END=2954587], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=2952232, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=48, WASH_BLOCK_END=2952184], seq:[0, 1, 2] [861]OB_MALLOC COST TOO MUCH TIME, cost_time=995820, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=48, WASH_BLOCK_END=995772], seq:[0, 1, 2] [733]OB_MALLOC COST TOO MUCH TIME, cost_time=995772, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=24, WASH_BLOCK_END=995748], seq:[0, 1, 2] [571]OB_MALLOC COST TOO MUCH TIME, cost_time=994980, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=24, WASH_BLOCK_END=994956], seq:[0, 1, 2] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=2945834, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=48, WASH_BLOCK_END=2945786], seq:[0, 1, 2] [276]OB_MALLOC COST TOO MUCH TIME, cost_time=2951337, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=53, WASH_BLOCK_END=2951284], seq:[0, 1, 2] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=2945875, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=42, WASH_BLOCK_END=2945833], seq:[0, 1, 2] [513]OB_MALLOC COST TOO MUCH TIME, cost_time=2951555, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=45, WASH_BLOCK_END=2951510], seq:[0, 1, 2] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=1950824, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=956630, WASH_BLOCK_END=994194], seq:[0, 1, 2] [376]OB_MALLOC COST TOO MUCH TIME, cost_time=994182, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=24, WASH_BLOCK_END=994157], seq:[0, 1, 2] [2025-08-14 09:04:20.770267] EDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=2992801][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1951440, owner: ObLog, click_count: 8, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=102, ObMalloc#start=23, ObMalloc#end=1950832, ObMalloc#start=122, ObMalloc#end=36, FREE_END=315], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:19.773694] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2947012, owner: ObLog, click_count: 6, time dist:[FORMAT_END=6, ObMalloc#start=1, ObMalloc#end=2945882, ObMalloc#start=18, ObMalloc#end=36, FREE_END=1069], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [1275]OB_MALLOC COST TOO MUCH TIME, cost_time=987570, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 16, time dist:[ALLOC_CHUNK_END=16, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=987479, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] [2025-08-14 09:04:19.773586] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2013396][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2947248, owner: ObLog, click_count: 6, time dist:[FORMAT_END=6, ObMalloc#start=0, ObMalloc#end=2945842, ObMalloc#start=19, ObMalloc#end=36, FREE_END=1345], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:09.780665] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=145][errcode=0] LOGGER COST TOO MUCH TIME, cost: 11981740, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:19.766863] EDIAG eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=540][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2954176, owner: ObLog, click_count: 6, time dist:[FORMAT_END=13, ObMalloc#start=1, ObMalloc#end=2952239, ObMalloc#start=17, ObMalloc#end=35, FREE_END=1871], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x27a8a8c8 0x97b7e1a 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:19.772761] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2948601, owner: ObLog, click_count: 8, time dist:[ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=25, FORMAT_END=38, ObMalloc#start=0, ObMalloc#end=2945879, ObMalloc#start=18, ObMalloc#end=47, FREE_END=2590], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:19.774066] EDIAG [STORAGE] scheduler_transfer_handler_ (ob_transfer_service.cpp:200) [866][T1002_TransferS][T1002][YB42AC4D0007-00063C495F199DA2-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2947388, owner: ObLog, click_count: 6, time dist:[FORMAT_END=2942124, ObMalloc#start=1, ObMalloc#end=32, ObMalloc#start=20, ObMalloc#end=22, FREE_END=5189], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b2d5f98 0x1b2d480d 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:14.847428] EDIAG try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=3067033][errcode=0] LOGGER COST TOO MUCH TIME, cost: 7866626, , BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x26499505 0x97074a3 0x26498fa9 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:19.772618] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2002454][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2948896, owner: ObLog, click_count: 8, time dist:[ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=22, FORMAT_END=36, ObMalloc#start=0, ObMalloc#end=2946121, ObMalloc#start=18, ObMalloc#end=40, FREE_END=2656], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:19.766839] EDIAG [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=4971635][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2954764, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1958710, ObMalloc#start=0, ObMalloc#end=994191, ObMalloc#start=40, ObMalloc#end=31, FREE_END=1792], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1fce4e91 0x1fce455d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:13.874264] EDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=832972][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 8838466, owner: ObLog, click_count: 8, time dist:[FORMAT_END=5884919, ObMalloc#start=1, ObMalloc#end=2951563, ObMalloc#start=17, ObMalloc#end=35, ObMalloc#start=808, ObMalloc#end=30, FREE_END=1093], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x9757ced 0x97569df 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:19.852564] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [571][T1_ArcSender][T1][YB42AC4D0007-00063C495BA99C16-0-0] [lt=715680][errcode=-4018] LOGGER COST TOO MUCH TIME, cost: 2869229, owner: ObLog, click_count: 8, time dist:[FORMAT_END=1871912, ObMalloc#start=0, ObMalloc#end=994989, ObMalloc#start=17, ObMalloc#end=35, ObMalloc#start=1042, ObMalloc#end=31, FREE_END=1203], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3b4b 0x113d34aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:19.851704] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=2887933][errcode=-4018] LOGGER COST TOO MUCH TIME, cost: 2870111, owner: ObLog, click_count: 8, time dist:[FORMAT_END=1871783, ObMalloc#start=2, ObMalloc#end=995827, ObMalloc#start=17, ObMalloc#end=38, ObMalloc#start=1417, ObMalloc#end=29, FREE_END=998], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3b4b 0x113d34aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:19.851227] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=2897546][errcode=-4018] LOGGER COST TOO MUCH TIME, cost: 2870609, owner: ObLog, click_count: 8, time dist:[FORMAT_END=1872383, ObMalloc#start=0, ObMalloc#end=995779, ObMalloc#start=17, ObMalloc#end=37, ObMalloc#start=1268, ObMalloc#end=29, FREE_END=1096], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3b4b 0x113d34aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:21.275789] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1448218, owner: ObLog, click_count: 8, time dist:[FORMAT_END=27, ObMalloc#start=1, ObMalloc#end=1445086, ObMalloc#start=17, ObMalloc#end=37, ObMalloc#start=1786, ObMalloc#end=37, FREE_END=1227], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:17.318456] EDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=336][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 5399656, owner: ObLog, click_count: 8, time dist:[FORMAT_END=2446000, ObMalloc#start=1, ObMalloc#end=2951345, ObMalloc#start=17, ObMalloc#end=36, ObMalloc#start=988, ObMalloc#end=36, FREE_END=1233], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x9757ced 0x97569df 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:21.095618] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1628331, owner: ObLog, click_count: 8, time dist:[FORMAT_END=29, ObMalloc#start=1, ObMalloc#end=1624759, ObMalloc#start=17, ObMalloc#end=38, ObMalloc#start=2163, ObMalloc#end=36, FREE_END=1288], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:19.764354] EDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3961998][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2957607, owner: ObLog, click_count: 8, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=2954674, ObMalloc#start=18, ObMalloc#end=30, ObMalloc#start=1845, ObMalloc#end=39, FREE_END=989], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:21.211825] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1512429, owner: ObLog, click_count: 8, time dist:[FORMAT_END=49, ObMalloc#start=1, ObMalloc#end=1508786, ObMalloc#start=17, ObMalloc#end=38, ObMalloc#start=2627, ObMalloc#end=35, FREE_END=875], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:20.817829] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1905904, owner: ObLog, click_count: 8, time dist:[FORMAT_END=23, ObMalloc#start=0, ObMalloc#end=1901979, ObMalloc#start=27, ObMalloc#end=30, ObMalloc#start=2899, ObMalloc#end=43, FREE_END=903], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:21.245600] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=2][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1478921, owner: ObLog, click_count: 8, time dist:[FORMAT_END=35, ObMalloc#start=1, ObMalloc#end=1475151, ObMalloc#start=17, ObMalloc#end=36, ObMalloc#start=1759, ObMalloc#end=58, FREE_END=1864], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:20.798729] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1925323, owner: ObLog, click_count: 8, time dist:[FORMAT_END=30, ObMalloc#start=1, ObMalloc#end=1920946, ObMalloc#start=21, ObMalloc#end=30, ObMalloc#start=4147, ObMalloc#end=35, FREE_END=113], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:20.814878] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1909944, owner: ObLog, click_count: 6, time dist:[FORMAT_END=8, ObMalloc#start=1, ObMalloc#end=1902008, ObMalloc#start=27, ObMalloc#end=67, FREE_END=7833], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:20.770219] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1034013][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1963828, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1963615, ObMalloc#start=1, ObMalloc#end=78, ObMalloc#start=26, ObMalloc#end=39, FREE_END=69], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:20.770152] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2068203][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1964019, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1963900, ObMalloc#start=0, ObMalloc#end=37, ObMalloc#start=19, ObMalloc#end=29, FREE_END=34], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:20.770178] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1964048, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1963774, ObMalloc#start=0, ObMalloc#end=38, ObMalloc#start=19, ObMalloc#end=36, FREE_END=181], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:20.770193] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1033835][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1964090, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1963646, ObMalloc#start=1, ObMalloc#end=50, ObMalloc#start=24, ObMalloc#end=30, FREE_END=339], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:20.770774] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=406][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1963646, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1963381, ObMalloc#start=1, ObMalloc#end=59, ObMalloc#start=0, ObMalloc#end=55, FREE_END=150], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:15.745114] EDIAG [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10134][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5018959, owner: ObLog, click_count: 5, time dist:[FORMAT_END=5018950, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1af3d850 0x1af3c294 0x20071394 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [808]OB_MALLOC COST TOO MUCH TIME, cost_time=2027704, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=2027702], seq:[0] [2025-08-14 09:04:19.766974] EDIAG [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=2992605][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2990105, owner: ObLog, click_count: 6, time dist:[FORMAT_END=2973241, ObMalloc#start=1, ObMalloc#end=16711, ObMalloc#start=21, ObMalloc#end=43, FREE_END=87], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0xfb90f6f 0xfb90ac1 0x9941516 0x993eecd 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:20.770257] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1027831][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1989424, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1964106, ObMalloc#start=0, ObMalloc#end=24119, ObMalloc#start=33, ObMalloc#end=861, FREE_END=305], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:18.758899] EDIAG [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=4983580][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4002824, owner: ObLog, click_count: 6, time dist:[FORMAT_END=4001537, ObMalloc#start=1, ObMalloc#end=49, ObMalloc#start=18, ObMalloc#end=36, FREE_END=1183], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a5bbf2 0x94da542 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:20.769246] EDIAG [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5325][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1995805, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1995545, ObMalloc#start=1, ObMalloc#end=55, ObMalloc#start=20, ObMalloc#end=38, FREE_END=146], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1f9d7fcf 0x99b8be2 0x1f981833 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:20.770822] EDIAG [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=218][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1994486, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1993982, ObMalloc#start=1, ObMalloc#end=68, ObMalloc#start=23, ObMalloc#end=48, FREE_END=364], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x12377491 0x1236e8e9 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:21.736191] EDIAG [RS.COMPACTION] try_idle (ob_major_merge_scheduler.cpp:226) [1275][T1001_MergeSche][T1001][YB42AC4D0007-00063C4963E99CBE-0-0] [lt=29][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1030700, owner: ObLog, click_count: 8, time dist:[ALLOC_BLOCK#start=18, ALLOC_BLOCK#end=987602, FORMAT_END=42763, ObMalloc#start=0, ObMalloc#end=39, ObMalloc#start=26, ObMalloc#end=117, FREE_END=135], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1125d502 0x1125cf81 0x1125a4c3 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:18.820255] EDIAG [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [412][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=964270][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3942249, owner: ObLog, click_count: 6, time dist:[FORMAT_END=3942076, ObMalloc#start=0, ObMalloc#end=51, ObMalloc#start=18, ObMalloc#end=34, FREE_END=70], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a5bbf2 0x94da542 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:20.764077] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [339][EventTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2001834, owner: ObLog, click_count: 8, time dist:[FORMAT_END=2001044, ObMalloc#start=0, ObMalloc#end=32, ObMalloc#start=19, ObMalloc#end=65, ObMalloc#start=145, ObMalloc#end=47, FREE_END=482], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:19.766872] EDIAG [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [854][T1002_ObXAHbWor][T1002][Y0-0000000000000000-0-0] [lt=13866][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2997580, owner: ObLog, click_count: 6, time dist:[FORMAT_END=2996538, ObMalloc#start=0, ObMalloc#end=45, ObMalloc#start=18, ObMalloc#end=584, FREE_END=394], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b64d8ba 0x1b64d154 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:20.770241] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1018954][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1996416, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1964141, ObMalloc#start=1, ObMalloc#end=25040, ObMalloc#start=23, ObMalloc#end=44, FREE_END=7167], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [392]OB_MALLOC COST TOO MUCH TIME, cost_time=1974793, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1974791], seq:[0] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=2935964, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=961075, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1974843], seq:[0, 1, 2, 3, 4] [2025-08-14 09:04:20.774621] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2935990, owner: ObLog, click_count: 5, time dist:[FORMAT_END=5, ObMalloc#start=0, ObMalloc#end=2935976, ALLOC_END=2, APPEND_END=7], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:20.774621] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:23.715095] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:23.715136] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:23.726132] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=1463] PNIO [ratelimit] time: 1755162263726130, bytes: 0, bw: 0.000000 MB/s, add_ts: 1000593, add_bytes: 0 [2025-08-14 09:04:23.735571] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=346] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:23.740310] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=339] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [808]OB_MALLOC COST TOO MUCH TIME, cost_time=3022430, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2027717], seq:[0, 1] [2025-08-14 09:04:16.166036] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=267][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:16.166036] EDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=267][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 7563582, owner: ObLog, click_count: 7, time dist:[FORMAT_END=3602382, ObMalloc#start=1, ObMalloc#end=938695, ObMalloc#start=41, ObMalloc#end=3022453, ALLOC_END=2, APPEND_END=8], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x9757ced 0x97569df 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:23.741090] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=7563582][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:23.748357] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=807] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:23.729528] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=673] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:22.746567] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=3963508] temporary file module use no blocks [2025-08-14 09:04:22.746567] EDIAG [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=3963508][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1008054, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1008042, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=8], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x1d0f5b0a 0x1d0f57c6 0x1d17ad7c 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:22.744322] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=3951559] temporary file module use no blocks [2025-08-14 09:04:22.744322] EDIAG [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=3951559][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1014408, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1014395, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=8], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x1d0f5b0a 0x1d0f57c6 0x1d17ad7c 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:23.764128] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=7064] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:23.740344] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=251] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [705]OB_MALLOC COST TOO MUCH TIME, cost_time=993417, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=993414], seq:[0] [832]OB_MALLOC COST TOO MUCH TIME, cost_time=987149, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=987147], seq:[0] [832]OB_MALLOC COST TOO MUCH TIME, cost_time=993714, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=43, WASH_BLOCK_END=6504, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=987162], seq:[0, 1, 2, 3, 4] [2025-08-14 09:04:22.744328] EDIAG [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=3959308][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2014981, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1021249, ObMalloc#start=1, ObMalloc#end=993724, ALLOC_END=1, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x1d0f5b0a 0x1d0f57c6 0x1d17ad7c 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:09.747540] EDIAG [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=480363][errcode=0] LOGGER COST TOO MUCH TIME, cost: 12958579, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b52e5d4 0x996d21d 0x996cfd8 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:20.774383] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3986896, owner: ObLog, click_count: 8, time dist:[ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=1, FORMAT_END=3984140, ObMalloc#start=1, ObMalloc#end=65, ObMalloc#start=24, ObMalloc#end=60, FREE_END=2602], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [705]OB_MALLOC COST TOO MUCH TIME, cost_time=1001089, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=55, WASH_BLOCK_END=69, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=993439], seq:[0, 1, 2, 3, 4] [2025-08-14 09:04:22.744328] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=3959308] temporary file module use no blocks [2025-08-14 09:04:23.766202] WDIAG [RPC.OBRPC] in_black (ob_net_keepalive.cpp:257) [705][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=197][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:24.774117] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=226] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:24.774769] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=230] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:24.774912] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=238] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:04:24.775489] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2258] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:24.768727] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162264768726, start=1755162263845139, delta=923587) [2025-08-14 09:04:24.727642] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=811] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:24.778818] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=51098][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:09.768068] EDIAG [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] LOGGER COST TOO MUCH TIME, cost: 14970346, owner: ObLog, click_count: 6, time dist:[FORMAT_END=14969471, ObMalloc#start=2, ObMalloc#end=736, ObMalloc#start=21, ObMalloc#end=43, FREE_END=73], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x9c6174d 0xfd347d6 0xfd344b7 0xfd34254 0x1c029398 0x97a1ad0 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [387]OB_MALLOC COST TOO MUCH TIME, cost_time=940483, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=940476], seq:[0] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=939337, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=939306, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=939404, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=939346, WASH_BLOCK_END=58], seq:[0, 1, 2] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=934927, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=934900, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=934968, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=934953, WASH_BLOCK_END=15], seq:[0, 1, 2] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=928286, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=928241, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=928303, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=928295, WASH_BLOCK_END=8], seq:[0, 1, 2] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=911767, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=911723, ALLOC_CHUNK_END=11], seq:[0, 1, 2, 3, 4, 5] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=909655, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=909621, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3, 4, 5] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=909683, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=909668, WASH_BLOCK_END=15], seq:[0, 1, 2] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=911834, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=911790, WASH_BLOCK_END=44], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=895663, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=16, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=895622, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=895721, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=895681, WASH_BLOCK_END=40], seq:[0, 1, 2] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=881378, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=881347, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=632011, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=631977, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=632043, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=632028, WASH_BLOCK_END=14], seq:[0, 1, 2] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=600209, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=600177, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=600236, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=600220, WASH_BLOCK_END=16], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=751088, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=25, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=751036, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=751382, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=751367, WASH_BLOCK_END=14], seq:[0, 1, 2] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=881399, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=881390, WASH_BLOCK_END=9], seq:[0, 1, 2] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=739597, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=739567, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=739628, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=739614, WASH_BLOCK_END=14], seq:[0, 1, 2] [387]OB_MALLOC COST TOO MUCH TIME, cost_time=963573, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=55, WASH_BLOCK_END=82, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=940491], seq:[0, 1, 2, 3, 4] [2025-08-14 09:04:24.768884] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=38070] log compressor cycles once. (ret=0, cost_time=15050789, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25880477696, fast_delete_log_mode=false) [2025-08-14 09:04:25.746763] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=976298] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25880477696, fast_delete_log_mode=false) [2025-08-14 09:04:24.746182] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:25.747030] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=999335][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:25.747053] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:23.780242] EDIAG [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=1014408][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1964227, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1964218, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9826e08 0x98267df 0x982650b 0x98262e0 0x1d188ec1 0x1d17b8ec 0x1d17ad88 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:23.773502] EDIAG [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=1008054][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1970994, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1970988, ObMalloc#start=0, ObMalloc#end=2, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9826e08 0x98267df 0x982650b 0x98262e0 0x1d188ec1 0x1d17b8ec 0x1d17ad88 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:23.780242] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=1014408] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:04:23.773502] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=1008054] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [372]OB_MALLOC COST TOO MUCH TIME, cost_time=967950, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=967948], seq:[0] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=967970, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=967965], seq:[0, 1] [2025-08-14 09:04:09.759726] EDIAG [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] LOGGER COST TOO MUCH TIME, cost: 15969134, owner: ObLog, click_count: 3, time dist:[FORMAT_END=15969130, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b51df0f 0x986bcde 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:08.739228] EDIAG [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=0] LOGGER COST TOO MUCH TIME, cost: 16990794, owner: ObLog, click_count: 3, time dist:[FORMAT_END=16990785, ALLOC_END=4, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b51df0f 0x986bcde 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:24.783218] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:09.759726] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=23] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136826952869001, v:0}, clog_base_lsn:{lsn:5099962368}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1755071646732636000, v:0}, all_id_meta:{id_meta:[{limited_id:1755161555795565000, latest_log_ts:{val:1755161545796642001, v:0}}, {limited_id:22000001, latest_log_ts:{val:1755135645787806003, v:0}}, {limited_id:60000001, latest_log_ts:{val:1755135646400653001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:50, palf_env_:0x7f9950bf4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-08-14 09:04:25.723811] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=251] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:04:25.754060] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=30190] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:04:08.739228] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=20] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136829134139001, v:0}, clog_base_lsn:{lsn:8186781696}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1753840867372367034, v:0}, all_id_meta:{id_meta:[{limited_id:1755161729775179000, latest_log_ts:{val:1755161719775179001, v:0}}, {limited_id:18000001, latest_log_ts:{val:1755157440846556001, v:0}}, {limited_id:60000001, latest_log_ts:{val:1755135657266496001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:46, palf_env_:0x7f99321fa030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-08-14 09:04:25.724869] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=7866626] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:04:25.746625] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5018959] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:04:25.758822] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12177] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:04:25.758836] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:04:25.758845] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:04:25.711945] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=2954764] blacklist_loop exec finished(cost_time=19, is_enabled=true, send_cnt=0) [2025-08-14 09:04:25.763209] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=304] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:24.855705] INFO [ARCHIVE] do_thread_task_ (ob_archive_timer.cpp:125) [734][T1001_ArcTimer][T1001][YB42AC4D0007-00063C495CC99C16-0-0] [lt=1902512] archive round not in doing status, just skip(key={incarnation:-1, dest_id:-1, round:-1}, state={status:"INVALID"}) [2025-08-14 09:04:25.735325] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:199) [801][T1002_TntTransf][T1002][YB42AC4D0007-00063C496259A60F-0-0] [lt=11007305][errcode=-4012] result set close failed(ret=-4012) [2025-08-14 09:04:25.753784] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:826) [964][T1002_L0_G0][T1002][YB42AC4D0007-00063C4962EA9002-0-0] [lt=989253] can not find enough memory block to wash(ret=-4273, size_washed=0, size_to_wash=6291456) [2025-08-14 09:04:25.769777] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [339][EventTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:25.774460] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:25.774782] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:25.774991] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=15] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:04:25.775327] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=200] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:25.775366] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:25.779828] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=489] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:25.791750] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:25.794704] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=930082] gc stale ls task succ [2025-08-14 09:04:25.811312] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=911971] gc stale ls task succ [2025-08-14 09:04:25.856827] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=31796] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:04:25.882731] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=358][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:25.893788] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=370][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:25.929766] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:25.930324] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:26.021921] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:26.030481] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:26.096399] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:26.129343] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=600609][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:26.244718] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:26.253081] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:26.508376] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:26.715268] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1161] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:26.727633] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:951) [801][T1002_TntTransf][T1002][YB42AC4D0007-00063C496259A60F-0-0] [lt=29433][errcode=-4012] failed to close result(close_ret=-4012, ret=-4012) [2025-08-14 09:04:26.728524] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=174] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:26.729862] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=359] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:25.758853] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:04:26.736699] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=976366] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:04:25.767278] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:845) [964][T1002_L0_G0][T1002][YB42AC4D0007-00063C4962EA9002-0-0] [lt=13470] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_to_wash=6291456) [2025-08-14 09:04:26.719661] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=1970994] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:04:26.721262] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=1964227] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:04:26.719376] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=984562] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:04:26.738278] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=232] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:26.738456] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=167] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:26.736712] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:04:26.741785] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5063] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:04:26.741796] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:04:26.728577] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=252] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:26.747139] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18532][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:26.747356] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=215] PNIO [ratelimit] time: 1755162266747354, bytes: 0, bw: 0.000000 MB/s, add_ts: 1012873, add_bytes: 0 [2025-08-14 09:04:26.747451] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=568] current monitor number(seq_=-1) [2025-08-14 09:04:26.729901] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=172] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:26.748378] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18449][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:22.760713] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11893955] [thread pool stat](pool_stat={pool:0x7f99667cbcf0, last_check_time:1755162262760709, last_idle_time:34871961304, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x7f99667cbcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:34875884514, tenant_id:500}, qsize=-1, current_idle_time=34871961304) [2025-08-14 09:04:22.760713] EDIAG [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11893955][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3985607, owner: ObLog, click_count: 5, time dist:[FORMAT_END=3985597, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x989c577 0x989bc96 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:26.756690] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=3985607] [thread pool stat](pool_stat={pool:0x7f99667ccaf0, last_check_time:1755162266756689, last_idle_time:26650329148, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x7f99667ccaf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:26650329148, tenant_id:500}, qsize=-1, current_idle_time=26650329148) [2025-08-14 09:04:26.756717] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=24] [thread pool stat](pool_stat={pool:0x7f9969de56b0, last_check_time:1755162266756716, last_idle_time:102938681542, last_thread_count:17}, *pool={name:TimerWK, this:0x7f9969de56b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:16, threads_idle_time:102938681542, tenant_id:1}, qsize=10, current_idle_time=102938681542) [2025-08-14 09:04:26.756729] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] [thread pool stat](pool_stat={pool:0x7f99356b0e70, last_check_time:1755162266756728, last_idle_time:102366343177, last_thread_count:15}, *pool={name:TimerWK, this:0x7f99356b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:12, threads_idle_time:102366343177, tenant_id:1001}, qsize=12, current_idle_time=102366343177) [2025-08-14 09:04:26.756740] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] [thread pool stat](pool_stat={pool:0x7f99230b0e70, last_check_time:1755162266756739, last_idle_time:102534969158, last_thread_count:17}, *pool={name:TimerWK, this:0x7f99230b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:16, threads_idle_time:102534969158, tenant_id:1002}, qsize=15, current_idle_time=102534969158) [2025-08-14 09:04:25.764378] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15969134] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:04:25.761435] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=16990794] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:04:26.760668] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=4002824] [sql nio session](*s={this:0x7f98f72d0030, session_id:3221589415, trace_id:YB42AC4D0007-00063C4965FA43DC-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:245}, err:0, last_decode_time:1755161751714366, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1662, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:04:26.760700] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=32] [sql nio session](*s={this:0x7f98f72d5030, session_id:3221590200, trace_id:YB42AC4D0007-00063C4965FA43E0-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:248}, err:0, last_decode_time:1755161776777167, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1593, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:04:26.773592] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:26.773641] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:26.774492] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:26.774592] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:26.780825] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:26.794767] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=16] gc stale ls task succ [2025-08-14 09:04:26.812264] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=24] gc stale ls task succ [2025-08-14 09:04:26.857634] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=17] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:04:27.716462] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=30] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:27.727314] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:27.727865] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:27.729073] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:27.729714] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:27.730152] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=599] gc stale ls task succ [2025-08-14 09:04:27.731994] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=524] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:26.748417] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:04:27.739381] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:27.741259] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:27.741296] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=205] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:27.749095] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=21] PNIO [ratelimit] time: 1755162267749092, bytes: 0, bw: 0.000000 MB/s, add_ts: 1001738, add_bytes: 0 [2025-08-14 09:04:27.751984] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:66) [964][T1002_L0_G0][T1002][YB42AC4D0007-00063C4962EA9002-0-0] [lt=968232][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-08-14 09:04:27.769274] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:27.771262] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:27.774262] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:27.775580] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=232] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:04:27.776275] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:27.783280] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:27.812734] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=24] gc stale ls task succ [2025-08-14 09:04:27.867926] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=18] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:04:28.685702] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=71450][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:28.716683] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=49] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:28.727890] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:28.728695] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:28.729275] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:28.730110] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=35] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:28.730164] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3079] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:28.730201] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:27.733565] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12958579][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:28.740106] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:28.741609] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:28.756704] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=30] PNIO [ratelimit] time: 1755162268756700, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007608, add_bytes: 0 [401]OB_MALLOC COST TOO MUCH TIME, cost_time=1996954, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1996953], seq:[0] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=1997102, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=64, WASH_BLOCK_END=58, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1996975], seq:[0, 1, 2, 3, 4] [2025-08-14 09:04:26.763218] EDIAG [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1997133, owner: ObLog, click_count: 5, time dist:[FORMAT_END=7, ObMalloc#start=0, ObMalloc#end=1997120, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a1f867 0x27a135c3 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:26.737057] EDIAG [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=17654][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2023336, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2023329, ObMalloc#start=0, ObMalloc#end=3, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1d17bb52 0x1d17ada0 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:26.763218] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=13] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:28.763396] WDIAG ev_invoke_pending (ev.c:2136) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=1997133][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:28.763413] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=10] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:28.763425] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=8] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:26.737057] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=17654] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623152218754, flush_io_finished_ret_:0, flush_io_finished_round_:26334, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:27.826411] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=2870111][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:04:26.737007] EDIAG [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=17320][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2029712, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2029700, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=1, APPEND_END=7], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1d17bb52 0x1d17ada0 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:26.737007] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=17320] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623152215785, flush_io_finished_ret_:0, flush_io_finished_round_:26366, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:27.831255] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=2870609][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:04:26.737035] EDIAG [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=15749][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2030292, owner: ObLog, click_count: 6, time dist:[FORMAT_END=2029738, ObMalloc#start=1, ObMalloc#end=71, ObMalloc#start=22, ObMalloc#end=41, FREE_END=419], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1d17bb52 0x1d17ada0 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:28.777698] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=17] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:04:28.779983] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:28.780021] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:28.780056] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:28.812800] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=39] gc stale ls task succ [2025-08-14 09:04:28.880039] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=23] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:04:29.716759] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:29.726414] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:29.728148] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:29.729215] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:29.731139] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:29.731175] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:28.762991] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=30954][errcode=-4015] clock out of order(cur_ts=1755162266777300, cur_ts_=1755162266752383, delta=24917) [2025-08-14 09:04:26.953352] EDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 2777854, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2777842, ObMalloc#start=1, ObMalloc#end=5, ALLOC_END=0, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x9757ced 0x97569df 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [383]OB_MALLOC COST TOO MUCH TIME, cost_time=992196, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=34, WASH_BLOCK_END=992159, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1, 2, 3, 4] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=1003773, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=60, WASH_BLOCK_END=1003667, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=45], seq:[0, 1, 2, 3, 4] [575]OB_MALLOC COST TOO MUCH TIME, cost_time=999339, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=60, WASH_BLOCK_END=999279], seq:[0, 1, 2] [737]OB_MALLOC COST TOO MUCH TIME, cost_time=999324, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=999284], seq:[0, 1, 2] [387]OB_MALLOC COST TOO MUCH TIME, cost_time=992668, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=992628], seq:[0, 1, 2] [389]OB_MALLOC COST TOO MUCH TIME, cost_time=992593, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=32, WASH_BLOCK_END=992561], seq:[0, 1, 2] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=954454, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 16, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=954384, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] [2025-08-14 09:04:26.953352] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:28.741741] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2773] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:29.735465] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=992222][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:29.736041] WDIAG ev_invoke_pending (ev.c:2136) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=992738][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:29.736073] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=992925][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:27.733708] EDIAG [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=994622][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1999370, owner: ObLog, click_count: 6, time dist:[FORMAT_END=999675, ObMalloc#start=0, ObMalloc#end=999332, ObMalloc#start=19, ObMalloc#end=51, FREE_END=293], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b51d070 0x986d718 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:27.733696] EDIAG [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=990904][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1999412, owner: ObLog, click_count: 6, time dist:[FORMAT_END=999568, ObMalloc#start=1, ObMalloc#end=999348, ObMalloc#start=27, ObMalloc#end=53, FREE_END=415], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b51d070 0x986d718 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:29.739139] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:29.739975] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:29.740242] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:29.741993] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:28.770624] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=983857] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:04:29.740110] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:29.762092] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21947][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:29.736123] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=1999370] try gc retain ctx [2025-08-14 09:04:29.736147] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1999412] try gc retain ctx [2025-08-14 09:04:29.764341] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=38] PNIO [ratelimit] time: 1755162269764339, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007639, add_bytes: 0 [2025-08-14 09:04:29.766322] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=7] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:29.770433] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [571]OB_MALLOC COST TOO MUCH TIME, cost_time=2971476, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=2971473], seq:[0] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=1968309, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=1968272, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=1968412, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1968333, WASH_BLOCK_END=79], seq:[0, 1, 2] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=1965502, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=1965482], seq:[0, 1, 2] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=1965523, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1965521], seq:[0, 1] [2025-08-14 09:04:29.774088] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1965549, owner: ObLog, click_count: 5, time dist:[FORMAT_END=11, ObMalloc#start=0, ObMalloc#end=1965530, ALLOC_END=2, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [332]OB_MALLOC COST TOO MUCH TIME, cost_time=1961936, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1961915, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=1961995, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1961944, WASH_BLOCK_END=51], seq:[0, 1, 2] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=1959452, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1959430, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=1959476, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1959461, WASH_BLOCK_END=15], seq:[0, 1, 2] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=1959549, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1959529, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=1959566, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1959557, WASH_BLOCK_END=9], seq:[0, 1, 2] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=1953017, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1952991, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=1953035, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1953026, WASH_BLOCK_END=9], seq:[0, 1, 2] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=1921016, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1920992, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=1921042, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1921033, WASH_BLOCK_END=9], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=1859758, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=1859720, ALLOC_CHUNK_END=14], seq:[0, 1, 2, 3] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=1859778, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1859767, WASH_BLOCK_END=11], seq:[0, 1, 2] [2025-08-14 09:04:29.880786] EDIAG try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1859817, owner: ObLog, click_count: 5, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=1859785, ALLOC_END=18, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x26499505 0x97074a3 0x26498fa9 0x7f9973e081cf 0x7f9973a39dd3 [392]OB_MALLOC COST TOO MUCH TIME, cost_time=1024202, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=18, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1024161, ALLOC_CHUNK_END=13], seq:[0, 1, 2, 3] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=1024227, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1024211, WASH_BLOCK_END=16], seq:[0, 1, 2] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=1012731, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=1012703, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=1012749, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1012740, WASH_BLOCK_END=9], seq:[0, 1, 2] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=1012783, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1012764, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=1012799, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1012791, WASH_BLOCK_END=8], seq:[0, 1, 2] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=1012865, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1012845, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=1012888, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1012878, WASH_BLOCK_END=10], seq:[0, 1, 2] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=1012946, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1012927, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=1012963, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1012954, WASH_BLOCK_END=8], seq:[0, 1, 2] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=1010877, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=1010853, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=1010894, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1010885, WASH_BLOCK_END=9], seq:[0, 1, 2] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=1004352, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1004328, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=1004370, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1004361, WASH_BLOCK_END=9], seq:[0, 1, 2] [2025-08-14 09:04:27.971652] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [869][T1002_MultiVers][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3766837, owner: ObLog, click_count: 3, time dist:[FORMAT_END=3766831, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0xf9fb357 0x97ae92e 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [385]OB_MALLOC COST TOO MUCH TIME, cost_time=1003293, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1003267, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=1003311, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1003303, WASH_BLOCK_END=8], seq:[0, 1, 2] [387]OB_MALLOC COST TOO MUCH TIME, cost_time=1003361, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1003343, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [387]OB_MALLOC COST TOO MUCH TIME, cost_time=1003376, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1003368, WASH_BLOCK_END=8], seq:[0, 1, 2] [389]OB_MALLOC COST TOO MUCH TIME, cost_time=1003441, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1003423, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [389]OB_MALLOC COST TOO MUCH TIME, cost_time=1003457, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1003450, WASH_BLOCK_END=7], seq:[0, 1, 2] [383]OB_MALLOC COST TOO MUCH TIME, cost_time=997147, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=997121, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [383]OB_MALLOC COST TOO MUCH TIME, cost_time=997196, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=997163, WASH_BLOCK_END=33], seq:[0, 1, 2] [575]OB_MALLOC COST TOO MUCH TIME, cost_time=997177, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=997158, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [575]OB_MALLOC COST TOO MUCH TIME, cost_time=997198, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=997186, WASH_BLOCK_END=12], seq:[0, 1, 2] [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=984865, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=984842, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=984882, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=984873, WASH_BLOCK_END=9], seq:[0, 1, 2] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=976813, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=976789, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=976831, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=976823, WASH_BLOCK_END=8], seq:[0, 1, 2] [2025-08-14 09:04:29.774088] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:31.742641] WDIAG ev_invoke_pending (ev.c:2136) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1965549][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:31.742667] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:29.880786] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=21] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:04:27.971652] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [869][T1002_MultiVers][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:31.744223] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [869][T1002_MultiVers][T1002][Y0-0000000000000000-0-0] [lt=3766837][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:30.746636] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:31.745392] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=997246][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:31.745416] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:30.739521] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1004755, owner: ObLog, click_count: 6, time dist:[FORMAT_END=8, ObMalloc#start=0, ObMalloc#end=1003383, ObMalloc#start=24, ObMalloc#end=41, FREE_END=1298], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:31.745818] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1004755][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:31.745846] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:29.762207] EDIAG [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=26020][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1980686, owner: ObLog, click_count: 6, time dist:[FORMAT_END=983070, ObMalloc#start=1, ObMalloc#end=997204, ObMalloc#start=19, ObMalloc#end=38, FREE_END=354], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x1b51d27b 0x986d830 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:30.731637] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1012748, owner: ObLog, click_count: 6, time dist:[FORMAT_END=9, ObMalloc#start=0, ObMalloc#end=1010901, ObMalloc#start=25, ObMalloc#end=41, FREE_END=1771], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:31.745936] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1012748][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:31.745953] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:30.729459] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2094][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1014984, owner: ObLog, click_count: 6, time dist:[FORMAT_END=6, ObMalloc#start=0, ObMalloc#end=1012969, ObMalloc#start=25, ObMalloc#end=41, FREE_END=1943], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:31.745996] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1014984][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:31.746013] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:30.729389] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1015111, owner: ObLog, click_count: 6, time dist:[FORMAT_END=7, ObMalloc#start=0, ObMalloc#end=1012805, ObMalloc#start=25, ObMalloc#end=43, FREE_END=2231], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:31.746052] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1015111][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:29.787134] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [477][T1_StandbySchem][T1][YB42AC4D0007-00063C495B6A0380-0-0] [lt=26][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1955996, owner: ObLog, click_count: 6, time dist:[FORMAT_END=12, ObMalloc#start=2, ObMalloc#end=1953041, ObMalloc#start=26, ObMalloc#end=42, FREE_END=2873], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:30.739551] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1005052, owner: ObLog, click_count: 6, time dist:[FORMAT_END=6, ObMalloc#start=0, ObMalloc#end=1003463, ObMalloc#start=26, ObMalloc#end=43, FREE_END=1513], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:31.746139] WDIAG ev_invoke_pending (ev.c:2136) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1005052][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:31.746157] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:31.746180] WDIAG ev_invoke_pending (ev.c:2136) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=977332][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:29.780482] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=170][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1962789, owner: ObLog, click_count: 6, time dist:[FORMAT_END=7, ObMalloc#start=0, ObMalloc#end=1959573, ObMalloc#start=25, ObMalloc#end=46, FREE_END=3138], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:31.746258] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1962789][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:31.746284] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:30.729316] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1015458, owner: ObLog, click_count: 6, time dist:[FORMAT_END=14, ObMalloc#start=1, ObMalloc#end=1012755, ObMalloc#start=29, ObMalloc#end=44, FREE_END=2614], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:31.746326] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=1015458][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:31.746343] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:30.729420] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1015410, owner: ObLog, click_count: 6, time dist:[FORMAT_END=5, ObMalloc#start=0, ObMalloc#end=1012896, ObMalloc#start=25, ObMalloc#end=43, FREE_END=2441], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:30.739470] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1005419, owner: ObLog, click_count: 6, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=1003318, ObMalloc#start=26, ObMalloc#end=41, FREE_END=2021], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:29.819288] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=22][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1924375, owner: ObLog, click_count: 6, time dist:[FORMAT_END=20, ObMalloc#start=1, ObMalloc#end=1921048, ObMalloc#start=27, ObMalloc#end=48, FREE_END=3231], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:31.746383] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1015410][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:30.717711] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=94][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1027446, owner: ObLog, click_count: 6, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=1024233, ObMalloc#start=20, ObMalloc#end=45, FREE_END=3135], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:31.746729] WDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1027446][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:31.746747] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:29.777793] EDIAG do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1966060, owner: ObLog, click_count: 6, time dist:[FORMAT_END=11, ObMalloc#start=0, ObMalloc#end=1962003, ObMalloc#start=23, ObMalloc#end=47, FREE_END=3976], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x202c530b 0x202c6888 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:31.746429] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1005419][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:31.746866] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=427] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:29.780440] EDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] LOGGER COST TOO MUCH TIME, cost: 1963485, owner: ObLog, click_count: 8, time dist:[FORMAT_END=16, ObMalloc#start=0, ObMalloc#end=1959495, ObMalloc#start=18, ObMalloc#end=46, ObMalloc#start=3782, ObMalloc#end=42, FREE_END=86], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x13b06ffb 0x958a39e 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:31.746920] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=1963485][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:29.770571] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [776][T1002_StandbySc][T1002][YB42AC4D0007-00063C495DFA0371-0-0] [lt=205][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1974443, owner: ObLog, click_count: 6, time dist:[FORMAT_END=7, ObMalloc#start=0, ObMalloc#end=1968426, ObMalloc#start=35, ObMalloc#end=57, FREE_END=5918], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [399]OB_MALLOC COST TOO MUCH TIME, cost_time=987823, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=987801, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=987864, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=987839, WASH_BLOCK_END=25], seq:[0, 1, 2] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=987781, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=18, ALLOC_CHUNK_END=987744, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=987836, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=987800, WASH_BLOCK_END=36], seq:[0, 1, 2] [737]OB_MALLOC COST TOO MUCH TIME, cost_time=987094, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=987075, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [737]OB_MALLOC COST TOO MUCH TIME, cost_time=987123, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=987104, WASH_BLOCK_END=19], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=985252, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=985229, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=985269, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=985261, WASH_BLOCK_END=8], seq:[0, 1, 2] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=979919, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=979897, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=979935, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=979927, WASH_BLOCK_END=8], seq:[0, 1, 2] [2025-08-14 09:04:31.152418] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=1628331][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:30.729366] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:29.762179] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=26014] try to retry start_working [2025-08-14 09:04:29.766419] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1990347, owner: ObLog, click_count: 8, time dist:[FORMAT_END=1973366, ObMalloc#start=1, ObMalloc#end=48, ObMalloc#start=25, ObMalloc#end=52, ObMalloc#start=16672, ObMalloc#end=38, FREE_END=145], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:31.759497] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=988738][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:29.771732] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:29.771732] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1992548, owner: ObLog, click_count: 9, time dist:[FORMAT_END=1967006, ObMalloc#start=3, ObMalloc#end=20265, ObMalloc#start=26, ObMalloc#end=45, ObMalloc#start=5190, ObMalloc#end=5, ALLOC_END=0, APPEND_END=8], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:31.768653] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=81] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:31.772753] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:31.779549] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=985521] PNIO [ratelimit] time: 1755162271779547, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007383, add_bytes: 0 [2025-08-14 09:04:31.780367] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:30.738280] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1006758, owner: ObLog, click_count: 6, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=1004377, ObMalloc#start=26, ObMalloc#end=50, FREE_END=2293], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [396]OB_MALLOC COST TOO MUCH TIME, cost_time=991227, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 12, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=991166, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] [466]OB_MALLOC COST TOO MUCH TIME, cost_time=936747, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=45, WASH_BLOCK_END=936701], seq:[0, 1, 2] [571]OB_MALLOC COST TOO MUCH TIME, cost_time=4949259, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2971503], seq:[0, 1] [2025-08-14 09:04:27.829269] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [571][T1_ArcSender][T1][YB42AC4D0007-00063C495BA99C16-0-0] [lt=218][errcode=-4018] LOGGER COST TOO MUCH TIME, cost: 5883536, owner: ObLog, click_count: 5, time dist:[FORMAT_END=934239, ObMalloc#start=0, ObMalloc#end=4949287, ALLOC_END=2, APPEND_END=7], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3b4b 0x113d34aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [395]OB_MALLOC COST TOO MUCH TIME, cost_time=972858, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=33, WASH_BLOCK_END=972779, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=45], seq:[0, 1, 2, 3, 4] [2025-08-14 09:04:27.829269] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [571][T1_ArcSender][T1][YB42AC4D0007-00063C495BA99C16-0-0] [lt=218][errcode=-4018] try free send task failed(ret=-4018) [394]OB_MALLOC COST TOO MUCH TIME, cost_time=989243, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=34, WASH_BLOCK_END=989209], seq:[0, 1, 2] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=987475, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=33, WASH_BLOCK_END=987442], seq:[0, 1, 2] [864]OB_MALLOC COST TOO MUCH TIME, cost_time=987465, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=34, WASH_BLOCK_END=987431], seq:[0, 1, 2] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=987499, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=33, WASH_BLOCK_END=987466], seq:[0, 1, 2] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=984302, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=37, WASH_BLOCK_END=984265], seq:[0, 1, 2] [386]OB_MALLOC COST TOO MUCH TIME, cost_time=3977431, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=3977429], seq:[0] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=1034213, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=33, WASH_BLOCK_END=1034179], seq:[0, 1, 2] [2025-08-14 09:04:32.747561] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=614][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1034402, owner: ObLog, click_count: 6, time dist:[FORMAT_END=8, ObMalloc#start=0, ObMalloc#end=1034231, ObMalloc#start=22, ObMalloc#end=46, FREE_END=95], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [332]OB_MALLOC COST TOO MUCH TIME, cost_time=1037072, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=34, WASH_BLOCK_END=1037038], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=1035028, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=34, WASH_BLOCK_END=1034994], seq:[0, 1, 2] [2025-08-14 09:04:32.747618] EDIAG do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=1966060][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1037284, owner: ObLog, click_count: 6, time dist:[FORMAT_END=10, ObMalloc#start=0, ObMalloc#end=1037102, ObMalloc#start=30, ObMalloc#end=46, FREE_END=96], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x202c530b 0x202c6888 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:29.841142] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3939467, owner: ObLog, click_count: 8, time dist:[FORMAT_END=2904152, ObMalloc#start=0, ObMalloc#end=1035057, ObMalloc#start=20, ObMalloc#end=53, ObMalloc#start=71, ObMalloc#end=42, FREE_END=72], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [393]OB_MALLOC COST TOO MUCH TIME, cost_time=1995367, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=56, WASH_BLOCK_END=1995271, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40], seq:[0, 1, 2, 3, 4] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=1983325, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=33, WASH_BLOCK_END=1983291], seq:[0, 1, 2] [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=1977530, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=33, WASH_BLOCK_END=1977497], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=1975496, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=1975456], seq:[0, 1, 2] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=1972544, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=37, WASH_BLOCK_END=1972507], seq:[0, 1, 2] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=1996920, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=45, WASH_BLOCK_END=1996875], seq:[0, 1, 2] [383]OB_MALLOC COST TOO MUCH TIME, cost_time=1983689, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=32, WASH_BLOCK_END=1983656], seq:[0, 1, 2] [2025-08-14 09:04:32.728253] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1997256, owner: ObLog, click_count: 6, time dist:[FORMAT_END=8, ObMalloc#start=0, ObMalloc#end=1996928, ObMalloc#start=19, ObMalloc#end=51, FREE_END=250], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:32.747395] EDIAG [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=985529][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1978374, owner: ObLog, click_count: 6, time dist:[FORMAT_END=10, ObMalloc#start=0, ObMalloc#end=1977541, ObMalloc#start=28, ObMalloc#end=44, FREE_END=751], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1f832f91 0x97c36fe 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:32.741606] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1984199, owner: ObLog, click_count: 6, time dist:[FORMAT_END=5, ObMalloc#start=0, ObMalloc#end=1983697, ObMalloc#start=20, ObMalloc#end=48, FREE_END=429], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:30.013825] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4708000, owner: ObLog, click_count: 8, time dist:[FORMAT_END=2731582, ObMalloc#start=1, ObMalloc#end=1975505, ObMalloc#start=17, ObMalloc#end=49, ObMalloc#start=383, ObMalloc#end=41, FREE_END=422], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:29.840948] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4880648, owner: ObLog, click_count: 8, time dist:[FORMAT_END=2907240, ObMalloc#start=1, ObMalloc#end=1972553, ObMalloc#start=19, ObMalloc#end=48, ObMalloc#start=237, ObMalloc#end=49, FREE_END=501], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:32.729259] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1996732, owner: ObLog, click_count: 6, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=1995391, ObMalloc#start=35, ObMalloc#end=63, FREE_END=1232], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [392]OB_MALLOC COST TOO MUCH TIME, cost_time=2010326, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=66, WASH_BLOCK_END=2010260], seq:[0, 1, 2] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=1974132, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=45, WASH_BLOCK_END=1974087], seq:[0, 1, 2] [389]OB_MALLOC COST TOO MUCH TIME, cost_time=1985714, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=33, WASH_BLOCK_END=1985681], seq:[0, 1, 2] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=1980152, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=41, WASH_BLOCK_END=1980111], seq:[0, 1, 2] [2025-08-14 09:04:32.741547] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1985932, owner: ObLog, click_count: 6, time dist:[FORMAT_END=5, ObMalloc#start=0, ObMalloc#end=1985728, ObMalloc#start=45, ObMalloc#end=92, FREE_END=62], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [472]OB_MALLOC COST TOO MUCH TIME, cost_time=1969402, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=49, WASH_BLOCK_END=1969352], seq:[0, 1, 2] [2025-08-14 09:04:32.747300] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [477][T1_StandbySchem][T1][YB42AC4D0007-00063C495B6A0381-0-0] [lt=1955996][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1980424, owner: ObLog, click_count: 6, time dist:[FORMAT_END=15, ObMalloc#start=1, ObMalloc#end=1980163, ObMalloc#start=22, ObMalloc#end=65, FREE_END=158], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:32.716260] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2011469, owner: ObLog, click_count: 6, time dist:[FORMAT_END=15, ObMalloc#start=1, ObMalloc#end=2010346, ObMalloc#start=27, ObMalloc#end=60, FREE_END=1020], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:29.832780] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4890880, owner: ObLog, click_count: 8, time dist:[FORMAT_END=2921031, ObMalloc#start=12, ObMalloc#end=1969413, ObMalloc#start=21, ObMalloc#end=50, ObMalloc#start=211, ObMalloc#end=59, FREE_END=83], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:30.749374] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:288) [884][T1002_ArcFetche][T1002][YB42AC4D0007-00063C495ED99C16-0-0] [lt=19405][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3988806, owner: ObLog, click_count: 6, time dist:[FORMAT_END=3985707, ObMalloc#start=1, ObMalloc#end=41, ObMalloc#start=26, ObMalloc#end=41, FREE_END=2990], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113b2889 0x979b621 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:33.742976] EDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1990347][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1000030, owner: ObLog, click_count: 8, time dist:[FORMAT_END=996507, ObMalloc#start=0, ObMalloc#end=43, ObMalloc#start=21, ObMalloc#end=41, ObMalloc#start=3057, ObMalloc#end=49, FREE_END=311], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ed35c 0x97aead0 0x97ad969 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:33.742960] EDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1992548][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1000241, owner: ObLog, click_count: 8, time dist:[FORMAT_END=998647, ObMalloc#start=2, ObMalloc#end=63, ObMalloc#start=23, ObMalloc#end=56, ObMalloc#start=1206, ObMalloc#end=42, FREE_END=201], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ed35c 0x97aead0 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [386]OB_MALLOC COST TOO MUCH TIME, cost_time=5940336, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=3977448], seq:[0, 1] [2025-08-14 09:04:29.770469] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5940377, owner: ObLog, click_count: 5, time dist:[FORMAT_END=7, ObMalloc#start=0, ObMalloc#end=5940366, ALLOC_END=2, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:32.768322] EDIAG [RS.COMPACTION] try_idle (ob_major_merge_scheduler.cpp:226) [1275][T1001_MergeSche][T1001][YB42AC4D0007-00063C4963E99CBE-0-0] [lt=1030700][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2947432, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2947431, ALLOC_END=1, APPEND_END=0, ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=50], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1125d502 0x1125cf81 0x1125a4c3 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:27.751610] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=11981740][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5978312, owner: ObLog, click_count: 6, time dist:[FORMAT_END=4990633, ObMalloc#start=0, ObMalloc#end=987508, ObMalloc#start=17, ObMalloc#end=41, FREE_END=113], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:33.745323] EDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=937480][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1979001, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1978999, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ed35c 0x97aead0 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:27.751772] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=3957543][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5978203, owner: ObLog, click_count: 6, time dist:[FORMAT_END=4993762, ObMalloc#start=1, ObMalloc#end=984310, ObMalloc#start=19, ObMalloc#end=41, FREE_END=70], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [1057]OB_MALLOC COST TOO MUCH TIME, cost_time=989140, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 12, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=989090, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=989259, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 12, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=989205, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=989252, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 12, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=989205, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=987010, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 14, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=986949, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13] [833]OB_MALLOC COST TOO MUCH TIME, cost_time=986535, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 14, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=986465, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13] [279]OB_MALLOC COST TOO MUCH TIME, cost_time=986566, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=32896, owner: ALLOC_BLOCK, click_count: 14, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=986490, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13] [387]OB_MALLOC COST TOO MUCH TIME, cost_time=991663, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 14, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=991599, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=985177, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 14, time dist:[ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=985105, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=3022274, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=3022267], seq:[0] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=3022299, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=3022295], seq:[0, 1] [2025-08-14 09:04:28.730214] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=16][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1004421, owner: ObLog, click_count: 8, time dist:[FORMAT_END=10, ObMalloc#start=0, ObMalloc#end=1003784, ObMalloc#start=2, ObMalloc#end=83, FREE_END=542, ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=2], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:30.770296] EDIAG [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=49593][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5009898, owner: ObLog, click_count: 6, time dist:[FORMAT_END=5007773, ObMalloc#start=1, ObMalloc#end=59, ObMalloc#start=24, ObMalloc#end=51, FREE_END=1990], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1fce4e91 0x1fce455d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:31.778053] EDIAG [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [726][T1001_ObXAHbWor][T1001][Y0-0000000000000000-0-0] [lt=3019608][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4005165, owner: ObLog, click_count: 3, time dist:[FORMAT_END=4005164, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b64d8ba 0x1b64d154 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:28.780083] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 6999068, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=2, ALLOC_BLOCK#end=954468, FORMAT_END=6044594, ObMalloc#start=0, ObMalloc#end=3, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:27.798031] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=20][errcode=0] LOGGER COST TOO MUCH TIME, cost: 7979707, owner: ObLog, click_count: 3, time dist:[FORMAT_END=7979707, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:27.752014] EDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:93) [964][T1002_L0_G0][T1002][YB42AC4D0007-00063C4962EA9002-0-0] [lt=30][errcode=-4273] LOGGER COST TOO MUCH TIME, cost: 8025859, owner: ObLog, click_count: 10, time dist:[ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=4, FORMAT_END=8022695, ObMalloc#start=1, ObMalloc#end=60, ObMalloc#start=21, ObMalloc#end=53, ObMalloc#start=2847, ObMalloc#end=70, FREE_END=104], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], BACKTRACE: 0x26a1d7e0 0x26a180ba 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf998a7e 0x96f95a1 0x96f8efa 0x9312121 0x930ecab 0x930dc16 0x930b653 0x92d4d44 0x92d1397 0x9292579 0x9285f13 0x927efad 0x926c154 0x12c5a0c9 0x2668bbb2 [968]OB_MALLOC COST TOO MUCH TIME, cost_time=1016774, tenant_id=1002, label=HeapVar, ctx_id=0, prio=0, size=48, [2025-08-14 09:04:31.746401] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1990150, owner: ObLog, click_count: 6, time dist:[FORMAT_END=999788, ObMalloc#start=0, ObMalloc#end=989266, ObMalloc#start=27, ObMalloc#end=90, FREE_END=979], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:27.753937] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=3970876][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5975992, owner: ObLog, click_count: 6, time dist:[FORMAT_END=4988259, ObMalloc#start=0, ObMalloc#end=987474, ObMalloc#start=20, ObMalloc#end=42, FREE_END=197], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:32.758474] EDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] LOGGER COST TOO MUCH TIME, cost: 3022334, owner: ObLog, click_count: 5, time dist:[FORMAT_END=23, ObMalloc#start=1, ObMalloc#end=3022307, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x13b06ffb 0x958a39e 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:29.770469] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:35.719864] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5940377][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:35.719886] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:34.745695] EDIAG [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=982874][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2010507, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2010506, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x18bdb084 0x18bda6b1 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [384]OB_MALLOC COST TOO MUCH TIME, cost_time=1933822, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 9, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=18, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=1933755], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=1922636, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1922589, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=1922685, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1922661, WASH_BLOCK_END=24], seq:[0, 1, 2] [339]OB_MALLOC COST TOO MUCH TIME, cost_time=1000032, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=999994, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [339]OB_MALLOC COST TOO MUCH TIME, cost_time=1000078, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1000055, WASH_BLOCK_END=23], seq:[0, 1, 2] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=997045, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=21, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=996986, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [383]OB_MALLOC COST TOO MUCH TIME, cost_time=989931, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=989890, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [383]OB_MALLOC COST TOO MUCH TIME, cost_time=989982, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=989957, WASH_BLOCK_END=24], seq:[0, 1, 2] [400]OB_MALLOC COST TOO MUCH TIME, cost_time=978932, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=978896, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [400]OB_MALLOC COST TOO MUCH TIME, cost_time=978951, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=978942, WASH_BLOCK_END=9], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=979005, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=978972, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=979023, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=979014, WASH_BLOCK_END=9], seq:[0, 1, 2] [2025-08-14 09:04:35.805530] EDIAG eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=208][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1923730, owner: ObLog, click_count: 6, time dist:[FORMAT_END=14, ObMalloc#start=1, ObMalloc#end=1922696, ObMalloc#start=6, ObMalloc#end=49, FREE_END=963], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x27a8a8c8 0x97b7e1a 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [660]OB_MALLOC COST TOO MUCH TIME, cost_time=973408, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=973379, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=973429, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=973420, WASH_BLOCK_END=9], seq:[0, 1, 2] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=973470, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=973438, ALLOC_CHUNK_END=11], seq:[0, 1, 2, 3, 4, 5] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=973488, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=973479, WASH_BLOCK_END=9], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=971605, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=22, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=971557, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=971625, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=971617, WASH_BLOCK_END=8], seq:[0, 1, 2] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=966727, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=966697, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=966745, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=966737, WASH_BLOCK_END=8], seq:[0, 1, 2] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=963112, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=963083, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=963135, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=963127, WASH_BLOCK_END=8], seq:[0, 1, 2] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=950855, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=950830, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=950874, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=950865, WASH_BLOCK_END=8], seq:[0, 1, 2] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=957893, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=957869, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=957911, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=957903, WASH_BLOCK_END=8], seq:[0, 1, 2] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=997070, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=997058, WASH_BLOCK_END=12], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=972124, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=972093, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=972143, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=972134, WASH_BLOCK_END=9], seq:[0, 1, 2] [2025-08-14 09:04:31.844304] EDIAG try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=1859817][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5879853, owner: ObLog, click_count: 8, time dist:[ALLOC_BLOCK#start=6, ALLOC_BLOCK#end=3, FORMAT_END=4899916, ObMalloc#start=1, ObMalloc#end=979030, ObMalloc#start=28, ObMalloc#end=43, FREE_END=826], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x26499505 0x97074a3 0x26498fa9 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:35.789477] EDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1000030][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 1940800, owner: ObLog, click_count: 8, time dist:[FORMAT_END=976968, ObMalloc#start=1, ObMalloc#end=963141, ObMalloc#start=26, ObMalloc#end=38, ObMalloc#start=124, ObMalloc#end=43, FREE_END=459], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1b53d414 0x97ad9aa 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:32.210061] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=692110][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5514891, owner: ObLog, click_count: 8, time dist:[FORMAT_END=4534665, ObMalloc#start=2, ObMalloc#end=978958, ObMalloc#start=28, ObMalloc#end=42, ObMalloc#start=900, ObMalloc#end=47, FREE_END=249], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:35.789466] EDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=16248][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 1942636, owner: ObLog, click_count: 8, time dist:[FORMAT_END=966474, ObMalloc#start=0, ObMalloc#end=973494, ObMalloc#start=19, ObMalloc#end=41, ObMalloc#start=544, ObMalloc#end=234, FREE_END=1830], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1b53d414 0x97ad9aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:35.743594] EDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=16847][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 1988506, owner: ObLog, click_count: 8, time dist:[FORMAT_END=1014163, ObMalloc#start=0, ObMalloc#end=971631, ObMalloc#start=27, ObMalloc#end=44, ObMalloc#start=549, ObMalloc#end=51, FREE_END=2041], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1b53d414 0x97ad9aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:35.789473] EDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1000241][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 1942873, owner: ObLog, click_count: 8, time dist:[FORMAT_END=966438, ObMalloc#start=1, ObMalloc#end=973436, ObMalloc#start=18, ObMalloc#end=610, ObMalloc#start=155, ObMalloc#end=47, FREE_END=2168], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1b53d414 0x97ad9aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:35.772561] EDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=42191][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 1960014, owner: ObLog, click_count: 8, time dist:[FORMAT_END=985132, ObMalloc#start=1, ObMalloc#end=972151, ObMalloc#start=26, ObMalloc#end=48, ObMalloc#start=242, ObMalloc#end=61, FREE_END=2353], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1b53d414 0x97ad9aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:30.780177] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [339][EventTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 6944979, owner: ObLog, click_count: 8, time dist:[FORMAT_END=5940885, ObMalloc#start=1, ObMalloc#end=1000086, ObMalloc#start=26, ObMalloc#end=38, ObMalloc#start=385, ObMalloc#end=61, FREE_END=3497], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:33.762655] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=640][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3967668, owner: ObLog, click_count: 3, time dist:[FORMAT_END=3967667, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:34.612449] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=64785][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3120302, owner: ObLog, click_count: 8, time dist:[FORMAT_END=3119881, ObMalloc#start=0, ObMalloc#end=85, ObMalloc#start=22, ObMalloc#end=56, ObMalloc#start=78, ObMalloc#end=68, FREE_END=112], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:31.809851] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5920980, owner: ObLog, click_count: 10, time dist:[ALLOC_BLOCK#start=8, ALLOC_BLOCK#end=3, FORMAT_END=5920473, ObMalloc#start=1, ObMalloc#end=66, ObMalloc#start=22, ObMalloc#end=38, ObMalloc#start=128, ObMalloc#end=38, FREE_END=203], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:33.762663] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=208][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3966601, owner: ObLog, click_count: 6, time dist:[FORMAT_END=3962987, ObMalloc#start=1, ObMalloc#end=48, ObMalloc#start=21, ObMalloc#end=47, FREE_END=3497], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [964]OB_MALLOC COST TOO MUCH TIME, cost_time=985903, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=985901], seq:[0] [964]OB_MALLOC COST TOO MUCH TIME, cost_time=985924, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=985919], seq:[0, 1] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=1950182, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1933858], seq:[0, 1] [2025-08-14 09:04:32.747878] EDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1006758][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4989317, owner: ObLog, click_count: 5, time dist:[FORMAT_END=3039126, ObMalloc#start=1, ObMalloc#end=1950189, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:35.720175] EDIAG [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2021601, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2021600, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1f9d7fcf 0x99b8be2 0x1f981833 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:29.949353] EDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 7783727, owner: ObLog, click_count: 3, time dist:[FORMAT_END=7783726, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x9757ced 0x97569df 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:33.745171] EDIAG [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=659][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3994897, owner: ObLog, click_count: 3, time dist:[FORMAT_END=3994896, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x12377491 0x1236e8e9 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:35.719899] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:33.745323] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=937480][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:32.758474] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:31.778053] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [726][T1001_ObXAHbWor][T1001][Y0-0000000000000000-0-0] [lt=3019608] XA scheduler heartbeat task statistics(avg_time=1) [2025-08-14 09:04:28.780083] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:35.789760] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6999068][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:35.789777] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:35.789788] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:27.798031] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:04:35.789824] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=7979707] gc stale ls task succ [2025-08-14 09:04:35.789978] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=228] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:04:35.791546] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1990150][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:35.791566] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:35.791579] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:36.720899] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:36.741911] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=331] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:35.785290] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=1979001][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:34.745695] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=982874] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:04:35.795607] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=964466][errcode=-4389] timer task use too much time(end=1755162274752518, start=1755162269841141, delta=4911377) [2025-08-14 09:04:36.784419] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=152] gc stale ls task succ [2025-08-14 09:04:36.786744] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=3318] gc stale ls task succ [2025-08-14 09:04:36.789889] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=4] gc stale ls task succ [2025-08-14 09:04:36.790068] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=18] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:04:36.792485] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:36.793262] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:36.919985] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [402][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=697277][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:37.720259] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:33.762655] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=640] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:04:36.757281] INFO sync_wash_ (ob_tenant_ctx_allocator.cpp:442) [964][T1002_L0_G0][T1002][YB42AC4D0007-00063C4962EA9002-0-0] [lt=1587] [MEM][WASH] tenant_id: 1002, ctx_id: 16, washed_size: 4390912 [2025-08-14 09:04:32.747878] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1006758][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:36.767901] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=970816][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:37.744773] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162277744773, start=1755162276767899, delta=976874) [2025-08-14 09:04:37.744779] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4989317] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:37.744791] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:35.720175] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:04:35.771940] INFO [ARCHIVE] produce_log_fetch_task_ (ob_archive_sequencer.cpp:166) [731][T1001_ArcSeq][T1001][YB42AC4D0007-00063C495C999C16-0-0] [lt=50439] archive round not in doing status, just skip(key={incarnation:-1, dest_id:-1, round:-1}, state={status:"INVALID"}) [2025-08-14 09:04:29.949353] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:33.745171] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=659] finish to resize log external storage handler(current_ts=1755162270780176, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [964]OB_MALLOC COST TOO MUCH TIME, cost_time=29963548, tenant_id=1002, label=SqlExecutor, ctx_id=0, prio=0, size=245760, owner: ObMalloc, click_count: 21, time dist:[ObMalloc#start=21104, ObMalloc#end=8, ObLog#start=1976143, ObLog#end=10014181, ObLog#start=3995599, ObLog#end=989254, ObLog#start=991225, ObLog#end=13470, ObLog#end=968233, ObLog#start=1013474, ObLog#end=31, ObLog#end=8025877, ObLog#end=1588, ObLog#start=964195, ObLog#end=985946, ALLOC_CHUNK_END=3190], seq:[1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 14, 17, 18, 19, 20] [2025-08-14 09:04:28.770030] EDIAG [RS] get_valid_sts_after (ob_tenant_info_loader.cpp:454) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=997882][errcode=0] LOGGER COST TOO MUCH TIME, cost: 8965565, owner: ObLog, click_count: 5, time dist:[FORMAT_END=8965556, ObMalloc#start=0, ObMalloc#end=4, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1001d942 0x996d843 0x996d15a 0x996cfd8 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:37.748581] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=3022334][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:37.740498] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:37.750685] WDIAG ev_invoke_pending (ev.c:2136) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10169][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:37.751269] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][YB42AC4D0007-00063C495F8A1956-0-0] [lt=950][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:36.569422] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=56][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1181468, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1181453, ObMalloc#start=2, ObMalloc#end=7, ALLOC_END=0, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:36.569422] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=56][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:34.773009] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2975847, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:34.773009] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:37.753391] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2975847][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:37.753442] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:37.753456] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:37.753649] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=2023336] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:36.754028] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=970096][errcode=-4015] clock out of order(cur_ts=1755162275780959, cur_ts_=1755162275757840, delta=23119) [2025-08-14 09:04:37.758511] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=2030292] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:30.729366] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1027837, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:37.758607] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1027837][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:37.758630] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:37.758645] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:29.991872] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3741742, , BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:37.758692] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=3741742][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:37.758953] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=3994897] snapshot_traffic_info once [2025-08-14 09:04:34.739784] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1832][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3015135, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:34.739784] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1832] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:37.759502] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3015135][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:37.759519] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:37.759532] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:30.132632] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=2][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4591591, , BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:32.741485] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1984177, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:37.759627] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1984177][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:37.759651] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:37.759666] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:33.761981] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [663][T1001_StandbySc][T1001][YB42AC4D0007-00063C495C7A036C-0-0] [lt=201][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3991725, , BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:31.746070] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1993866, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:37.754074] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=2029712] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:37.759868] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1993866][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:37.759901] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:37.759920] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:33.783610] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=57009][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:33.783610] EDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=57009][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3970340, , BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:37.759982] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=3970340] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:37.760007] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:33.874432] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=605240][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:33.874432] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=605240][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3879748, , BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:31.741820] EDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:76) [829][T1002_LogRessvr][T1002][YB42AC4D0007-00063C495D199C16-0-0] [lt=10615][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4034473, , BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x1135bce6 0x9a0eb0a 0x1134e1b4 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:33.762667] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=358][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3991797, , BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:34.730662] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1619][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1040333, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:33.762667] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=358] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:04:37.765380] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=737] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:37.772924] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2962] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:37.781302] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=605] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:37.784482] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=12] gc stale ls task succ [2025-08-14 09:04:37.786806] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=7] gc stale ls task succ [2025-08-14 09:04:37.789961] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=10] gc stale ls task succ [2025-08-14 09:04:37.790153] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=17] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:04:37.793203] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:37.835147] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:37.835623] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:37.840165] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:30.751077] EDIAG [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:233) [529][T1_RFLWorker][T1][YB42AC4D0007-00063C495B199C16-0-0] [lt=2018968][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3987447, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x11359281 0x98b739c 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [393]OB_MALLOC COST TOO MUCH TIME, cost_time=974898, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=50, WASH_BLOCK_END=74, ALLOC_BLOCK#start=974752, ALLOC_BLOCK#end=21], seq:[0, 1, 2, 3, 4] [466]OB_MALLOC COST TOO MUCH TIME, cost_time=876223, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=876221], seq:[0] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=890612, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=890611], seq:[0] [2025-08-14 09:04:32.749839] EDIAG [RS.COMPACTION] try_idle (ob_major_merge_scheduler.cpp:226) [1057][T1_MergeSchedul][T1][YB42AC4D0007-00063C4961599CC3-0-0] [lt=3014978][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5982207, , BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1125d502 0x1125cf81 0x1125a4c3 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [466]OB_MALLOC COST TOO MUCH TIME, cost_time=892626, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=876239], seq:[0, 1] [2025-08-14 09:04:37.859477] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=912199, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=890639], seq:[0, 1] [2025-08-14 09:04:38.758627] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:37.845246] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:38.758870] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162278758869, start=1755162277845244, delta=913625) [2025-08-14 09:04:38.761355] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:38.761382] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:38.761405] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:38.763263] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:38.763302] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:38.765358] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=25] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:38.766605] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:38.739609] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [854][T1002_ObXAHbWor][T1002][Y0-0000000000000000-0-0] [lt=2997580] XA scheduler heartbeat task statistics(avg_time=327034) [399]OB_MALLOC COST TOO MUCH TIME, cost_time=1029312, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 15, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1029244], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14] [2025-08-14 09:04:38.732330] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:38.770908] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=23] temporary file module use no blocks [2025-08-14 09:04:38.770930] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=18] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:04:37.844942] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=7783727][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:38.771691] INFO print_info (ob_kvcache_hazard_domain.cpp:138) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14970346] [KVCACHE-HAZARD-POINTER] retired memory size: 0, blocks size: 94, free list size: 1003, memory_efficiency: 0.897691 [2025-08-14 09:04:38.776260] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:35.761548] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=990822][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:35.761548] EDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=990822][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3014403, , BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:38.780555] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3014403] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:38.780577] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:38.781288] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:37.791252] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:38.782454] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=989703][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [400]OB_MALLOC COST TOO MUCH TIME, cost_time=847737, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=31, WASH_BLOCK_END=847705, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1, 2, 3, 4] [2025-08-14 09:04:37.933448] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=5514891][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:38.721268] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:38.782560] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=61199][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:38.782613] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:38.771858] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=5780] temporary file module use no blocks [2025-08-14 09:04:38.784313] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12434] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [858]OB_MALLOC COST TOO MUCH TIME, cost_time=968504, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=968502], seq:[0] [2025-08-14 09:04:38.771707] EDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:1131) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1002233, owner: ObLog, click_count: 8, time dist:[FORMAT_END=968573, ObMalloc#start=1, ObMalloc#end=56, ObMalloc#start=33326, ObMalloc#end=75, ObMalloc#start=68, ObMalloc#end=59, FREE_END=75], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1d7e0 0x26a180ba 0x99daaa6 0x99da47c 0xf9845d6 0x1fc93059 0x1fc8ea63 0x1fc7a380 0x96f94a6 0x96f8efa 0x97c3dbc 0x93131eb 0x9329467 0x932778f 0x9315ac5 0x27aa2827 0x27a872f9 0x1c0470a6 0x1c0340a3 0x1c02b713 0x97a97cd 0x97a1ba3 0x97a1536 0x2668daa5 0x2668bbb2 [2025-08-14 09:04:38.967494] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=1110][errcode=-4018] LOGGER COST TOO MUCH TIME, cost: 1750869, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1750867, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3b4b 0x113d34aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:38.970342] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=283][errcode=-4018] LOGGER COST TOO MUCH TIME, cost: 1755489, owner: ObLog, click_count: 8, time dist:[FORMAT_END=1747915, ObMalloc#start=1, ObMalloc#end=40, ObMalloc#start=21, ObMalloc#end=37, ObMalloc#start=3465, ObMalloc#end=68, FREE_END=3941], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3b4b 0x113d34aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [832]OB_MALLOC COST TOO MUCH TIME, cost_time=1953601, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1953594], seq:[0] [832]OB_MALLOC COST TOO MUCH TIME, cost_time=1953621, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1953616], seq:[0, 1] [2025-08-14 09:04:38.771944] EDIAG [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1966052, owner: ObLog, click_count: 7, time dist:[FORMAT_END=5, ObMalloc#start=0, ObMalloc#end=12393, ObMalloc#start=24, ObMalloc#end=1953628, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x1d0f5b0a 0x1d0f57c6 0x1d17ad7c 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:39.781257] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:38.967494] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=1110][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:04:40.721058] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=5879853] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:04:38.771944] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28] temporary file module use no blocks [2025-08-14 09:04:40.740996] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=1966052] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:04:40.741011] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:04:40.742425] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=221] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:40.743143] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:845) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=361] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_to_wash=6291456) [2025-08-14 09:04:40.743173] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:66) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-08-14 09:04:40.743180] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:93) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-08-14 09:04:40.743203] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] can not find enough memory block to wash(ret=-4273, size_washed=0, size_to_wash=6291456) [2025-08-14 09:04:40.743209] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:845) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_to_wash=6291456) [2025-08-14 09:04:40.743215] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:66) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-08-14 09:04:40.743220] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:93) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [567]OB_MALLOC COST TOO MUCH TIME, cost_time=2000747, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16448, owner: ALLOC_BLOCK, click_count: 20, time dist:[ALLOC_CHUNK_END=6, ObMalloc#start=32119, ObMalloc#end=8, ObLog#start=56, ObLog#end=13, ObLog#end=1002269, ObLog#start=965817, ObLog#end=361, ObLog#start=12, ObLog#end=26, ObLog#end=7, ObLog#end=6, ALLOC_CHUNK_END=6, ObLog#start=11, ObLog#end=6, ObLog#end=6], seq:[0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 12, 14, 16, 17, 18, 19] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=2000779, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=2000776], seq:[0, 1] [2025-08-14 09:04:38.784342] EDIAG [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1957306, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1957297, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0xfd250b6 0xfd247d3 0xfd244ea 0xfd242a2 0x1d19629b 0x1d17baa3 0x1d17ad94 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:38.770959] EDIAG [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1970840, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1970833, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0xfd250b6 0xfd247d3 0xfd244ea 0xfd242a2 0x1d19629b 0x1d17baa3 0x1d17ad94 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:40.739988] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [339][EventTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:38.784342] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=29] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:04:38.770959] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=25] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:04:40.745080] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=223] PNIO [ratelimit] time: 1755162280745079, bytes: 0, bw: 0.000000 MB/s, add_ts: 1004914, add_bytes: 0 [2025-08-14 09:04:40.745863] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=379] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [858]OB_MALLOC COST TOO MUCH TIME, cost_time=1964155, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=968522], seq:[0, 1] [2025-08-14 09:04:38.784553] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=23] gc stale ls task succ [2025-08-14 09:04:38.784553] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=23][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1964188, owner: ObLog, click_count: 5, time dist:[FORMAT_END=9, ObMalloc#start=0, ObMalloc#end=1964170, ALLOC_END=1, APPEND_END=8], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:40.751735] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=1964188] gc stale ls task succ [2025-08-14 09:04:38.741052] EDIAG [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2008160, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2008152, ObMalloc#start=0, ObMalloc#end=2, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x260b4cc1 0x260b9960 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:38.741052] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=15] log compressor cycles once. (ret=0, cost_time=5016244, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25879846912, fast_delete_log_mode=false) [2025-08-14 09:04:35.740581] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [412][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=3942249] [sql nio session](*s={this:0x7f98f733a030, session_id:3221590255, trace_id:YB42AC4D0007-00063C49651A3232-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:257}, err:0, last_decode_time:1755161952736711, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1716, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:04:40.752889] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [408][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=933322] [sql nio session](*s={this:0x7f98f72d9030, session_id:3221590656, trace_id:YB42AC4D0007-00063C4962EA9002-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:240}, err:0, last_decode_time:1755162243738749, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1390, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:04:40.756245] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=9468] current monitor number(seq_=-1) [2025-08-14 09:04:40.760533] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=375] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:40.760569] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=314] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:40.761168] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2100] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:40.761206] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1970] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:40.761704] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=1156] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:38.736840] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=3966601] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:04:38.736840] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=3966601][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2022105, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=6, ALLOC_BLOCK#end=47353, FORMAT_END=1974711, ObMalloc#start=1, ObMalloc#end=29, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:35.740581] EDIAG [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [412][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=3942249][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5004310, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a5bbf2 0x94da542 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:40.770887] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=2008160] log compressor cycles once. (ret=0, cost_time=19051, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25880436736, fast_delete_log_mode=false) [2025-08-14 09:04:40.760256] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4452] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:40.771897] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11621][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:40.774297] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=173] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:40.774317] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=196] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:25.760249] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:254) [14862][T1001_TimerWK5_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=3972104] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1755136829134139001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1755136829134139001, v:0}}, }, *this={this:0x7f99297d7150, is_inited:true, freezing_scn:{val:1755136801284520000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136829134139001, v:0}, clog_base_lsn:{lsn:8186781696}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1753840867372367034, v:0}, all_id_meta:{id_meta:[{limited_id:1755161729775179000, latest_log_ts:{val:1755161719775179001, v:0}}, {limited_id:18000001, latest_log_ts:{val:1755157440846556001, v:0}}, {limited_id:60000001, latest_log_ts:{val:1755135657266496001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:46, palf_env_:0x7f99321fa030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-08-14 09:04:40.780446] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=231] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:40.781886] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:40.782352] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:40.787036] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=224] gc stale ls task succ [2025-08-14 09:04:40.790181] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=202] gc stale ls task succ [2025-08-14 09:04:40.790741] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=196] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:04:40.791407] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7139] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:40.793278] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=273] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:40.821189] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=17] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:04:40.846779] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:40.851470] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:40.851519] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:40.860403] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:40.969891] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:41.017332] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:41.027486] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:41.721606] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=316] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:40.745303] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:233) [709][T1001_RFLWorker][T1001][YB42AC4D0007-00063C495BE99C16-0-0] [lt=8988] ObRemoteFetchWorker is running(thread_index=0) [2025-08-14 09:04:40.969865] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=331][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:41.736550] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=246] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:41.739077] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2146] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:41.739885] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1657] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:41.740375] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:25.760249] EDIAG [MDS] get_rec_scn (mds_table_mgr.cpp:254) [14862][T1001_TimerWK5_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=3972104][errcode=0] LOGGER COST TOO MUCH TIME, cost: 14992337, owner: ObLog, click_count: 5, time dist:[FORMAT_END=14992324, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=9], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1dc07aa1 0x9a7e8fd 0x1dc07469 0x1bf31c14 0x9aaab88 0x1c3fa49f 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:38.722298] EDIAG [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=8965565][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3015543, owner: ObLog, click_count: 5, time dist:[FORMAT_END=3015532, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1b52e1b7 0x996d441 0x996cfd8 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:38.722298] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=8965565] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:04:41.743677] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=25] finish to resize log external storage handler(current_ts=1755162281743673, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-08-14 09:04:41.743701] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=23] snapshot_traffic_info once [2025-08-14 09:04:40.742444] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:288) [860][T1002_ArcFetche][T1002][YB42AC4D0007-00063C495E399C16-0-0] [lt=1820] ObArchiveFetcher is running(thread_index=0) [2025-08-14 09:04:41.767349] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=25] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:40.751753] EDIAG [RS] skip_check_member_list_change_ (ob_ls_recovery_stat_handler.cpp:109) [777][T1002_TeRec][T1002][YB42AC4D0007-00063C495DDA006D-0-0] [lt=4945058][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1014317, owner: ObLog, click_count: 5, time dist:[FORMAT_END=979287, ObMalloc#start=1, ObMalloc#end=35024, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x1067fb2c 0x1067ecc2 0x1067e03d 0x1002f2a3 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:39.743590] EDIAG [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [807][T1002_TimerWK3_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=36713][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2022324, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1986216, ObMalloc#start=1, ObMalloc#end=36102, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x9870878 0x1c3fed66 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:41.251304] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:41.768244] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:41.768272] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:39.743590] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [807][T1002_TimerWK3_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=36713] ====== check clog disk timer task ====== [2025-08-14 09:04:41.742425] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=3015543] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:04:41.749320] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=14] PNIO [ratelimit] time: 1755162281749318, bytes: 0, bw: 0.000000 MB/s, add_ts: 1004239, add_bytes: 0 [2025-08-14 09:04:41.753057] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=10] gc stale ls task succ [2025-08-14 09:04:41.761823] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:41.761905] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=21] current monitor number(seq_=-1) [2025-08-14 09:04:41.777358] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:41.780472] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:41.782330] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:41.783370] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] [sql nio session](*s={this:0x7f98f72d0030, session_id:3221589415, trace_id:YB42AC4D0007-00063C4965FA43DC-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:245}, err:0, last_decode_time:1755161751714366, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1662, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:04:41.783401] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=31] [sql nio session](*s={this:0x7f98f72d5030, session_id:3221590200, trace_id:YB42AC4D0007-00063C4965FA43E0-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:248}, err:0, last_decode_time:1755161776777167, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1593, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:04:41.791259] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=15] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:04:41.791651] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:41.794266] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:41.801452] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:41.863095] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:41.863221] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:41.971591] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:41.985414] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=847776][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:42.093609] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:42.247709] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:42.274962] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:41.745638] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2021601] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:04:42.720910] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=973757] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:04:42.720966] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:04:42.721003] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:04:42.721500] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=32] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:42.724670] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28321][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:42.732898] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=216] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [396]OB_MALLOC COST TOO MUCH TIME, cost_time=963314, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=963313], seq:[0] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=963538, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=80, WASH_BLOCK_END=86, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=963368], seq:[0, 1, 2, 3, 4] [2025-08-14 09:04:41.253389] EDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] LOGGER COST TOO MUCH TIME, cost: 1479226, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=6, ALLOC_BLOCK#end=4, FORMAT_END=1479212, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x13b06ffb 0x958a39e 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:41.768040] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:42.733066] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=963567][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:42.734160] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [807][T1002_TimerWK3_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=2022324] get_disk_usage(ret=0, capacity(MB):=24883, used(MB):=8642) [2025-08-14 09:04:42.734199] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:41.747125] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:42.734776] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=986159][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:41.253389] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:42.734941] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=1479226][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:41.823680] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=19] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:04:42.746277] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:42.747586] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:42.747634] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:42.747666] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:40.744800] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=1970840] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623172175563, flush_io_finished_ret_:0, flush_io_finished_round_:26356, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:40.744800] EDIAG [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=1970840][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2003170, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2003156, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=1, APPEND_END=8], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1d17bb52 0x1d17ada0 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:40.744653] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=1957306] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623172175160, flush_io_finished_ret_:0, flush_io_finished_round_:26367, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:40.744653] EDIAG [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=1957306][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2003390, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2003379, ObMalloc#start=0, ObMalloc#end=3, ALLOC_END=0, APPEND_END=8], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1d17bb52 0x1d17ada0 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:40.741061] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=50] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623172176427, flush_io_finished_ret_:0, flush_io_finished_round_:26335, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:40.741061] EDIAG [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=50][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2007052, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2007041, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=8], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1d17bb52 0x1d17ada0 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:42.753129] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=17953] gc stale ls task succ [2025-08-14 09:04:42.755252] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=0] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:42.756850] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=21642] PNIO [ratelimit] time: 1755162282756848, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007530, add_bytes: 0 [2025-08-14 09:04:42.759130] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:26.747074] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=2990105] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:8607757, max_lsn:1604090331, max_scn:{val:1755161756730847001, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:1556737504}, reuse_lsn:{lsn:1604090331}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:1562147291}}, last_submit_lsn:{lsn:1604090203}, last_submit_end_lsn:{lsn:1604090331}, last_submit_log_id:8607757, last_submit_log_pid:50, max_flushed_lsn:{lsn:1604090203}, max_flushed_end_lsn:{lsn:1604090331}, max_flushed_log_pid:50, committed_end_lsn:{lsn:1604090090}, last_slide_log_id:8607755, last_slide_scn:{val:1755161749741193000, v:0}, last_slide_lsn:{lsn:1604089968}, last_slide_end_lsn:{lsn:1604090090}, last_slide_log_pid:50, last_slide_log_accum_checksum:1446576759, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f990bfe4530}, [StateMgr]={this:0x7f990bfe89b0, self:"172.77.0.7:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"PENDING", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:51}, leader:"0.0.0.0:0", leader_epoch:-1, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:1604090331}, scan_disk_log_finished:true, last_check_start_id:8607756, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161901828039, palf_role_change_cb:0x7f990bfef470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:50, prev:{version:1, config:{config_version:{proposal_id:49, config_seq:45}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:50, config_seq:46}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_mode_pid:50}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_end_lsn:{lsn:1604088821}, prev_mode_pid:50}, reconfig_barrier:{prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_end_lsn:{lsn:1604088821}, prev_mode_pid:50}, persistent_config_version:{proposal_id:50, config_seq:46}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755161738854289, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755161738854289, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f990bfe5c70}, [ModeMgr]={palf_id:1, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:50, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:1543409664}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:51}, log_config_meta:{version:3, proposal_id:50, prev:{version:1, config:{config_version:{proposal_id:49, config_seq:45}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:50, config_seq:46}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_mode_pid:50}, log_snapshot_meta:{version:2, base_lsn:{lsn:1543409664}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:712704}, readable_log_tail:{lsn:712704}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:1610514432}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/meta", dir_fd:225, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66392064, block_header_serialize_buf_:0x7f990bfeabb8, flashback_version:0}, log_storage:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1755161901828095, this:0x7f990bfe90f0}, [Reconfirm]={palf_id:1, self:"172.77.0.7:2882", state:"RECONFIRM_MODE_META", new_proposal_id:51, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"172.77.0.7:2882", majority_max_accept_pid:50, majority_max_lsn:{lsn:1604090331}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:1755161901828373, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f990bfe8b30}) [2025-08-14 09:04:42.761275] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:42.761345] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:42.741804] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [14862][T1001_TimerWK5_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=14992337][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:42.766054] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9567] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:42.766196] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=18] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:42.767263] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9432] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:42.772415] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=10379] current monitor number(seq_=-1) [2025-08-14 09:04:42.774254] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:42.780420] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:42.782410] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:42.791319] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=19] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:04:42.793306] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:42.802023] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:42.835507] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=910336] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:04:42.864060] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:42.866729] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:43.172284] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:43.214581] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:43.256736] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:43.305323] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:43.669207] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=409][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:26.747074] EDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=2990105][errcode=0] LOGGER COST TOO MUCH TIME, cost: 16948644, owner: ObLog, click_count: 5, time dist:[FORMAT_END=15988491, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=8, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0xfc0e73d 0x993f560 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:43.721513] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:42.790336] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=267] gc stale ls task succ [2025-08-14 09:04:43.724115] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [807][T1002_TimerWK3_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=0] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):258, sequence:1}) [2025-08-14 09:04:43.724161] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:97) [807][T1002_TimerWK3_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=1] BlockGCTimerTask success(ret=0, cost_time_us=975412, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.77.0.7:2882", log_dir:"/root/ob/observer/store/clog/tenant_1002", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):258, sequence:1}, log_alloc_mgr_:{flying_log_task:1, flying_meta_task:2}}) [2025-08-14 09:04:42.787165] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=213] gc stale ls task succ [2025-08-14 09:04:42.791606] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:43.733840] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=940808][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:43.734017] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:43.740477] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:43.740516] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:42.756058] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=984848] [COMPACTION DAG STATUS] (start_time=1755162255745111, end_time=1755162282756056, dag_status={}) [2025-08-14 09:04:42.754458] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=5978203] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:04:42.754585] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=5978312] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:04:43.747172] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=970932] blacklist_loop exec finished(cost_time=18, is_enabled=true, send_cnt=0) [2025-08-14 09:04:43.747219] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:42.733510] EDIAG [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1013616, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1013607, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x1f99e657 0x1f981d5d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:42.733510] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:04:43.748695] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=1013616] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:04:43.740539] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:43.753205] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12645][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:43.754938] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=19] gc stale ls task succ [2025-08-14 09:04:43.755255] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:43.760537] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:43.761069] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:43.761226] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:43.761323] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=38] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:43.761362] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:43.761909] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=983901] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:04:42.751229] EDIAG [RS] get_valid_sts_after (ob_tenant_info_loader.cpp:454) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1010856, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1010843, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=1, APPEND_END=8], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1001d942 0x996d843 0x996d15a 0x996cfd8 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:43.763646] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1010856] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:04:43.763676] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:43.763694] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:43.763708] WDIAG [RPC.OBRPC] in_black (ob_net_keepalive.cpp:257) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:43.769401] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=2007052] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:43.769447] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27] temporary file module use no blocks [2025-08-14 09:04:43.769461] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:04:43.769905] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=2003390] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:43.769919] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] temporary file module use no blocks [2025-08-14 09:04:43.769927] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:04:43.761940] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=30] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:04:43.775512] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=11] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:43.775709] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:43.776334] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=32] PNIO [ratelimit] time: 1755162283776331, bytes: 0, bw: 0.000000 MB/s, add_ts: 1019483, add_bytes: 0 [2025-08-14 09:04:43.780396] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:43.782409] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:43.787229] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=938559] gc stale ls task succ [2025-08-14 09:04:43.789944] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:43.791402] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=17] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:04:43.840577] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=19] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:04:43.863743] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:43.865063] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:43.963802] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:44.063863] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:44.163923] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:44.263948] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:44.364006] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:44.464064] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:44.564123] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:44.664196] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:43.772281] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10324] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:04:44.723506] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:43.737202] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=5975992] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:04:44.733311] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:43.769938] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:04:44.737772] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=966371] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623185190865, flush_io_finished_ret_:0, flush_io_finished_round_:26368, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:43.280156] EDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 1471665, owner: ObLog, click_count: 8, time dist:[FORMAT_END=1470408, ObMalloc#start=0, ObMalloc#end=800, ObMalloc#start=19, ObMalloc#end=46, ObMalloc#start=255, ObMalloc#end=46, FREE_END=91], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x9757ced 0x97569df 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [832]OB_MALLOC COST TOO MUCH TIME, cost_time=976659, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=976637], seq:[0] [832]OB_MALLOC COST TOO MUCH TIME, cost_time=976692, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=976687], seq:[0, 1] [2025-08-14 09:04:43.769476] EDIAG [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1943513, owner: ObLog, click_count: 5, time dist:[FORMAT_END=966809, ObMalloc#start=1, ObMalloc#end=976701, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0xfd250b6 0xfd247d3 0xfd244ea 0xfd242a2 0x1d19629b 0x1d17baa3 0x1d17ad94 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:44.723508] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=2003170] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:44.754409] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=30845] temporary file module use no blocks [2025-08-14 09:04:44.754420] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:04:44.754432] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:04:44.754444] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623185190801, flush_io_finished_ret_:0, flush_io_finished_round_:26357, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:44.761410] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:44.769309] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=22] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:44.780258] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=28] PNIO [ratelimit] time: 1755162284780256, bytes: 0, bw: 0.000000 MB/s, add_ts: 1003925, add_bytes: 0 [2025-08-14 09:04:43.769476] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:04:45.715954] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=1943513] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623185190970, flush_io_finished_ret_:0, flush_io_finished_round_:26336, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:43.793213] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1920384, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=3, FORMAT_END=1920371, ObMalloc#start=0, ObMalloc#end=2, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [393]OB_MALLOC COST TOO MUCH TIME, cost_time=976044, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=13405], seq:[0, 1] [2025-08-14 09:04:43.793213] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:45.716533] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1920384][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:45.716554] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:44.739473] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:45.717023] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=976073][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:44.723449] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=949729] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:04:45.717068] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=992115] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:04:45.717078] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:04:45.717050] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17351] current log_task status(palf_id=1, self="172.77.0.7:2882", log_id=8607756, log_task={header:{begin_lsn:{lsn:1604090090}, end_lsn:{lsn:1604090203}, log_id:8607756, min_scn:{val:1755161749741193001, v:0}, max_scn:{val:1755161749741193001, v:0}, data_len:57, proposal_id:50, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:1604090090}, data_checksum:2031283194, accum_checksum:759186326, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1755161750716165, freeze_ts:1755161750716173, submit_ts:1755161750716183, flushed_ts:1755161853765770, gen_to_freeze cost time:8, gen_to_submit cost time:18, submit_to_flush cost time:103049587}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161974728707, last_flushed_end_lsn:{lsn:1604090331}}]) [2025-08-14 09:04:43.854994] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:43.854994] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1859514, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1859513, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:43.790461] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=932217][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1924006, owner: ObLog, click_count: 8, time dist:[ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=1, FORMAT_END=1922913, ObMalloc#start=1, ObMalloc#end=96, ObMalloc#start=26, ObMalloc#end=51, FREE_END=913], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:43.986680] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=2][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1728113, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1728104, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:43.986680] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:45.717402] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=1924006] gc stale ls task succ [2025-08-14 09:04:45.721714] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:45.724427] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=985384] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:04:45.733104] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161101770923, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755161572801635) [2025-08-14 09:04:45.735035] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=72] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:44.738076] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=999189][errcode=-4015] clock out of order(cur_ts=1755162283772949, cur_ts_=1755162283723776, delta=49173) [2025-08-14 09:04:45.719734] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:147) [807][T1002_TimerWK3_][T1002][Y0-0000000000000000-0-0] [lt=0] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-08-14 09:04:45.739902] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:45.739934] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=668] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:45.740352] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=0] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:45.745325] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7166] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:45.717088] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:04:45.746848] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=29712] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:04:45.753626] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=8528] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:04:45.753893] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=10520] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:04:45.754259] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=485] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:45.755164] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=452] gc stale ls task succ [2025-08-14 09:04:45.746862] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:04:45.758925] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12040] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:04:45.762127] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=492] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:45.762164] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:45.763980] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1232] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:45.764017] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=420] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:45.764043] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=701] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:45.739500] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=40] get_max_applied_scn(scn={val:1755161547738516001, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:50, palf_committed_end_lsn_:{lsn:5282737655}, palf_committed_end_scn_:{val:1755161547738516001, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161547738516001, v:0}}) [2025-08-14 09:04:45.758973] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=43] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:04:45.764285] INFO [COMMON] loop_running_dag_net_list (ob_tenant_dag_scheduler.cpp:3845) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5298] loop running dag net list(ret=0, running_dag_net_list_size=0, slow_dag_net_cnt=0, finished_dag_net_list_size=0) [2025-08-14 09:04:45.768569] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=12] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:45.768650] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:45.768769] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:45.773126] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=23] current monitor number(seq_=-1) [2025-08-14 09:04:45.775365] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=539] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:45.775393] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:45.777766] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:45.780252] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=189] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:45.781252] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=168] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:45.787873] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=15] PNIO [ratelimit] time: 1755162285787871, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007615, add_bytes: 0 [2025-08-14 09:04:45.788249] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=218] gc stale ls task succ [2025-08-14 09:04:45.791066] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=203] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:45.791659] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:45.792251] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=176] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:04:45.842761] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=195] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:04:45.869516] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=301][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:45.969499] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=1471665][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:45.969612] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:46.042775] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=380][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:46.069771] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:46.169981] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:46.270170] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:46.370245] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:46.470320] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:46.570388] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:46.670450] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:46.684533] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=70][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:46.716050] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=34] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:46.716083] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27] temporary file module use no blocks [2025-08-14 09:04:46.716093] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:04:46.717465] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=10] gc stale ls task succ [2025-08-14 09:04:46.724343] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=30] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:46.734528] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:45.768755] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:45.772786] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:46.738823] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=47] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:46.738847] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=20] temporary file module use no blocks [2025-08-14 09:04:46.738856] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:04:46.738875] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=19] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:04:46.738889] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623188178560, flush_io_finished_ret_:0, flush_io_finished_round_:26370, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:46.739281] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:46.759569] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:04:46.759680] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:04:46.759984] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:46.760272] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:46.760855] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:46.760880] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:46.765564] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=19] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:46.768168] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=18] gc stale ls task succ [807]OB_MALLOC COST TOO MUCH TIME, cost_time=1011364, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1011363], seq:[0] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=961517, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=39, WASH_BLOCK_END=961478], seq:[0, 1, 2] [2025-08-14 09:01:00.748926] EDIAG [SQL.PC] cache_evict (ob_plan_cache.cpp:1507) [14888][T1002_TimerWK7_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=12170][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1008144, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1008138, ObMalloc#start=0, ObMalloc#end=2, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1535920e 0x153639eb 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [383]OB_MALLOC COST TOO MUCH TIME, cost_time=965074, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=965071], seq:[0] [800]OB_MALLOC COST TOO MUCH TIME, cost_time=976871, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=42, WASH_BLOCK_END=976828], seq:[0, 1, 2] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=976239, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=976199], seq:[0, 1, 2] [807]OB_MALLOC COST TOO MUCH TIME, cost_time=1980388, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=56, WASH_BLOCK_END=66, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1011387], seq:[0, 1, 2, 3, 4] [2025-08-14 09:04:45.738840] EDIAG [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [807][T1002_TimerWK3_][T1002][Y0-0000000000000000-0-0] [lt=19063][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2006551, owner: ObLog, click_count: 5, time dist:[FORMAT_END=26144, ObMalloc#start=1, ObMalloc#end=1980399, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1c450d3c 0x1c44d613 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:45.738840] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [807][T1002_TimerWK3_][T1002][Y0-0000000000000000-0-0] [lt=19063] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-08-14 09:04:47.748452] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [807][T1002_TimerWK3_][T1002][Y0-0000000000000000-0-0] [lt=2006551] [tabletchange] task check ls(ls->get_ls_id()={id:1001}, tablet_persist_trigger=0) [383]OB_MALLOC COST TOO MUCH TIME, cost_time=1941586, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=965098], seq:[0, 1] [2025-08-14 09:04:46.741280] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1980595, owner: ObLog, click_count: 7, time dist:[FORMAT_END=9, ObMalloc#start=0, ObMalloc#end=38934, ObMalloc#start=30, ObMalloc#end=1941618, ALLOC_END=3, APPEND_END=1], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [391]OB_MALLOC COST TOO MUCH TIME, cost_time=1995089, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1995080], seq:[0] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=1998236, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1998202, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=1998294, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1998270, WASH_BLOCK_END=23], seq:[0, 1, 2] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=1998344, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1998310, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=1998371, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1998360, WASH_BLOCK_END=11], seq:[0, 1, 2] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=1996933, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=1996905, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=1996961, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1996953, WASH_BLOCK_END=8], seq:[0, 1, 2] [369]OB_MALLOC COST TOO MUCH TIME, cost_time=1996924, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=1996901, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [369]OB_MALLOC COST TOO MUCH TIME, cost_time=1996946, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1996939, WASH_BLOCK_END=7], seq:[0, 1, 2] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=1996972, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=1996934, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=1996988, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1996981, WASH_BLOCK_END=7], seq:[0, 1, 2] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=1993800, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=1993773, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=1993815, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1993808, WASH_BLOCK_END=7], seq:[0, 1, 2] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=1989761, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1989726, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=1989775, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1989769, WASH_BLOCK_END=6], seq:[0, 1, 2] [2025-08-14 09:04:46.785489] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1989817, owner: ObLog, click_count: 5, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=1989781, ALLOC_END=26, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [776]OB_MALLOC COST TOO MUCH TIME, cost_time=1989835, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1989800, ALLOC_CHUNK_END=13], seq:[0, 1, 2, 3, 4, 5] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=1989852, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1989843, WASH_BLOCK_END=9], seq:[0, 1, 2] [2025-08-14 09:04:45.765176] EDIAG [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3008863, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1011363, ObMalloc#start=0, ObMalloc#end=1996994, ObMalloc#start=17, ObMalloc#end=42, FREE_END=447], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1f9d7fcf 0x99b8be2 0x1f981833 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:46.776356] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [477][T1_StandbySchem][T1][YB42AC4D0007-00063C495B6A038D-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1999278, owner: ObLog, click_count: 6, time dist:[FORMAT_END=8, ObMalloc#start=0, ObMalloc#end=1998378, ObMalloc#start=19, ObMalloc#end=38, FREE_END=835], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:46.781334] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1994336, owner: ObLog, click_count: 6, time dist:[FORMAT_END=10, ObMalloc#start=0, ObMalloc#end=1993821, ObMalloc#start=26, ObMalloc#end=45, FREE_END=434], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:46.785541] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [776][T1002_StandbySc][T1002][YB42AC4D0007-00063C495DFA037E-0-0] [lt=21][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1990269, owner: ObLog, click_count: 6, time dist:[FORMAT_END=16, ObMalloc#start=0, ObMalloc#end=1989858, ObMalloc#start=20, ObMalloc#end=37, FREE_END=338], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:46.776097] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [663][T1001_StandbySc][T1001][YB42AC4D0007-00063C495C7A0375-0-0] [lt=15][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1999985, owner: ObLog, click_count: 6, time dist:[FORMAT_END=13, ObMalloc#start=1, ObMalloc#end=1998304, ObMalloc#start=30, ObMalloc#end=46, FREE_END=1591], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [730]OB_MALLOC COST TOO MUCH TIME, cost_time=1986930, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1986898, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=1986984, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1986954, WASH_BLOCK_END=30], seq:[0, 1, 2] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=1984715, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=21, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1984673, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=1984734, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1984725, WASH_BLOCK_END=9], seq:[0, 1, 2] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=1984200, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=1984174, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=1984217, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1984209, WASH_BLOCK_END=8], seq:[0, 1, 2] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=1981495, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1981461, ALLOC_CHUNK_END=3], seq:[0, 1, 2, 3, 4, 5] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=1981515, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1981508, WASH_BLOCK_END=7], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=1981054, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1981022, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=1981073, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1981064, WASH_BLOCK_END=9], seq:[0, 1, 2] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=1954454, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=19, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=1954405, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=1954472, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1954464, WASH_BLOCK_END=8], seq:[0, 1, 2] [2025-08-14 09:04:46.795091] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1981716, owner: ObLog, click_count: 6, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=1981522, ObMalloc#start=26, ObMalloc#end=37, FREE_END=119], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [568]OB_MALLOC COST TOO MUCH TIME, cost_time=1060709, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=19, ALLOC_CHUNK_END=16, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1060655, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=1060728, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1060719, WASH_BLOCK_END=9], seq:[0, 1, 2] [836]OB_MALLOC COST TOO MUCH TIME, cost_time=1055835, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=15, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1055788, ALLOC_CHUNK_END=11], seq:[0, 1, 2, 3, 4, 5] [836]OB_MALLOC COST TOO MUCH TIME, cost_time=1055861, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1055845, WASH_BLOCK_END=16], seq:[0, 1, 2] [726]OB_MALLOC COST TOO MUCH TIME, cost_time=1055784, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1055754, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [726]OB_MALLOC COST TOO MUCH TIME, cost_time=1055801, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1055793, WASH_BLOCK_END=8], seq:[0, 1, 2] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=1055145, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1055126, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=1055162, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1055154, WASH_BLOCK_END=8], seq:[0, 1, 2] [339]OB_MALLOC COST TOO MUCH TIME, cost_time=1045552, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1045504, ALLOC_CHUNK_END=11], seq:[0, 1, 2, 3, 4, 5] [339]OB_MALLOC COST TOO MUCH TIME, cost_time=1045568, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1045561, WASH_BLOCK_END=7], seq:[0, 1, 2] [386]OB_MALLOC COST TOO MUCH TIME, cost_time=1055222, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=40, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1055160, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [386]OB_MALLOC COST TOO MUCH TIME, cost_time=1055240, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1055232, WASH_BLOCK_END=8], seq:[0, 1, 2] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=1038302, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=1038274, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=1038326, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1038310, WASH_BLOCK_END=16], seq:[0, 1, 2] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=1046229, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1046198, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=1046246, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1046238, WASH_BLOCK_END=8], seq:[0, 1, 2] [733]OB_MALLOC COST TOO MUCH TIME, cost_time=1030663, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1030625, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [733]OB_MALLOC COST TOO MUCH TIME, cost_time=1030681, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1030672, WASH_BLOCK_END=8], seq:[0, 1, 2] [861]OB_MALLOC COST TOO MUCH TIME, cost_time=1030758, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1030719, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [861]OB_MALLOC COST TOO MUCH TIME, cost_time=1030786, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1030777, WASH_BLOCK_END=9], seq:[0, 1, 2] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=1030577, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1030553, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [2025-08-14 09:04:46.726768] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=1750869][errcode=-4018] LOGGER COST TOO MUCH TIME, cost: 2050569, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1019750, ObMalloc#start=1, ObMalloc#end=1030793, ALLOC_END=25, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3b4b 0x113d34aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [856]OB_MALLOC COST TOO MUCH TIME, cost_time=1030594, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1030587, WASH_BLOCK_END=7], seq:[0, 1, 2] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=1030744, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1030714, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=2040936, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=55, WASH_BLOCK_END=1010126, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1030754], seq:[0, 1, 2, 3, 4] [800]OB_MALLOC COST TOO MUCH TIME, cost_time=1030763, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=1030738, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [2025-08-14 09:04:46.770548] EDIAG [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2006994, owner: ObLog, click_count: 6, time dist:[FORMAT_END=24, ObMalloc#start=0, ObMalloc#end=976247, ObMalloc#start=1, ObMalloc#end=1030612, FREE_END=110], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1b52e1b7 0x996d441 0x996cfd8 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [800]OB_MALLOC COST TOO MUCH TIME, cost_time=1030780, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1030771, WASH_BLOCK_END=9], seq:[0, 1, 2] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=1019220, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=18, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1019174, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=1019237, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1019228, WASH_BLOCK_END=9], seq:[0, 1, 2] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=1019194, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1019170, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=1019212, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1019203, WASH_BLOCK_END=9], seq:[0, 1, 2] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=1017826, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=1017799, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=1017844, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1017835, WASH_BLOCK_END=9], seq:[0, 1, 2] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=1014404, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=1014377, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=1014421, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1014413, WASH_BLOCK_END=8], seq:[0, 1, 2] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=1010468, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=23, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1010422, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=1010484, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1010477, WASH_BLOCK_END=7], seq:[0, 1, 2] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=1019790, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1019765, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=1019807, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1019799, WASH_BLOCK_END=8], seq:[0, 1, 2] [466]OB_MALLOC COST TOO MUCH TIME, cost_time=1906911, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=21, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=1906869, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [466]OB_MALLOC COST TOO MUCH TIME, cost_time=1906928, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1906919, WASH_BLOCK_END=9], seq:[0, 1, 2] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=1011485, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1011458, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=1011509, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1011500, WASH_BLOCK_END=9], seq:[0, 1, 2] [2025-08-14 09:04:46.736364] EDIAG [COMMON] loop_finished_dag_net_list (ob_tenant_dag_scheduler.cpp:3868) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2042007, owner: ObLog, click_count: 6, time dist:[FORMAT_END=33447, ObMalloc#start=2, ObMalloc#end=976888, ObMalloc#start=31, ObMalloc#end=1030787, FREE_END=852], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x200612d4 0x20070ebe 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:47.740334] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1039614, owner: ObLog, click_count: 6, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=1038332, ObMalloc#start=18, ObMalloc#end=38, FREE_END=1214], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:47.768375] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=28][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1011703, owner: ObLog, click_count: 6, time dist:[FORMAT_END=26, ObMalloc#start=1, ObMalloc#end=1011515, ObMalloc#start=25, ObMalloc#end=39, FREE_END=97], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:47.759970] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1020203, owner: ObLog, click_count: 6, time dist:[FORMAT_END=8, ObMalloc#start=0, ObMalloc#end=1019218, ObMalloc#start=17, ObMalloc#end=42, FREE_END=918], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:46.725190] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=1755489][errcode=-4018] LOGGER COST TOO MUCH TIME, cost: 2053432, owner: ObLog, click_count: 8, time dist:[FORMAT_END=1021366, ObMalloc#start=0, ObMalloc#end=1030686, ObMalloc#start=22, ObMalloc#end=46, ObMalloc#start=1136, ObMalloc#end=44, FREE_END=132], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3b4b 0x113d34aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:45.751879] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [339][EventTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3025308, owner: ObLog, click_count: 8, time dist:[FORMAT_END=1978118, ObMalloc#start=1, ObMalloc#end=1045574, ObMalloc#start=17, ObMalloc#end=37, ObMalloc#start=1124, ObMalloc#end=72, FREE_END=364], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:46.792325] EDIAG do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1986459, owner: ObLog, click_count: 6, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=1984223, ObMalloc#start=18, ObMalloc#end=42, FREE_END=2164], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x202c530b 0x202c6888 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:46.584534] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2193946, owner: ObLog, click_count: 8, time dist:[FORMAT_END=1136256, ObMalloc#start=2, ObMalloc#end=1055868, ObMalloc#start=25, ObMalloc#end=47, ObMalloc#start=1632, ObMalloc#end=37, FREE_END=79], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:47.717537] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1062675, owner: ObLog, click_count: 6, time dist:[FORMAT_END=22, ObMalloc#start=2, ObMalloc#end=1060735, ObMalloc#start=25, ObMalloc#end=35, FREE_END=1856], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:47.769036] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1011358, owner: ObLog, click_count: 6, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=1010490, ObMalloc#start=25, ObMalloc#end=43, FREE_END=787], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:46.795577] EDIAG eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1983467, owner: ObLog, click_count: 6, time dist:[FORMAT_END=13, ObMalloc#start=1, ObMalloc#end=1981079, ObMalloc#start=25, ObMalloc#end=41, FREE_END=2308], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x27a8a8c8 0x97b7e1a 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:47.732441] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=69][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1048155, owner: ObLog, click_count: 6, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=1046252, ObMalloc#start=25, ObMalloc#end=44, FREE_END=1821], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:47.764983] EDIAG [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1015831, owner: ObLog, click_count: 6, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=1014429, ObMalloc#start=25, ObMalloc#end=43, FREE_END=1322], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a1f867 0x27a135c3 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:47.759850] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=19][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1021094, owner: ObLog, click_count: 6, time dist:[FORMAT_END=29, ObMalloc#start=0, ObMalloc#end=1019243, ObMalloc#start=18, ObMalloc#end=909, FREE_END=895], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:47.759920] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1021050, owner: ObLog, click_count: 6, time dist:[FORMAT_END=7, ObMalloc#start=1, ObMalloc#end=1019814, ObMalloc#start=24, ObMalloc#end=326, FREE_END=878], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:47.761444] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1019561, owner: ObLog, click_count: 6, time dist:[FORMAT_END=8, ObMalloc#start=0, ObMalloc#end=1017850, ObMalloc#start=25, ObMalloc#end=47, FREE_END=1631], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:46.791717] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1987940, owner: ObLog, click_count: 8, time dist:[ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=1, FORMAT_END=5, ObMalloc#start=0, ObMalloc#end=1984741, ObMalloc#start=26, ObMalloc#end=2112, FREE_END=1050], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:47.723270] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=41][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1057916, owner: ObLog, click_count: 6, time dist:[FORMAT_END=23, ObMalloc#start=31, ObMalloc#end=1055246, ObMalloc#start=20, ObMalloc#end=58, FREE_END=2538], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:46.822307] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1957680, owner: ObLog, click_count: 8, time dist:[FORMAT_END=36, ObMalloc#start=2, ObMalloc#end=1954478, ObMalloc#start=24, ObMalloc#end=43, ObMalloc#start=2455, ObMalloc#end=40, FREE_END=602], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:46.789331] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=18][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1990707, owner: ObLog, click_count: 6, time dist:[FORMAT_END=13, ObMalloc#start=1, ObMalloc#end=1986993, ObMalloc#start=26, ObMalloc#end=53, FREE_END=3621], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:46.871516] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1908759, owner: ObLog, click_count: 8, time dist:[FORMAT_END=37, ObMalloc#start=2, ObMalloc#end=1906934, ObMalloc#start=31, ObMalloc#end=45, ObMalloc#start=1450, ObMalloc#end=40, FREE_END=220], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:45.789470] EDIAG [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [726][T1001_ObXAHbWor][T1001][Y0-0000000000000000-0-0] [lt=4005165][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2987714, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1930212, ObMalloc#start=1, ObMalloc#end=1055807, ObMalloc#start=24, ObMalloc#end=34, FREE_END=1636], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b64d8ba 0x1b64d154 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [385]OB_MALLOC COST TOO MUCH TIME, cost_time=956636, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 14, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13] [2025-08-14 09:04:46.760306] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2018130, owner: ObLog, click_count: 8, time dist:[FORMAT_END=9, ObMalloc#start=0, ObMalloc#end=961543, ObMalloc#start=32, ObMalloc#end=1055175, FREE_END=1371, ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=956652], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:46.239987] EDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] LOGGER COST TOO MUCH TIME, cost: 3496085, owner: ObLog, click_count: 8, time dist:[FORMAT_END=3494809, ObMalloc#start=1, ObMalloc#end=45, ObMalloc#start=23, ObMalloc#end=41, ObMalloc#start=818, ObMalloc#end=58, FREE_END=290], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x13b06ffb 0x958a39e 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [391]OB_MALLOC COST TOO MUCH TIME, cost_time=2963690, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1995123], seq:[0, 1] [2025-08-14 09:04:46.774257] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2963723, owner: ObLog, click_count: 5, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=2963706, ALLOC_END=4, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:46.783608] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=2022105][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2957437, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2957433, ALLOC_END=3, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:46.759504] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=54][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2016289, owner: ObLog, click_count: 6, time dist:[FORMAT_END=18362, ObMalloc#start=1, ObMalloc#end=1996967, ObMalloc#start=20, ObMalloc#end=38, FREE_END=901], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:46.768145] EDIAG [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2974026, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=9, ALLOC_BLOCK#end=3, FORMAT_END=2974013, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1d189030 0x1d18285e 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:46.726733] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3015483, owner: ObLog, click_count: 8, time dist:[ALLOC_BLOCK#start=42098, ALLOC_BLOCK#end=4, FORMAT_END=2972855, ObMalloc#start=0, ObMalloc#end=68, ObMalloc#start=30, ObMalloc#end=58, FREE_END=370], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:46.765637] EDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=997350][errcode=-4015] LOGGER COST TOO MUCH TIME, cost: 2010375, owner: ObLog, click_count: 8, time dist:[FORMAT_END=12356, ObMalloc#start=1, ObMalloc#end=1996952, ObMalloc#start=19, ObMalloc#end=37, ObMalloc#start=569, ObMalloc#end=39, FREE_END=401], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x279e59be 0x971b1c1 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [389]OB_MALLOC COST TOO MUCH TIME, cost_time=1997295, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1997288], seq:[0] [389]OB_MALLOC COST TOO MUCH TIME, cost_time=3004633, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=45, WASH_BLOCK_END=1007273, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1997309], seq:[0, 1, 2, 3, 4] [2025-08-14 09:04:46.739316] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3004652, owner: ObLog, click_count: 5, time dist:[FORMAT_END=8, ObMalloc#start=0, ObMalloc#end=3004643, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:46.741280] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:48.724920] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1980595][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:48.724940] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:48.739041] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:46.785489] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:48.778338] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1989817][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:46.726768] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=1750869][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:04:48.778364] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:49.741630] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:46.774257] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:49.742486] WDIAG ev_invoke_pending (ev.c:2136) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2963723][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:49.742501] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:49.743641] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4006] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:49.743656] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:49.741881] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:275) [807][T1002_TimerWK3_][T1002][Y0-0000000000000000-0-0] [lt=14] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=2, times=1291) [2025-08-14 09:04:46.783608] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=2022105] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:04:46.768145] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=19] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:49.746703] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=2974026] temporary file module use no blocks [2025-08-14 09:04:46.739316] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:49.748541] WDIAG ev_invoke_pending (ev.c:2136) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3004652][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:49.748559] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:49.748571] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:49.742513] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:48.781838] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1236][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:49.749271] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=965967][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:48.781789] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [339][EventTimer][T0][Y0-0000000000000000-0-0] [lt=3025308][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:46.768215] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2976734, owner: ObLog, click_count: 6, time dist:[FORMAT_END=2974417, ObMalloc#start=0, ObMalloc#end=57, ObMalloc#start=23, ObMalloc#end=50, FREE_END=2187], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:46.783599] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=185][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2961576, owner: ObLog, click_count: 6, time dist:[FORMAT_END=2957405, ObMalloc#start=2, ObMalloc#end=2904, ObMalloc#start=32, ObMalloc#end=42, FREE_END=1191], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:49.749474] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2976734][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:49.749948] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=464] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:49.749963] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [387]OB_MALLOC COST TOO MUCH TIME, cost_time=966570, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 11, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=966517], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10] [2025-08-14 09:04:47.760748] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1986911, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [474]OB_MALLOC COST TOO MUCH TIME, cost_time=966257, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 10, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9] [2025-08-14 09:04:45.994322] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2784463, owner: ObLog, click_count: 10, time dist:[FORMAT_END=741025, ObMalloc#start=1, ObMalloc#end=2040943, ObMalloc#start=17, ObMalloc#end=44, ObMalloc#start=1077, ObMalloc#end=55, FREE_END=1301, ALLOC_BLOCK#start=2, ALLOC_BLOCK#end=966268], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [876]OB_MALLOC COST TOO MUCH TIME, cost_time=965088, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 10, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9] [2025-08-14 09:04:47.760748] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:49.750683] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1986911][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:49.750699] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:49.750790] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=2784463][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:49.737336] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:49.750997] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13639][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:46.783605] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=733][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2963095, , BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:46.783605] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=733] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:04:49.754354] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7657] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:49.754389] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:49.742087] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=959071] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:49.754636] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12529][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:49.754655] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:49.760619] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:49.761391] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=805] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:49.761433] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=710] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:49.762104] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:49.741708] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=957715] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:49.764245] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22502][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:49.764268] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:49.764285] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=296] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:49.766160] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:49.775975] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:49.776212] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:49.780996] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2287] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:49.781095] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:49.781692] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=1628] gc stale ls task succ [2025-08-14 09:04:49.781827] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=1986459] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:04:49.781901] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=1730] gc stale ls task succ [2025-08-14 09:04:49.782580] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=1021094] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:04:49.783131] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=938] gc stale ls task succ [2025-08-14 09:04:49.783301] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=1020203] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:04:49.785566] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=1487] PNIO [ratelimit] time: 1755162289785564, bytes: 0, bw: 0.000000 MB/s, add_ts: 1003492, add_bytes: 0 [2025-08-14 09:04:49.788587] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:49.789513] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:49.791565] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=204] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:49.792508] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=194] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:49.846244] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:49.883709] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:46.845320] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=22] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:04:46.845320] EDIAG try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3867493, owner: ObLog, click_count: 5, time dist:[FORMAT_END=3867430, ObMalloc#start=2, ObMalloc#end=52, ALLOC_END=0, APPEND_END=9], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x26499505 0x97074a3 0x26498fa9 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:50.720962] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=625] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:50.721344] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:50.048817] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:49.746711] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:04:46.716108] EDIAG [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4001768, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=3, FORMAT_END=4001755, ObMalloc#start=0, ObMalloc#end=2, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0xfd250b6 0xfd247d3 0xfd244ea 0xfd242a2 0x1d19629b 0x1d17baa3 0x1d17ad94 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:46.716108] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:04:50.740486] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:50.741198] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=271] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:50.743610] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:50.743641] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=24] temporary file module use no blocks [2025-08-14 09:04:50.743651] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:04:50.744280] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=63] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:50.745278] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6703] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:50.745315] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:50.749317] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=3015483] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:04:50.750253] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:50.753145] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [389]OB_MALLOC COST TOO MUCH TIME, cost_time=986568, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=986560], seq:[0] [279]OB_MALLOC COST TOO MUCH TIME, cost_time=987521, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=32896, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=987518], seq:[0] [2025-08-14 09:04:51.752280] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:51.745686] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:51.749426] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=25] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:04:51.740448] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:233) [529][T1_RFLWorker][T1][YB42AC4D0007-00063C495B199C16-0-0] [lt=3987447] ObRemoteFetchWorker is running(thread_index=0) [2025-08-14 09:04:51.745778] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:51.756274] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10479][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:51.757064] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1169] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:51.757787] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=3867493] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:04:48.783359] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=237] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:04:48.783359] EDIAG [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=237][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2969992, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2969982, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=7], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x18bdb084 0x18bda6b1 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [807]OB_MALLOC COST TOO MUCH TIME, cost_time=1032987, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1032986], seq:[0] [807]OB_MALLOC COST TOO MUCH TIME, cost_time=1033207, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=62, WASH_BLOCK_END=125, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1033016], seq:[0, 1, 2, 3, 4] [2025-08-14 09:04:49.745323] EDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=3430][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 2010757, owner: ObLog, click_count: 9, time dist:[ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=87, ALLOC_BLOCK#start=11560, ALLOC_BLOCK#end=1, FORMAT_END=965882, ObMalloc#start=1, ObMalloc#end=1033216, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x96ecc48 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [389]OB_MALLOC COST TOO MUCH TIME, cost_time=1015022, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=986590], seq:[0, 1] [2025-08-14 09:04:50.753280] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1015059, owner: ObLog, click_count: 5, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=1015039, ALLOC_END=5, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:49.765336] EDIAG [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2001943, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2001935, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1fce4e91 0x1fce455d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:49.745323] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=3430][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:51.759152] WDIAG alarm_if_necessary (ob_timer_service.cpp:143) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=2010757][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:50.768934] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=15] current monitor number(seq_=-1) [2025-08-14 09:04:46.808744] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] [thread pool stat](pool_stat={pool:0x564d6e8922c0, last_check_time:1755162286808742, last_idle_time:102477764956, last_thread_count:12}, *pool={name:TimerWK, this:0x564d6e8922c0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:10, threads_idle_time:102477764956, tenant_id:500}, qsize=10, current_idle_time=102477764956) [2025-08-14 09:04:50.753280] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:51.769908] WDIAG ev_invoke_pending (ev.c:2136) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1015059][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:51.769931] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:49.765336] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=32] blacklist_loop exec finished(cost_time=16, is_enabled=true, send_cnt=0) [2025-08-14 09:04:51.772034] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=179] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:51.772075] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:51.771815] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:51.777175] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:51.781363] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=278] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:51.781426] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=331] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:51.782520] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=192] gc stale ls task succ [2025-08-14 09:04:51.782880] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=585] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:04:51.783657] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=194] gc stale ls task succ [2025-08-14 09:04:51.783846] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=175] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:04:51.789279] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=200] gc stale ls task succ [2025-08-14 09:04:51.789317] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=933] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:04:51.789764] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:51.790743] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:51.791327] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=181] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:51.792743] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=175] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:51.800862] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=227] PNIO [ratelimit] time: 1755162291800860, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007632, add_bytes: 0 [2025-08-14 09:04:51.857915] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=24] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:04:51.995954] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:52.027068] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:52.123377] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=382][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:52.725484] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:52.732395] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=248] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:52.732789] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=403] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:52.740658] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2965] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:52.691425] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:46.808744] EDIAG [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4952041, owner: ObLog, click_count: 5, time dist:[FORMAT_END=4952026, ObMalloc#start=1, ObMalloc#end=8, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x989c577 0x989bc96 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:52.743522] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9860] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:52.743662] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=334] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:52.745256] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:52.750383] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:52.760313] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3379] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:52.761254] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2814] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:52.761283] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=2995] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:52.761308] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3153] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:52.761665] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=991147] current monitor number(seq_=-1) [2025-08-14 09:04:50.771836] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=508] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623188194300, flush_io_finished_ret_:0, flush_io_finished_round_:26359, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:50.771478] EDIAG [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=4001768][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1988430, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1988421, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1d17bb52 0x1d17ada0 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:50.771478] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=4001768] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623188155757, flush_io_finished_ret_:0, flush_io_finished_round_:26337, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:52.754530] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:52.755899] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=6369] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [856]OB_MALLOC COST TOO MUCH TIME, cost_time=1000048, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1000047], seq:[0] [2025-08-14 09:04:52.756181] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:52.772357] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16151][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:50.768668] EDIAG [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=178][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2001242, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2001234, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1d17bb52 0x1d17ada0 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:50.768668] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=178] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623192183240, flush_io_finished_ret_:0, flush_io_finished_round_:26374, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:4, normal_idle_loop_cnt_:4, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:52.777847] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:52.781393] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=30] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:52.781737] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:52.782428] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:52.782581] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=18] gc stale ls task succ [2025-08-14 09:04:52.782937] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=19] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:04:52.783722] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=19] gc stale ls task succ [2025-08-14 09:04:52.783949] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:04:52.789406] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=13] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:04:52.790379] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:52.791245] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:52.792684] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:52.793257] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=19] gc stale ls task succ [2025-08-14 09:04:52.808426] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=35] PNIO [ratelimit] time: 1755162292808414, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007554, add_bytes: 0 [2025-08-14 09:04:52.855091] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:52.859780] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=28] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:04:52.886046] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:52.891438] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:53.010041] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:53.028964] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:53.124701] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=55][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:53.248310] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:52.741229] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=308][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:53.721431] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=34] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:53.726833] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=4952041] [thread pool stat](pool_stat={pool:0x7f99667cbcf0, last_check_time:1755162293726831, last_idle_time:34902873929, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x7f99667cbcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:34902873929, tenant_id:500}, qsize=-1, current_idle_time=34902873929) [2025-08-14 09:04:53.726873] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=38] [thread pool stat](pool_stat={pool:0x7f99667ccaf0, last_check_time:1755162293726873, last_idle_time:26677354348, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x7f99667ccaf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:26677354348, tenant_id:500}, qsize=-1, current_idle_time=26677354348) [2025-08-14 09:04:53.726886] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] [thread pool stat](pool_stat={pool:0x7f9969de56b0, last_check_time:1755162293726886, last_idle_time:102938681542, last_thread_count:17}, *pool={name:TimerWK, this:0x7f9969de56b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:16, threads_idle_time:102938681542, tenant_id:1}, qsize=10, current_idle_time=102938681542) [2025-08-14 09:04:53.726897] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] [thread pool stat](pool_stat={pool:0x7f99356b0e70, last_check_time:1755162293726897, last_idle_time:102366343177, last_thread_count:15}, *pool={name:TimerWK, this:0x7f99356b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:12, threads_idle_time:102366343177, tenant_id:1001}, qsize=12, current_idle_time=102366343177) [2025-08-14 09:04:53.726909] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] [thread pool stat](pool_stat={pool:0x7f99230b0e70, last_check_time:1755162293726908, last_idle_time:102534969161, last_thread_count:17}, *pool={name:TimerWK, this:0x7f99230b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:16, threads_idle_time:102534969161, tenant_id:1002}, qsize=13, current_idle_time=102534969161) [856]OB_MALLOC COST TOO MUCH TIME, cost_time=1954048, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1000060], seq:[0, 1] [2025-08-14 09:04:53.727314] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:49.849334] EDIAG [RS] get_valid_sts_after (ob_tenant_info_loader.cpp:454) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3872605, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1918506, ObMalloc#start=1, ObMalloc#end=1954067, ALLOC_END=2, APPEND_END=29], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1001d942 0x996d843 0x996d15a 0x996cfd8 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:52.755920] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=86][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:52.786740] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [854][T1002_ObXAHbWor][T1002][Y0-0000000000000000-0-0] [lt=27989] XA scheduler heartbeat task statistics(avg_time=1) [2025-08-14 09:04:53.732675] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [401]OB_MALLOC COST TOO MUCH TIME, cost_time=965737, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=965735], seq:[0] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=965891, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=45, WASH_BLOCK_END=90, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=965751], seq:[0, 1, 2, 3, 4] [2025-08-14 09:04:50.760344] EDIAG [SQL.PC] cache_evict (ob_plan_cache.cpp:1524) [14888][T1002_TimerWK7_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=1008144][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2970682, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2970676, ObMalloc#start=0, ObMalloc#end=2, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x15359551 0x15363f9c 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:50.771836] EDIAG [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=508][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1988035, owner: ObLog, click_count: 7, time dist:[FORMAT_END=1988024, ObMalloc#start=1, ObMalloc#end=4, ALLOC_END=0, APPEND_END=6, ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=13], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1d17bb52 0x1d17ada0 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:53.734823] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=970479] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:04:52.767495] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=19] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:53.734873] WDIAG ev_invoke_pending (ev.c:2136) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=965914][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:52.735570] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [412][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=5004310] [sql nio session](*s={this:0x7f98f733a030, session_id:3221590255, trace_id:YB42AC4D0007-00063C49651A3232-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:257}, err:0, last_decode_time:1755161952736711, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1716, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:04:50.760344] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1524) [14888][T1002_TimerWK7_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=1008144] end lib cache evict(tenant_id=1002, cache_evict_num=0, mem_hold=211812352, mem_limit=429496725, cache_obj_num=1331, cache_node_num=1330) [2025-08-14 09:04:49.743605] EDIAG [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24549][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3987610, owner: ObLog, click_count: 5, time dist:[FORMAT_END=3987590, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=16], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xff22be4 0x97a9cd7 0x97a9238 0x97a9307 0x97a8fe2 0x97a8cc7 0x97a88d2 0x97a7481 0x97a6a43 0x97a42c5 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:49.743605] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24549] get_min_unreplayed_log_info(lsn={lsn:5282737655}, scn={val:1755100947825033001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161624848121, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:13063726, ret_code_:0}}, next_to_submit_lsn_:{lsn:5282737655}, next_to_submit_scn_:{val:1755100947825033001, v:0}, base_lsn_:{lsn:4831543296}, base_scn_:{val:1755063445733337001, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7605000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:5282737655}, end_lsn:{lsn:5282737655}, read_buf:{buf_len_:3747840, buf_:0x7f98f7605000}, block_size:67104768, log_storage_:0x7f993e529bf0, log_storage_:{log_tail:{lsn:5282737655}, readable_log_tail:{lsn:5282737655}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:5234171904}, min_scn:{val:1755152244802535004, v:0}, curr_block_id:78, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:206, min_block_id:42, max_block_id:79, curr_writable_block_id:78}, logical_block_size_:67104768, curr_block_writable_size_:18539017, block_header_serialize_buf_:0x7f993e52a8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:85, read_io_size:301367296, read_disk_cost_ts:2364604, start_lsn:{lsn:4831543296}}}}}}) [2025-08-14 09:04:53.737273] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1049) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3987610] get_min_unreplayed_log_info(lsn={lsn:5282737655}, scn={val:1755100947825033001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161624848121, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:13063726, ret_code_:0}}, next_to_submit_lsn_:{lsn:5282737655}, next_to_submit_scn_:{val:1755100947825033001, v:0}, base_lsn_:{lsn:4831543296}, base_scn_:{val:1755063445733337001, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7605000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:5282737655}, end_lsn:{lsn:5282737655}, read_buf:{buf_len_:3747840, buf_:0x7f98f7605000}, block_size:67104768, log_storage_:0x7f993e529bf0, log_storage_:{log_tail:{lsn:5282737655}, readable_log_tail:{lsn:5282737655}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:5234171904}, min_scn:{val:1755152244802535004, v:0}, curr_block_id:78, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:206, min_block_id:42, max_block_id:79, curr_writable_block_id:78}, logical_block_size_:67104768, curr_block_writable_size_:18539017, block_header_serialize_buf_:0x7f993e52a8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:85, read_io_size:301367296, read_disk_cost_ts:2364604, start_lsn:{lsn:4831543296}}}}}}, ret=0) [2025-08-14 09:04:53.742440] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:53.743957] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:53.744008] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:53.745453] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:53.747387] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=982607] log compressor cycles once. (ret=0, cost_time=12296, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25880428544, fast_delete_log_mode=false) [2025-08-14 09:04:53.726817] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=8] finish to resize log external storage handler(current_ts=1755162291771590, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-08-14 09:04:53.735898] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=2042007] [COMPACTION DAG STATUS] (start_time=1755162282756056, end_time=1755162293735896, dag_status={}) [2025-08-14 09:04:53.742370] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3008863] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:04:53.748659] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6240] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:04:53.750407] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:53.750892] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8428] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:53.755338] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:53.760277] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:53.761254] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:53.761287] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:53.761311] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:53.763963] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=7934] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:04:53.764047] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=13] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:53.764693] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=1988430] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:53.764725] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=25] temporary file module use no blocks [2025-08-14 09:04:53.764735] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:04:53.764750] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:04:53.764766] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623195202718, flush_io_finished_ret_:0, flush_io_finished_round_:26338, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:53.764866] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:04:53.764910] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:04:53.767163] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=20948] snapshot_traffic_info once [2025-08-14 09:04:53.772408] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=2969992] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:04:53.773009] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=2001242] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:53.773031] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=17] temporary file module use no blocks [2025-08-14 09:04:53.773040] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:04:53.773053] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:04:53.773094] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=40] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623195212728, flush_io_finished_ret_:0, flush_io_finished_round_:26375, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:53.779411] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:53.781260] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:53.782651] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=14] gc stale ls task succ [2025-08-14 09:04:53.783020] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=15] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:04:53.783748] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=19] gc stale ls task succ [2025-08-14 09:04:53.784045] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=17] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:04:53.786272] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:53.789519] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=29] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:04:53.790623] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:53.791260] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:53.792261] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:53.793276] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=24] gc stale ls task succ [2025-08-14 09:04:53.800327] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:53.816043] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=20] PNIO [ratelimit] time: 1755162293816041, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007627, add_bytes: 0 [2025-08-14 09:04:53.860419] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=19] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:04:53.893553] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:54.011019] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:54.019700] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:54.129722] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=39][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:54.256383] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:54.272576] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:54.721424] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:53.755173] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [807][T1002_TimerWK3_TabletShell][T1002][Y0-0000000000000000-0-0] [lt=19] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-08-14 09:04:54.734145] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:54.735955] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=1988035] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:54.735985] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=24] temporary file module use no blocks [2025-08-14 09:04:54.735996] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:04:54.736024] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=26] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:04:54.736040] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623196175640, flush_io_finished_ret_:0, flush_io_finished_round_:26360, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:54.740290] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:54.744214] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:54.746021] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:44.726507] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:630) [14862][T1001_TimerWK5_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=20422] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x7f99297d7150, is_inited:true, freezing_scn:{val:1755136801284520000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136829134139001, v:0}, clog_base_lsn:{lsn:8186781696}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1753840867372367034, v:0}, all_id_meta:{id_meta:[{limited_id:1755161729775179000, latest_log_ts:{val:1755161719775179001, v:0}}, {limited_id:18000001, latest_log_ts:{val:1755157440846556001, v:0}}, {limited_id:60000001, latest_log_ts:{val:1755135657266496001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:46, palf_env_:0x7f99321fa030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1755136829134139001, v:0}, ls_id_={id:1}) [2025-08-14 09:02:28.722934] EDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:943) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE3-0-0] [lt=2025647][errcode=-4283] LOGGER COST TOO MUCH TIME, cost: 1016369, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=7542, ALLOC_BLOCK#end=1, FORMAT_END=1008825, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x148dcafc 0x93403a6 0x933e2ee 0x93363f3 0x932e8ab 0x96ac524 0x96a672d 0x96a5033 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x1e6837e2 0x1e683258 0x1056234f 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:54.751760] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:54.751797] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:54.752011] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:54.755728] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:54.730061] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [807][T1002_TimerWK3_TabletShell][T1002][Y0-0000000000000000-0-0] [lt=973406] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=2, times=1260) [2025-08-14 09:04:54.757449] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=27342][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:53.882241] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=2053432][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:04:54.760331] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:54.761310] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:54.761337] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:54.761361] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:53.747909] EDIAG [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11985][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1020125, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1020091, ObMalloc#start=23, ObMalloc#end=5, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x99721a1 0x997a72e 0x2006ed17 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:53.747909] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11985] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:04:54.747635] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=436] get wrs ts(ls_id={id:1}, delta=747007945, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:04:54.773193] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=21] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:54.782261] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [861]OB_MALLOC COST TOO MUCH TIME, cost_time=972721, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=972715], seq:[0] [861]OB_MALLOC COST TOO MUCH TIME, cost_time=973066, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=92, WASH_BLOCK_END=211, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=972758], seq:[0, 1, 2, 3, 4] [2025-08-14 09:04:53.880595] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=2050569][errcode=-4018] LOGGER COST TOO MUCH TIME, cost: 1848880, owner: ObLog, click_count: 5, time dist:[FORMAT_END=875791, ObMalloc#start=1, ObMalloc#end=973080, ALLOC_END=2, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3b4b 0x113d34aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=968645, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=21, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=968596, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3, 4, 5] [2025-08-14 09:04:53.880595] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=2050569][errcode=-4018] try free send task failed(ret=-4018) [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=968780, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=968654, WASH_BLOCK_END=123, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1], seq:[0, 1, 2, 3, 4] [2025-08-14 09:04:54.762212] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=17] current monitor number(seq_=-1) [864]OB_MALLOC COST TOO MUCH TIME, cost_time=966949, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=966919], seq:[0, 1, 2, 3, 4] [864]OB_MALLOC COST TOO MUCH TIME, cost_time=966958, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=966957], seq:[0, 1] [2025-08-14 09:04:54.764098] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=32] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [401]OB_MALLOC COST TOO MUCH TIME, cost_time=966137, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=966111], seq:[0, 1, 2, 3, 4] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=966148, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=966147], seq:[0, 1] [2025-08-14 09:04:54.764988] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=21] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:55.732620] WDIAG ev_invoke_pending (ev.c:2136) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=966167][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [14888]OB_MALLOC COST TOO MUCH TIME, cost_time=973165, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=12935], seq:[0, 1] [2025-08-14 09:04:53.760248] EDIAG [SQL.PC] runTimerTask (ob_plan_cache.cpp:2792) [14888][T1002_TimerWK7_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=2970682][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1969456, owner: ObLog, click_count: 5, time dist:[FORMAT_END=996281, ObMalloc#start=0, ObMalloc#end=973174, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x15365b5d 0x1536495d 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [832]OB_MALLOC COST TOO MUCH TIME, cost_time=977336, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=24, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=977292], seq:[0, 1, 2, 3, 4] [832]OB_MALLOC COST TOO MUCH TIME, cost_time=977365, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=977360], seq:[0, 1] [2025-08-14 09:04:54.743743] EDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1002464, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1002456, ALLOC_END=6, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0xfc0e73d 0x993f560 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [568]OB_MALLOC COST TOO MUCH TIME, cost_time=964140, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=964100, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=964166, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=964157, WASH_BLOCK_END=9], seq:[0, 1, 2] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=963918, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=963878, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=962919, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=962873, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=963949, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=963930, WASH_BLOCK_END=19], seq:[0, 1, 2] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=962947, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=962934, WASH_BLOCK_END=12], seq:[0, 1, 2] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=963382, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=963335, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=963405, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=963393, WASH_BLOCK_END=12], seq:[0, 1, 2] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=959656, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=959624, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=959702, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=959674, WASH_BLOCK_END=28], seq:[0, 1, 2] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=959714, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=959669, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=959732, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=959724, WASH_BLOCK_END=8], seq:[0, 1, 2] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=958211, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=958164, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=958230, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=958221, WASH_BLOCK_END=8], seq:[0, 1, 2] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=951747, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=951699, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=951766, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=951758, WASH_BLOCK_END=8], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=948813, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=948765, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=948832, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=948823, WASH_BLOCK_END=9], seq:[0, 1, 2] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=960243, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=960199, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=926161, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=926119, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=926188, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=926179, WASH_BLOCK_END=9], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=770068, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=770025, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=770085, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=770077, WASH_BLOCK_END=8], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=732000, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=731948, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=960403, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=960372, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=619108, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=20, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=619053, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=960457, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=960433, WASH_BLOCK_END=23], seq:[0, 1, 2] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=619126, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=619118, WASH_BLOCK_END=8], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=732026, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=732009, WASH_BLOCK_END=17], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=889308, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=889266, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=889326, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=889318, WASH_BLOCK_END=8], seq:[0, 1, 2] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=520948, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=520905, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=520973, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=520958, WASH_BLOCK_END=15], seq:[0, 1, 2] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=255301, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=255266, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=255318, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=255310, WASH_BLOCK_END=7], seq:[0, 1, 2] [2025-08-14 09:04:53.760248] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2792) [14888][T1002_TimerWK7_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=2970682] schedule next cache evict task(evict_interval=5000000) [2025-08-14 09:04:54.765913] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:54.769611] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=1020125] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:04:54.793254] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:55.746334] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=951639][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:54.782278] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:55.746594] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=962859][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:55.733614] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:55.747554] WDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13918][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:54.743743] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1001, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:8571658, max_lsn:7323687369, max_scn:{val:1755161749741193002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:7278610047}, reuse_lsn:{lsn:7323686131}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:7281744329}}, last_submit_lsn:{lsn:7323686131}, last_submit_end_lsn:{lsn:7323687369}, last_submit_log_id:8571658, last_submit_log_pid:49, max_flushed_lsn:{lsn:7323686009}, max_flushed_end_lsn:{lsn:7323686131}, max_flushed_log_pid:49, committed_end_lsn:{lsn:7323686009}, last_slide_log_id:8571656, last_slide_scn:{val:1755161747742105001, v:0}, last_slide_lsn:{lsn:7323685887}, last_slide_end_lsn:{lsn:7323686009}, last_slide_log_pid:49, last_slide_log_accum_checksum:2505458373, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f990112a530}, [StateMgr]={this:0x7f990112e9b0, self:"172.77.0.7:2882", palf_id:1001, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"PENDING", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:49}, leader:"0.0.0.0:0", leader_epoch:-1, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:7323687369}, scan_disk_log_finished:true, last_check_start_id:8571657, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161767744317, palf_role_change_cb:0x7f9901135470, allow_vote:true}, [ConfigMgr]={palf_id:1001, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:49, prev:{version:1, config:{config_version:{proposal_id:48, config_seq:47}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:49, config_seq:48}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_mode_pid:49}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_end_lsn:{lsn:7323685240}, prev_mode_pid:49}, reconfig_barrier:{prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_end_lsn:{lsn:7323685240}, prev_mode_pid:49}, persistent_config_version:{proposal_id:49, config_seq:48}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755161739755312, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755161739755312, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f990112bc70}, [ModeMgr]={palf_id:1001, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, accepted_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1001, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1753069671378640001, v:0}, base_lsn_for_block_gc:{lsn:7247314944}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:49}, log_config_meta:{version:3, proposal_id:49, prev:{version:1, config:{config_version:{proposal_id:48, config_seq:47}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:49, config_seq:48}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_mode_pid:49}, log_snapshot_meta:{version:2, base_lsn:{lsn:7247314944}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1753069671378640000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}}, log_meta_storage:{log_tail:{lsn:765952}, readable_log_tail:{lsn:765952}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7381524480}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/meta", dir_fd:232, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66338816, block_header_serialize_buf_:0x7f9901130bb8, flashback_version:0}, log_storage:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, palf_epoch:2, last_purge_throttling_ts:1755161769777679, this:0x7f990112f0f0}, [Reconfirm]={palf_id:1001, self:"172.77.0.7:2882", state:"WAITING_LOG_FLUSHED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:1755161769777679, this:0x7f990112eb30}) [2025-08-14 09:04:55.750083] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:04:55.750108] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:04:54.797823] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:54.980090] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:55.131409] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=65][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:55.018287] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:55.496254] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:55.694826] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:55.729323] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=33] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:55.752588] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23229][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:55.752575] INFO [CLOG] operator() (ob_log_replay_service.cpp:1678) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=478] get_replay_process success(id={id:1}, submitted_log_size=194888693, unsubmitted_log_size=0, replayed_log_size=194888693, unreplayed_log_size=0) [776]OB_MALLOC COST TOO MUCH TIME, cost_time=961772, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=960263, WASH_BLOCK_END=1509], seq:[0, 1, 2] [2025-08-14 09:04:55.740261] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=48] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:55.752624] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=45] get_min_unreplayed_log_info(lsn={lsn:7323685240}, scn={val:1755100946796590002, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873485, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:8946764, ret_code_:0}}, next_to_submit_lsn_:{lsn:7323685240}, next_to_submit_scn_:{val:1755100946796590002, v:0}, base_lsn_:{lsn:7180210176}, base_scn_:{val:1753840803166611000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:7323685240}, end_lsn:{lsn:7323685240}, read_buf:{buf_len_:3747840, buf_:0x7f98f7a05000}, block_size:67104768, log_storage_:0x7f9901131bf0, log_storage_:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:28, read_io_size:98508800, read_disk_cost_ts:645339, start_lsn:{lsn:7180210176}}}}}}) [2025-08-14 09:04:55.752676] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12396][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:55.743786] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:55.752677] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1085) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=52] replay status is not follower(min_unreplayed_lsn={lsn:7323685240}, base_lsn={lsn:7180210176}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873485, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:8946764, ret_code_:0}}, next_to_submit_lsn_:{lsn:7323685240}, next_to_submit_scn_:{val:1755100946796590002, v:0}, base_lsn_:{lsn:7180210176}, base_scn_:{val:1753840803166611000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:7323685240}, end_lsn:{lsn:7323685240}, read_buf:{buf_len_:3747840, buf_:0x7f98f7a05000}, block_size:67104768, log_storage_:0x7f9901131bf0, log_storage_:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:28, read_io_size:98508800, read_disk_cost_ts:645339, start_lsn:{lsn:7180210176}}}}}}) [2025-08-14 09:04:55.752721] INFO [CLOG] operator() (ob_log_replay_service.cpp:1678) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=43] get_replay_process success(id={id:1001}, submitted_log_size=143475064, unsubmitted_log_size=0, replayed_log_size=143475064, unreplayed_log_size=0) [2025-08-14 09:04:54.770533] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=2001943] blacklist_loop exec finished(cost_time=27, is_enabled=true, send_cnt=0) [2025-08-14 09:04:55.752750] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:174) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=8] dump tenant replay process(tenant_id=1002, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=322, replayed_log_size(MB)=322, last_submitted_log_size(MB)=322, last_replayed_log_size(MB)=322, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [660]OB_MALLOC COST TOO MUCH TIME, cost_time=450997, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=450966], seq:[0, 1, 2, 3, 4, 5, 6] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=451013, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=451012], seq:[0, 1] [2025-08-14 09:04:53.727826] EDIAG [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=3872605][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2022169, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2005080, ObMalloc#start=1, ObMalloc#end=17084, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1b52e1b7 0x996d441 0x996cfd8 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:55.301295] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:53.727826] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=3872605] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:04:55.753074] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=2022169] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:04:55.751627] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=958923][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:55.745815] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=974728] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:04:55.753329] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7500] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:04:55.753338] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:04:55.753347] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:04:55.753356] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:04:55.753846] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [408][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=25] [sql nio session](*s={this:0x7f98f72d9030, session_id:3221590656, trace_id:YB42AC4D0007-00063C4962EA9002-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:240}, err:0, last_decode_time:1755162243738749, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1390, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:04:54.894062] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:55.756606] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:55.756720] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:55.761737] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:55.761790] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:55.765243] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:55.765279] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:55.765668] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=11] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:55.753365] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:04:55.766328] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12942] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:04:55.766367] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=34] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:04:55.766380] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:04:55.767086] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:55.744790] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=977391] temporary file module use no blocks [2025-08-14 09:04:55.767307] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=22481] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:04:55.767329] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=20] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:04:55.767347] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=18] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623196204563, flush_io_finished_ret_:0, flush_io_finished_round_:26339, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:03:05.760006] EDIAG [SQL.DAS] retry_all_fail_tasks (ob_das_ref.cpp:364) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=1001018][errcode=-4038] LOGGER COST TOO MUCH TIME, cost: 109842074, owner: ObLog, click_count: 5, time dist:[FORMAT_END=109822147, ObMalloc#start=0, ObMalloc#end=6196, ALLOC_END=13726, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x15ebac0d 0x15eba8e6 0x9343ecf 0x9615b91 0x9613406 0x9613912 0x92df373 0x932ff51 0x932e8be 0x96ac524 0x96a672d 0x96a5033 0x97aa16f 0x124eac91 0x97a9944 0x261395f9 0xfa0e430 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:05.760006] WDIAG [SQL.DAS] retry_all_fail_tasks (ob_das_ref.cpp:364) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=1001018][errcode=-4038] can't do task level retry(ret=-4038, failed_task={parent:{tenant_id:1, task_id:50056313, op_type:2, errcode:-4038, can_part_retry:0, task_started:1, in_part_retry:0, in_stmt_retry:0, need_switch_param:0, trans_desc:{this:0x7f98c907e020, tx_id:{txid:21195872}, state:1, addr:"172.77.0.7:2882", tenant_id:1, session_id:1, assoc_session_id:1, client_sid:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1755162170771423, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999987, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1753069619, cluster_version:17180067074, seq_base:1755162170724889, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:2}, snapshot:{this:0x7f9956cd2798, valid:true, source:5, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:21195872}, scn:{branch:0, seq:2}}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, tablet_id:{id:154}, ls_id:{id:1}, tablet_loc:{tablet_id:{id:154}, ls_id:{id:1}, server:"172.77.0.7:2882", loc_meta:{table_loc_id:18446744073709551614, ref_table_id:154, related_table_ids:[101015, 101016], use_dist_das:0, select_leader:1, is_dup_table:0, is_weak_read:0, unuse_related_pruning:0, is_external_table:0, route_policy:0}, in_retry:0, partition_id:18446744073709551615, first_level_part_id:18446744073709551615}, related_ctdefs:[{ObDASBaseCtDef:{ObDASBaseDef:{op_type:2, children_cnt:0}, table_id:18446744073709551614, index_tid:101015, schema_version:1753069637599528, rowkey_cnt:4, spk_cnt:0, column_ids:[19, 16, 17, 18], column_types:[{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}], column_accuracys:[{length:64, precision:-1, scale:-1}, {length:-1, precision:25, scale:6}, {length:46, precision:-1, scale:-1}, {length:-1, precision:20, scale:0}], old_row_projector:[], new_row_projector:[3, 0, 1, 2], is_total_quantity_log:1, is_ignore:0, is_batch_stmt:0, is_insert_up:0, is_table_api:0, is_main_table_in_fts_ddl:0, tz_info:{id:-1, offset:28800, error_on_overlap_time:false}, table_param:{tenant_schema_version:1753073979411448, data_table:{table_id:101015, schema_version:1753069637599528, table_type:5, index_type:1, index_status:2, shadow_rowkey_column_num:0, doc_id_col_id:18446744073709551615, fulltext_col_id:18446744073709551615, index_name:"idx_server_module", fts_parser_name:"", fts_parser_properties:"", pk_name:"", columns:[{column_id:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:64, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:25, scale:6}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"EXT":"CURRENT_TIMESTAMP"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}, {column_id:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:46, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}, {column_id:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}], read_info:{schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, trans_col_index:-1, mview_old_new_col_index:-1, group_idx_col_index:-1, seq_read_column_count:4, max_col_index:3, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 2, 3], data:0x7f9827a3ae60, count:4, allocator:0x7f9827a1e6f0, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 3], data:0x7f9827a3ae80, count:4, allocator:0x7f9827a1e6f0, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9827a3ae20, count:4, allocator:0x7f9827a1e6f0, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, has_all_column_group:true, need_truncate_filter:false}, lob_inrow_threshold:4096}, col_descs:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], col_map:{is_inited_:true, use_array_:true, shadow_use_array_:false, has_:true, has_shadow_:false}}, encrypt_meta:[]}, table_rowkey_cids:[], table_rowkey_types:[]}, {ObDASBaseCtDef:{ObDASBaseDef:{op_type:2, children_cnt:0}, table_id:18446744073709551614, index_tid:101016, schema_version:1753069637600200, rowkey_cnt:4, spk_cnt:0, column_ids:[20, 16, 17, 18], column_types:[{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}], column_accuracys:[{length:64, precision:-1, scale:-1}, {length:-1, precision:25, scale:6}, {length:46, precision:-1, scale:-1}, {length:-1, precision:20, scale:0}], old_row_projector:[], new_row_projector:[4, 0, 1, 2], is_total_quantity_log:1, is_ignore:0, is_batch_stmt:0, is_insert_up:0, is_table_api:0, is_main_table_in_fts_ddl:0, tz_info:{id:-1, offset:28800, error_on_overlap_time:false}, table_param:{tenant_schema_version:1753073979411448, data_table:{table_id:101016, schema_version:1753069637600200, table_type:5, index_type:1, index_status:2, shadow_rowkey_column_num:0, doc_id_col_id:18446744073709551615, fulltext_col_id:18446744073709551615, index_name:"idx_server_event", fts_parser_name:"", fts_parser_properties:"", pk_name:"", columns:[{column_id:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:64, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:25, scale:6}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"EXT":"CURRENT_TIMESTAMP"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}, {column_id:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:46, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}, {column_id:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}], read_info:{schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, trans_col_index:-1, mview_old_new_col_index:-1, group_idx_col_index:-1, seq_read_column_count:4, max_col_index:3, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 2, 3], data:0x7f9827a3baf0, count:4, allocator:0x7f9827a1e6f0, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 3], data:0x7f9827a3bb10, count:4, allocator:0x7f9827a1e6f0, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9827a3bab0, count:4, allocator:0x7f9827a1e6f0, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, has_all_column_group:true, need_truncate_filter:false}, lob_inrow_threshold:4096}, col_descs:[column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], col_map:{is_inited_:true, use_array_:true, shadow_use_array_:false, has_:true, has_shadow_:false}}, encrypt_meta:[]}, table_rowkey_cids:[], table_rowkey_types:[]}], related_rtdefs:[{ObDASBaseRtDef:{ObDASBaseRtDef:{op_type:2, children_cnt:0}, timeout_ts:1755162198653066, sql_mode:4194304, prelock:false, tenant_schema_version:1753073979411448, is_for_foreign_key_check:false, affected_rows:0, is_immediate_row_conflict_check:true}, need_fetch_conflict:false, is_duplicated:false, direct_insert_task_id:0, use_put:false, ddl_task_id:0}, {ObDASBaseRtDef:{ObDASBaseRtDef:{op_type:2, children_cnt:0}, timeout_ts:1755162198653066, sql_mode:4194304, prelock:false, tenant_schema_version:1753073979411448, is_for_foreign_key_check:false, affected_rows:0, is_immediate_row_conflict_check:true}, need_fetch_conflict:false, is_duplicated:false, direct_insert_task_id:0, use_put:false, ddl_task_id:0}], task_status:1, related_tablet_ids:[{id:101015}, {id:101016}], das_task_node:, plan_line_id:0}, ins_ctdef:{ObDASBaseCtDef:{ObDASBaseDef:{op_type:2, children_cnt:0}, table_id:18446744073709551614, index_tid:154, schema_version:1753069637597680, rowkey_cnt:3, spk_cnt:0, column_ids:[16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33], column_types:[{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"LONGTEXT", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}], column_accuracys:[{length:-1, precision:25, scale:6}, {length:46, precision:-1, scale:-1}, {length:-1, precision:20, scale:0}, {length:64, precision:-1, scale:-1}, {length:64, precision:-1, scale:-1}, {length:256, precision:-1, scale:-1}, {length:512, precision:-1, scale:-1}, {length:256, precision:-1, scale:-1}, {length:536870911, precision:-1, scale:-1}, {length:256, precision:-1, scale:-1}, {length:512, precision:-1, scale:-1}, {length:256, precision:-1, scale:-1}, {length:512, precision:-1, scale:-1}, {length:256, precision:-1, scale:-1}, {length:512, precision:-1, scale:-1}, {length:256, precision:-1, scale:-1}, {length:512, precision:-1, scale:-1}, {length:512, precision:-1, scale:-1}], old_row_projector:[], new_row_projector:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17], is_total_quantity_log:1, is_ignore:0, is_batch_stmt:0, is_insert_up:0, is_table_api:0, is_main_table_in_fts_ddl:0, tz_info:{id:-1, offset:28800, error_on_overlap_time:false}, table_param:{tenant_schema_version:1753073979411448, data_table:{table_id:154, schema_version:1753069637597680, table_type:0, index_type:0, index_status:0, shadow_rowkey_column_num:0, doc_id_col_id:18446744073709551615, fulltext_col_id:18446744073709551615, index_name:"", fts_parser_name:"", fts_parser_properties:"", pk_name:"", columns:[{column_id:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:25, scale:6}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"EXT":"CURRENT_TIMESTAMP"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}, {column_id:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:46, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}, {column_id:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}, {column_id:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:64, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:64, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:256, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:512, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:256, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:24, meta_type:{type:"LONGTEXT", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:536870911, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:25, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:256, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:26, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:512, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:27, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:256, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:28, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:512, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:29, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:256, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:30, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:512, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:31, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:256, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:32, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:512, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:33, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:512, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}], read_info:{schema_column_count:18, schema_rowkey_cnt:3, rowkey_cnt:5, trans_col_index:-1, mview_old_new_col_index:-1, group_idx_col_index:-1, seq_read_column_count:3, max_col_index:19, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:18, 0, 1, 2, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19], data:0x7f9827a39e20, count:18, allocator:0x7f9827a1e6f0, capacity:18, init_cnt:18}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:18, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17], data:0x7f9827a39e70, count:18, allocator:0x7f9827a1e6f0, capacity:18, init_cnt:18}}, cols_desc:{ObIArray:[cnt:18, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=23 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=24 {type:"LONGTEXT", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=25 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=26 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=27 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=28 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=29 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=30 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=31 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=32 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=33 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0], data:0x7f9827a39d40, count:18, allocator:0x7f9827a1e6f0, capacity:18, init_cnt:18}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, has_all_column_group:true, need_truncate_filter:false}, lob_inrow_threshold:4096}, col_descs:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=23 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=24 {type:"LONGTEXT", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=25 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=26 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=27 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=28 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=29 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=30 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=31 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=32 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=33 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0], col_map:{is_inited_:true, use_array_:true, shadow_use_array_:false, has_:true, has_shadow_:false}}, encrypt_meta:[]}, table_rowkey_cids:[], table_rowkey_types:[]}, ins_rtdef:{ObDASBaseRtDef:{ObDASBaseRtDef:{op_type:2, children_cnt:0}, timeout_ts:1755162198653066, sql_mode:4194304, prelock:false, tenant_schema_version:1753073979411448, is_for_foreign_key_check:false, affected_rows:0, is_immediate_row_conflict_check:true}, need_fetch_conflict:false, is_duplicated:false, direct_insert_task_id:0, use_put:false, ddl_task_id:0}, insert_buffer:{mem_attr:tenant_id=1, label=DASInsertBuffer, ctx_id=0, prio=0, buffer_memory:427, buffer_size:1, datum_store:NULL}}) [2025-08-14 09:04:55.773305] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=27] temporary file module use no blocks [2025-08-14 09:04:55.773325] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=18] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:04:55.773340] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:04:55.773364] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=24] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623197212992, flush_io_finished_ret_:0, flush_io_finished_round_:26377, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:55.776982] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=260] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:04:55.779676] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:55.782222] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:55.783401] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=244] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:04:55.783849] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=963576] gc stale ls task succ [2025-08-14 09:04:55.787126] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=428] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:04:55.790259] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1556] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:55.793258] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:55.794259] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=953046] gc stale ls task succ [2025-08-14 09:04:55.831263] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=926680] PNIO [ratelimit] time: 1755162295831261, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007617, add_bytes: 0 [2025-08-14 09:04:55.865387] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:55.890672] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=485553, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=44, WASH_BLOCK_END=485508], seq:[0, 1, 2] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=595511, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=582994, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=12479, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=595587, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=595548, WASH_BLOCK_END=39], seq:[0, 1, 2] [2025-08-14 09:04:55.767687] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1002464] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [2025-08-14 09:04:56.755880] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2800) [14888][T1002_TimerWK7_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=980670] schedule next cache evict task(evict_interval=5000000) [2025-08-14 09:04:55.953004] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=890455] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:04:56.732711] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=966982] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:04:56.723697] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:56.756003] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=32255][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:56.756049] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=987447] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:04:56.756073] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:56.756081] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:56.755840] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=986650] current log_task status(palf_id=1001, self="172.77.0.7:2882", log_id=8571657, log_task={header:{begin_lsn:{lsn:7323686009}, end_lsn:{lsn:7323686131}, log_id:8571657, min_scn:{val:1755161749741193000, v:0}, max_scn:{val:1755161749741193000, v:0}, data_len:66, proposal_id:49, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:7323686009}, data_checksum:2222604892, accum_checksum:1531236791, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1755161749745909, freeze_ts:1755161749745914, submit_ts:1755161749745940, flushed_ts:1755161785758706, gen_to_freeze cost time:5, gen_to_submit cost time:31, submit_to_flush cost time:36012766}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [2025-08-14 09:04:56.756724] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:56.755906] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [14888][T1002_TimerWK7][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:56.735654] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:56.761680] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:56.761717] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:56.762403] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:56.762493] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:56.762689] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][YB42AC4D0007-00063C495F8A195C-0-0] [lt=1][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:56.764560] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=14] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:56.768046] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=30] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:56.768081] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=30] temporary file module use no blocks [2025-08-14 09:04:56.768087] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=5] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:04:56.768101] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:04:56.768118] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=17] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623198207147, flush_io_finished_ret_:0, flush_io_finished_round_:26340, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:56.773479] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:56.773505] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=24] temporary file module use no blocks [2025-08-14 09:04:56.773511] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=5] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:04:56.773525] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:04:56.773541] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623198213161, flush_io_finished_ret_:0, flush_io_finished_round_:26378, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:56.780613] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:56.782240] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=26] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:56.782808] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=968289] gc stale ls task succ [2025-08-14 09:04:56.783875] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=12] gc stale ls task succ [2025-08-14 09:04:56.789143] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] [sql nio session](*s={this:0x7f98f72d0030, session_id:3221589415, trace_id:YB42AC4D0007-00063C4965FA43DC-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:245}, err:0, last_decode_time:1755161751714366, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1662, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:04:56.789177] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=34] [sql nio session](*s={this:0x7f98f72d5030, session_id:3221590200, trace_id:YB42AC4D0007-00063C4965FA43E0-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:248}, err:0, last_decode_time:1755161776777167, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1593, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:04:56.791217] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:56.793271] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:56.794327] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=17] gc stale ls task succ [2025-08-14 09:04:56.838885] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=26] PNIO [ratelimit] time: 1755162296838883, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007622, add_bytes: 0 [2025-08-14 09:04:56.856152] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:56.956075] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=801717] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:04:56.956219] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:57.056304] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:57.063616] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:57.152775] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=599865][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:57.156410] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:57.256470] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:57.356499] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:57.456531] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:57.556590] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:57.656648] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:55.768098] EDIAG [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22044][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1945595, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1945583, ObMalloc#start=1, ObMalloc#end=5, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0xfa06846 0xfa05f62 0xfa05c84 0xfa05a50 0x1b51bd85 0x1b51b9c4 0x97a8d82 0x97a88d2 0x97a7481 0x97a6a43 0x97a42c5 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:55.768098] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22044] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:04:57.721008] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:57.716201] INFO [STORAGE] try_shrink (ob_compaction_memory_pool.cpp:518) [807][T1002_TimerWK3_MergeMemPool][T1002][Y0-0000000000000000-0-0] [lt=21] Compaction Mem Pool current stat: (ret=0, max_block_num=64, total_block_num=32, Chunk list size: =4) [2025-08-14 09:04:57.732640] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2234] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:57.740619] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=393] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:57.744086] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4383] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:57.746428] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:57.746562] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5626] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:57.748356] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:57.750135] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=17428] current monitor number(seq_=-1) [2025-08-14 09:04:57.750948] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:57.751360] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:57.751504] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=1938] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:04:57.751787] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=249] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:57.753074] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [880][T1002_ArcSender][T1002][YB42AC4D0007-00063C495EB99C16-0-0] [lt=32] ObArchiveSender is running(thread_index=5) [2025-08-14 09:04:57.753483] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:57.756056] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=23217] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:04:57.756926] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:57.752119] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=333] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:57.753564] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=407] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:04:57.753682] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=297] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:04:57.746632] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5688] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:57.759417] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12764][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:57.759421] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:57.753015] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=205] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:57.759675] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=6643] temporary file module use no blocks [2025-08-14 09:04:57.759685] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:04:57.759698] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:04:57.759713] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623199192713, flush_io_finished_ret_:0, flush_io_finished_round_:26363, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:57.753528] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:57.756823] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=56] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:57.760438] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:57.761415] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:57.761623] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:57.761666] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:57.765164] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=19] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:57.767784] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:57.773634] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=34] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:57.773676] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=37] temporary file module use no blocks [2025-08-14 09:04:57.773687] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:04:57.773701] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:04:57.773715] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623199213343, flush_io_finished_ret_:0, flush_io_finished_round_:26379, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:57.775277] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:57.775307] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=25] temporary file module use no blocks [2025-08-14 09:04:57.775318] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:04:57.775331] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:04:57.778647] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:57.780647] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:57.782255] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:57.782926] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:04:57.793260] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:57.794351] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=19] gc stale ls task succ [2025-08-14 09:04:57.846496] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=24] PNIO [ratelimit] time: 1755162297846493, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007610, add_bytes: 0 [2025-08-14 09:04:57.856976] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:57.956777] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=19] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:04:57.957033] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:58.057081] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:58.157145] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:58.161257] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:58.257216] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:04:58.357244] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=-4076] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:58.696507] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=57162][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:58.721053] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:57.775345] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623199207914, flush_io_finished_ret_:0, flush_io_finished_round_:26341, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:57.783946] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:04:57.791197] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:58.732556] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=939936][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:58.733265] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:02.721807] EDIAG [SERVER] runTimerTask (ob_eliminate_task.cpp:172) [510][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1004036, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1004032, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9826e08 0x98267df 0x982650b 0x98262e0 0x12e914a4 0x12e90ef0 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 08:59:24.764191] EDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [639][T1_L0_G0][T1][YB42AC4D0007-00063C4965DA1F34-0-0] [lt=25][errcode=0] LOGGER COST TOO MUCH TIME, cost: 101833540, owner: ObLog, click_count: 5, time dist:[FORMAT_END=983786, ObMalloc#start=1, ObMalloc#end=56, ALLOC_END=100849696, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x153ca731 0x92a261c 0x9293dd5 0x9285f13 0x927efad 0x926c154 0x12c5a0c9 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:58.736040] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=24] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25880428544, fast_delete_log_mode=false) [2025-08-14 09:04:58.740629] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:58.743083] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:58.743565] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:58.733806] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [510][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=1004036] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=530841, evict_batch_count=0, elapse_time=0, size_used=65536, mem_used=14558208) [2025-08-14 09:04:58.744841] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [510][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=11016][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:58.732286] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [854][T1002_ObXAHbWor][T1002][Y0-0000000000000000-0-0] [lt=940383] XA scheduler heartbeat task statistics(avg_time=1) [2025-08-14 09:04:58.737717] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=918] LogLoopThread round_cost_time(us)(round_cost_time=48857898) [2025-08-14 09:04:58.745494] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:08.761887] EDIAG [STORAGE.TRANS] handle_trans_keepalive (ob_trans_service_v4.cpp:914) [978][T1002_L0_G0][T1002][YB42AC4D0007-00063C4979C99C25-0-0] [lt=976684][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1974137, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1974126, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=7], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b831d02 0x1b7f1ede 0x984d055 0x9280839 0x926c154 0x12c5a0c9 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:58.733381] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [807][T1002_TimerWK3_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=0] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):258, sequence:1}) [2025-08-14 09:04:58.750241] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=20] current monitor number(seq_=-1) [2025-08-14 09:04:58.753637] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:97) [807][T1002_TimerWK3_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=1] BlockGCTimerTask success(ret=0, cost_time_us=994163, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.77.0.7:2882", log_dir:"/root/ob/observer/store/clog/tenant_1002", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):258, sequence:1}, log_alloc_mgr_:{flying_log_task:1, flying_meta_task:2}}) [2025-08-14 09:04:58.753732] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [807][T1002_TimerWK3_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=0] ====== check clog disk timer task ====== [2025-08-14 09:04:58.753841] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [807][T1002_TimerWK3_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=104] get_disk_usage(ret=0, capacity(MB):=24883, used(MB):=8642) [2025-08-14 09:04:58.759787] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=24] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:58.759805] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=14] temporary file module use no blocks [2025-08-14 09:04:58.759819] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:04:58.765262] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=5432] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:04:58.765277] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623200199507, flush_io_finished_ret_:0, flush_io_finished_round_:26364, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:58.768557] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=21] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:58.768832] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=3700][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:58.755694] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2959] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:58.771148] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15430][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:58.734227] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1945595] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162285733144}, version_delta=750995710) [2025-08-14 09:04:58.773785] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:58.773812] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=23] temporary file module use no blocks [2025-08-14 09:04:58.773820] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:04:58.773832] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:04:58.773844] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623200213509, flush_io_finished_ret_:0, flush_io_finished_round_:26380, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:58.771108] INFO [STORAGE.TRANS] process (ob_trans_rpc.cpp:150) [978][T1002_L0_G0][T1002][YB42AC4D0007-00063C4979C99C25-0-0] [lt=1974137] transaction rpc statistics(total_rt=238018429, total_process=1, avg_rt=119009214) [2025-08-14 09:04:58.780669] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:58.782295] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:58.782997] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=19] gc stale ls task succ [2025-08-14 09:04:58.784287] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=947136] gc stale ls task succ [2025-08-14 09:04:58.790665] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:58.793179] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:58.809790] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:58.854030] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:58.854132] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=26] PNIO [ratelimit] time: 1755162298854130, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007637, add_bytes: 0 [2025-08-14 09:04:58.868663] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=44][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:58.892679] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=45][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:58.957366] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=20] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:04:59.016032] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:59.164683] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:59.184243] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:59.720988] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:58.794412] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=18] gc stale ls task succ [2025-08-14 09:04:59.732282] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:59.733900] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=874962][errcode=-4076] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:59.736098] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=955574] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:59.736128] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=25] temporary file module use no blocks [2025-08-14 09:04:59.736138] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:04:59.736151] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:04:59.736166] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623201172134, flush_io_finished_ret_:0, flush_io_finished_round_:26342, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:58.768629] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:140) [14888][T1002_TimerWK7_CompactionRefre][T1002][YB42AC4D0007-00063C4976899CA3-0-0] [lt=0] InfoPoolResizeTask(cost_ts=998033) [2025-08-14 09:04:59.736253] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [14888][T1002_TimerWK7][T1002][YB42AC4D0007-00063C4976899CA3-0-0] [lt=966161][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:59.741690] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:59.743084] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:59.743992] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:59.745843] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:59.747717] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:147) [14888][T1002_TimerWK7_][T1002][Y0-0000000000000000-0-0] [lt=4251] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-08-14 09:04:59.749074] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11405] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:04:59.749133] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:04:59.749171] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:04:59.749204] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:04:59.749215] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:04:59.749227] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:04:59.749363] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6883] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:59.750328] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=284] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:59.752813] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:59.759093] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:59.752265] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=15] current monitor number(seq_=-1) [2025-08-14 09:04:59.759496] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:55.734319] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:647) [780][T1002_BalanceEx][T1002][YB42AC4D0007-00063C4964D9FEE3-0-0] [lt=1016369][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs, exec_ctx)(ret=-4283, session_id=3221590966, *tx_desc={this:0x7f98b5c049a0, tx_id:{txid:0}, state:1, addr:"172.77.0.7:2882", tenant_id:1002, session_id:3221590966, assoc_session_id:3221590966, client_sid:3221590966, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1755162141731585, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180067074, seq_base:1755162141721205, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}) [2025-08-14 09:04:59.764567] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=11] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:59.765324] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=22] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:59.765347] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=16] temporary file module use no blocks [2025-08-14 09:04:59.765354] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:04:59.783063] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=25] gc stale ls task succ [2025-08-14 09:04:59.784350] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=17] gc stale ls task succ [2025-08-14 09:04:59.791487] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:59.850260] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:04:59.860245] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=874139][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:04:59.860755] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=18] PNIO [ratelimit] time: 1755162299860752, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006622, add_bytes: 0 [2025-08-14 09:05:00.029122] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:00.721414] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=37] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:00.732315] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=30] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:00.734537] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:59.765366] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:04:59.780678] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:00.736613] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=954488][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:59.850306] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:00.738368] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=886680] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:00.738454] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=53] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:00.738496] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:05:00.738506] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:00.738514] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:04:59.957251] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:00.095908] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:00.759483] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=308] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:00.759542] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=201] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:00.760895] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:00.761840] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=205] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:59.747738] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [14888][T1002_TimerWK7_][T1002][Y0-0000000000000000-0-0] [lt=14] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-08-14 09:05:00.762714] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:00.774001] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=30] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:00.774019] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] temporary file module use no blocks [2025-08-14 09:05:00.774025] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=5] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:00.774035] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:00.774048] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623202213710, flush_io_finished_ret_:0, flush_io_finished_round_:26382, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:00.779703] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:00.781611] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=189] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:00.783260] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=10] gc stale ls task succ [2025-08-14 09:05:00.786267] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=9] gc stale ls task succ [2025-08-14 09:05:00.791540] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:00.793271] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=248] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:00.794541] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=247] gc stale ls task succ [2025-08-14 09:05:00.832417] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=1848880][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:05:00.861150] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=18] PNIO [ratelimit] time: 1755162300861149, bytes: 0, bw: 0.000000 MB/s, add_ts: 1000397, add_bytes: 0 [2025-08-14 09:05:00.938740] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:05:00.938799] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:00.938846] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:00.938889] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:00.938929] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:05:00.938945] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:00.938961] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:00.964307] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=251] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:01.039146] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:01.173670] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=1728113][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:01.216520] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:01.277076] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:01.698246] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:01.721693] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:01.732329] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:00.773433] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:288) [884][T1002_ArcFetche][T1002][YB42AC4D0007-00063C495ED99C16-0-0] [lt=3988806] ObArchiveFetcher is running(thread_index=1) [2025-08-14 09:05:01.736514] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:01.736547] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27] temporary file module use no blocks [2025-08-14 09:05:01.736558] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:01.736573] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:01.736588] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623203176024, flush_io_finished_ret_:0, flush_io_finished_round_:26344, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:00.758271] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [339][EventTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:01.744990] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:01.749255] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6434] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:01.749296] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3635] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:01.749323] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1609] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:01.750267] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6995] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:01.757208] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=392] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:01.759797] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=21] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:01.759866] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=6] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:00.861174] INFO pn_print_diag_info (group.c:775) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=22] PNIO client_send:0/0, client_queue_time=0, cnt=0, server_send:0/0, server_queue_time=0, cnt=0 [2025-08-14 09:05:01.760027] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=321] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:05:00.738962] INFO pn_print_diag_info (group.c:775) [322][pnio2][T0][Y0-0000000000000000-0-0] [lt=241] PNIO client_send:0/0, client_queue_time=0, cnt=0, server_send:0/0, server_queue_time=0, cnt=0 [2025-08-14 09:05:01.761244] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][YB42AC4D0007-00063C495F8A1961-0-0] [lt=1][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:01.761302] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:01.761566] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:01.743259] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4535] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:01.765121] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=225] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:01.765155] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:01.765467] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=554] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:01.768468] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:01.774263] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:01.776834] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:01.776861] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=22] temporary file module use no blocks [2025-08-14 09:05:01.776868] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:01.776883] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:01.776899] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=16] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623203213843, flush_io_finished_ret_:0, flush_io_finished_round_:26383, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:01.780702] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:01.781558] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:01.783317] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=25] gc stale ls task succ [2025-08-14 09:05:01.786292] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=23] gc stale ls task succ [2025-08-14 09:05:01.791402] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=36] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:01.793275] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:01.794604] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=19] gc stale ls task succ [2025-08-14 09:05:01.868765] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=897505] PNIO [ratelimit] time: 1755162301868763, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007614, add_bytes: 0 [2025-08-14 09:05:01.872325] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:01.894398] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:01.965137] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:01.968707] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=29] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:02.174291] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=37][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:02.266597] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:02.277206] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:02.726269] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:01.845062] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:05:02.732332] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:02.740692] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:02.741865] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=30] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:02.741906] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=32] temporary file module use no blocks [2025-08-14 09:05:02.741919] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:02.741936] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:02.741961] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=24] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623204176387, flush_io_finished_ret_:0, flush_io_finished_round_:26345, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:02.744264] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:02.040575] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:59.760675] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=251] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:00.165540] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=39][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:00.165540] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=39][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2576858, , BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:59.760675] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=251][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2980399, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:00.757261] EDIAG [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=980748][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2003695, owner: ObLog, click_count: 6, time dist:[FORMAT_END=2003486, ObMalloc#start=0, ObMalloc#end=54, ObMalloc#start=23, ObMalloc#end=34, FREE_END=98], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1fce4e91 0x1fce455d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:02.747285] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:00.736385] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=969551] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623201205067, flush_io_finished_ret_:0, flush_io_finished_round_:26365, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:00.745265] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:233) [709][T1001_RFLWorker][T1001][YB42AC4D0007-00063C495BE99C16-0-0] [lt=981122] ObRemoteFetchWorker is running(thread_index=0) [2025-08-14 09:05:02.764786] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=23] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:02.765565] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=357] current monitor number(seq_=-1) [2025-08-14 09:05:02.876384] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=26] PNIO [ratelimit] time: 1755162302876382, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007619, add_bytes: 0 [2025-08-14 09:05:02.762404] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [807][T1002_TimerWK3_TabletShell][T1002][Y0-0000000000000000-0-0] [lt=31] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-08-14 09:04:59.758914] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [663][T1001_StandbySc][T1001][YB42AC4D0007-00063C495C7A037F-0-0] [lt=667][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3976154, , BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:01.940618] EDIAG [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1797810, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=3, FORMAT_END=1797797, ObMalloc#start=0, ObMalloc#end=2, ALLOC_END=0, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1f9d7fcf 0x99b8be2 0x1f981833 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:00.736385] EDIAG [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=969551][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2009857, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=14, FORMAT_END=2009838, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1d17bb52 0x1d17ada0 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:59.761332] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=7038][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3974362, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:01.940618] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:05:03.741208] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=1797810] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:03.741257] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:03.741298] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:03.741336] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:05:03.741349] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:03.741361] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:04:59.761332] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=7038] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:03.741727] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=3974362][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:03.741746] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:03.741759] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:03.742278] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=949588][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:01.761818] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18528][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:01.762562] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1958859, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1957657, ObMalloc#start=1, ObMalloc#end=89, ObMalloc#start=24, ObMalloc#end=42, FREE_END=1046], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:01.761818] EDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18528][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1977502, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=6, ALLOC_BLOCK#end=2, FORMAT_END=1977487, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [357]OB_MALLOC COST TOO MUCH TIME, cost_time=1973112, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1973111], seq:[0] [357]OB_MALLOC COST TOO MUCH TIME, cost_time=2966424, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=60, WASH_BLOCK_END=51, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2966304], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:00.757648] EDIAG [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2982541, owner: ObLog, click_count: 5, time dist:[FORMAT_END=16097, ObMalloc#start=0, ObMalloc#end=2966438, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x18bdb084 0x18bda6b1 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [396]OB_MALLOC COST TOO MUCH TIME, cost_time=993655, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=25808], seq:[0, 1] [2025-08-14 09:04:59.761395] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=205][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3977349, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:01.762557] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1008121, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1007250, ObMalloc#start=1, ObMalloc#end=64, ObMalloc#start=21, ObMalloc#end=34, FREE_END=751], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:01.766137] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1004528, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1003808, ObMalloc#start=0, ObMalloc#end=43, ObMalloc#start=25, ObMalloc#end=33, FREE_END=619], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:00.738962] EDIAG pn_print_diag_info (group.c:775) [322][pnio2][T0][Y0-0000000000000000-0-0] [lt=241][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1019558, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1019553, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x27a90dc3 0x27a8aaf2 0x97b7c8d 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:00.745265] EDIAG [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:233) [709][T1001_RFLWorker][T1001][YB42AC4D0007-00063C495BE99C16-0-0] [lt=981122][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2001148, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2001147, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x11359281 0x98b739c 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:00.757298] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1989206, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1988965, ObMalloc#start=1, ObMalloc#end=73, ObMalloc#start=30, ObMalloc#end=36, FREE_END=101], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [474]OB_MALLOC COST TOO MUCH TIME, cost_time=992919, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=992913], seq:[0] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=1001683, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=55, WASH_BLOCK_END=107, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1001511], seq:[0, 1, 2, 3, 4] [2025-08-14 09:04:59.998696] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3744571, , BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:03.742349] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1977502] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:03.742367] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:32.736278] INFO [STORAGE] get_ls_tx_data_memory_info_ (ob_tenant_freezer.cpp:735) [491][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=17] TxData Memory Statistic(logstream info): (ls_id={id:1}, Frozen TxData Memory(MB)=0, Active TxData Memory(MB)=1) [2025-08-14 09:05:00.757648] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=32] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:04:59.761395] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=205] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:03.744801] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3977349][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:03.744826] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:03.744843] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:03.746503] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:59.998696] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:03.748255] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1989206][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:03.749249] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=971] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:03.749267] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:03.749553] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=503] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:03.755981] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=0] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:03.755960] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2325] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:03.761924] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1669] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:03.764273] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=1801] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:05:03.764340] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=1969] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:03.764372] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=2130] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:03.764417] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=11] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:03.767710] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=1257] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:03.769463] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [807][T1002_TimerWK3_TabletShell][T1002][Y0-0000000000000000-0-0] [lt=959420] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=2, times=1261) [2025-08-14 09:05:03.780266] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=170] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:03.782253] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=225] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:03.785324] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=304] gc stale ls task succ [2025-08-14 09:05:03.793988] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:03.794017] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=955255] gc stale ls task succ [2025-08-14 09:05:03.841557] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:05:03.841621] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:03.884001] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=15] PNIO [ratelimit] time: 1755162303883998, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007616, add_bytes: 0 [2025-08-14 09:05:03.970867] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=178] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:04.701370] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=51][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:04.723513] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=255] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:04.729941] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:04.736376] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2285] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:04.743215] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1665] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:04.743584] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1703] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:04.745588] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:04.751749] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:04.756470] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=57] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:04.758118] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:04.769109] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=27] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:03.799278] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=173] gc stale ls task succ [2025-08-14 09:05:03.841677] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:04.294255] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:04.743945] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:04.745623] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:03.748961] EDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=3744571][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1024592, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1024581, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=7], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ed35c 0x97aead0 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:03.748961] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=3744571][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:04.777793] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:04.777868] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=26] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:05:04.777937] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=13] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:04.777969] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=10] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:04.781295] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:04.782260] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:04.773721] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=930594] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:04.782351] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8582] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:05:04.782366] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:04.782378] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:04.785400] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:05:04.792287] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:04.794082] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=10] gc stale ls task succ [2025-08-14 09:05:04.800262] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=972776] gc stale ls task succ [2025-08-14 09:05:04.885301] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:04.891616] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=32] PNIO [ratelimit] time: 1755162304891614, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007616, add_bytes: 0 [2025-08-14 09:05:04.972776] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=22] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:05.721564] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=39] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:05.725967] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=36] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:05.732431] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:05.743071] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=1] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:05.745135] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:02.772819] EDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=7223][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2968162, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2968152, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0xfb622cd 0xfb619cb 0xfb892d3 0x994150a 0x993eecd 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:02.772819] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=7223][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:05.748682] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=54] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:05.748778] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:05.749385] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:05.749419] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:05.749479] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=32] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:05.756697] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:05.769297] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=27] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:03.769504] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:05.777983] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=16] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:05:05.778291] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=9] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:05.778333] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=9] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:05.781271] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=29] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:05.783293] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:05.791358] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:05.794148] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=15] gc stale ls task succ [2025-08-14 09:05:05.800342] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=19] gc stale ls task succ [2025-08-14 09:05:05.899229] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=23] PNIO [ratelimit] time: 1755162305899227, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007613, add_bytes: 0 [2025-08-14 09:05:06.721704] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:06.726306] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:06.733493] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [398]OB_MALLOC COST TOO MUCH TIME, cost_time=1961169, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1961167], seq:[0] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=1957459, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1957457], seq:[0] [2025-08-14 09:05:06.742106] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:06.743691] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:06.744090] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:06.744349] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:06.746585] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:06.746629] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:06.750444] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:06.762441] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=34] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:05.790673] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=15] gc stale ls task succ [2025-08-14 09:05:06.769388] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=26] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:06.778101] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:05:06.778494] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=20] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:06.778542] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=10] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:06.781255] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:06.782260] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [663]OB_MALLOC COST TOO MUCH TIME, cost_time=2968017, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=2968010], seq:[0] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=2998109, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=69, WASH_BLOCK_END=29995, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=2968039], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:04.743693] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [663][T1001_StandbySc][T1001][YB42AC4D0007-00063C495C7A0380-0-0] [lt=3976154][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2998149, owner: ObLog, click_count: 5, time dist:[FORMAT_END=21, ObMalloc#start=2, ObMalloc#end=2998119, ALLOC_END=1, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:05.762307] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1981087, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=5, FORMAT_END=1981072, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [401]OB_MALLOC COST TOO MUCH TIME, cost_time=2964056, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1957472], seq:[0, 1] [2025-08-14 09:05:04.777915] EDIAG [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2964074, owner: ObLog, click_count: 5, time dist:[FORMAT_END=6, ObMalloc#start=0, ObMalloc#end=2964064, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a1f867 0x27a135c3 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:05.748751] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [339][EventTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1994722, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1994717, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:06.728609] EDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=2010375][errcode=-4015] LOGGER COST TOO MUCH TIME, cost: 1016957, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1016952, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x279e59be 0x971b1c1 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [398]OB_MALLOC COST TOO MUCH TIME, cost_time=2968719, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1961190], seq:[0, 1] [2025-08-14 09:05:04.751805] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2990837, owner: ObLog, click_count: 7, time dist:[FORMAT_END=7, ObMalloc#start=0, ObMalloc#end=22097, ObMalloc#start=1, ObMalloc#end=2968728, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:06.591172] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1154893, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1154888, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [477]OB_MALLOC COST TOO MUCH TIME, cost_time=2965455, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=2965429, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=1966618, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=17, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1966586, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=2965489, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2965466, WASH_BLOCK_END=23], seq:[0, 1, 2] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=1966645, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1966631, WASH_BLOCK_END=14], seq:[0, 1, 2] [2025-08-14 09:05:05.762307] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:07.746421] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1981087][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:07.746442] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:04.777915] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=14] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:05.748751] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [339][EventTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [861]OB_MALLOC COST TOO MUCH TIME, cost_time=979943, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=17, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=979894, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [861]OB_MALLOC COST TOO MUCH TIME, cost_time=979964, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=979955, WASH_BLOCK_END=9], seq:[0, 1, 2] [733]OB_MALLOC COST TOO MUCH TIME, cost_time=977694, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=977643, ALLOC_CHUNK_END=12], seq:[0, 1, 2, 3, 4, 5, 6, 7] [733]OB_MALLOC COST TOO MUCH TIME, cost_time=977718, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=977705, WASH_BLOCK_END=13], seq:[0, 1, 2] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=955766, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=955719, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [2025-08-14 09:05:04.778017] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [477][T1_StandbySchem][T1][YB42AC4D0007-00063C495B6A039C-0-0] [lt=13][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2965705, owner: ObLog, click_count: 6, time dist:[FORMAT_END=7, ObMalloc#start=0, ObMalloc#end=2965497, ObMalloc#start=2, ObMalloc#end=46, FREE_END=153], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [777]OB_MALLOC COST TOO MUCH TIME, cost_time=955805, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=955779, WASH_BLOCK_END=26], seq:[0, 1, 2] [2025-08-14 09:05:04.770152] EDIAG [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [854][T1002_ObXAHbWor][T1002][Y0-0000000000000000-0-0] [lt=12613][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2973599, owner: ObLog, click_count: 6, time dist:[FORMAT_END=2972550, ObMalloc#start=1, ObMalloc#end=90, ObMalloc#start=34, ObMalloc#end=37, FREE_END=887], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b64d8ba 0x1b64d154 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [397]OB_MALLOC COST TOO MUCH TIME, cost_time=955444, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=955406, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=955462, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=955454, WASH_BLOCK_END=7], seq:[0, 1, 2] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=952730, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=23, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=952667, ALLOC_CHUNK_END=12], seq:[0, 1, 2, 3, 4, 5, 6, 7] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=952747, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=952740, WASH_BLOCK_END=7], seq:[0, 1, 2] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=946624, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=17, ALLOC_CHUNK_END=946571, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=946641, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=946633, WASH_BLOCK_END=8], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=840416, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=840374, ALLOC_CHUNK_END=3], seq:[0, 1, 2, 3, 4, 5, 6, 7] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=840433, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=840426, WASH_BLOCK_END=7], seq:[0, 1, 2] [2025-08-14 09:05:07.746492] WDIAG ev_invoke_pending (ev.c:2136) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=2964074][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:07.746507] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=8] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:07.746528] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=9] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:06.728609] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=2010375][errcode=-4015] clock out of order(cur_ts=1755162303766073, cur_ts_=1755162303739753, delta=26320) [2025-08-14 09:05:04.751805] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:07.747205] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=0] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:07.747219] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:06.591172] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:06.906847] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=66] PNIO [ratelimit] time: 1755162306906845, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007618, add_bytes: 0 [2025-08-14 09:05:04.961320] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=11][errcode=-4018] LOGGER COST TOO MUCH TIME, cost: 2783291, owner: ObLog, click_count: 7, time dist:[FORMAT_END=1804901, ObMalloc#start=1, ObMalloc#end=977726, ObMalloc#start=19, ObMalloc#end=642, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3b4b 0x113d34aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:05.778384] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [776][T1002_StandbySc][T1002][YB42AC4D0007-00063C495DFA038E-0-0] [lt=12][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1967584, owner: ObLog, click_count: 6, time dist:[FORMAT_END=10, ObMalloc#start=0, ObMalloc#end=1966653, ObMalloc#start=1, ObMalloc#end=43, FREE_END=876], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:05.932523] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=21][errcode=-4018] LOGGER COST TOO MUCH TIME, cost: 1813816, owner: ObLog, click_count: 8, time dist:[FORMAT_END=832881, ObMalloc#start=0, ObMalloc#end=979971, ObMalloc#start=29, ObMalloc#end=39, ObMalloc#start=786, ObMalloc#end=31, FREE_END=79], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3b4b 0x113d34aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:05.976358] EDIAG try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1770361, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1749616, ObMalloc#start=1, ObMalloc#end=20627, ObMalloc#start=2, ObMalloc#end=36, FREE_END=79], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x26499505 0x97074a3 0x26498fa9 0x7f9973e081cf 0x7f9973a39dd3 [396]OB_MALLOC COST TOO MUCH TIME, cost_time=966714, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=966712], seq:[0] [2025-08-14 09:05:06.790725] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=974502][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1958392, owner: ObLog, click_count: 8, time dist:[ALLOC_BLOCK#start=7, ALLOC_BLOCK#end=3, FORMAT_END=1954046, ObMalloc#start=1, ObMalloc#end=3865, ObMalloc#start=19, ObMalloc#end=53, FREE_END=398], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [1275]OB_MALLOC COST TOO MUCH TIME, cost_time=1024265, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1024258], seq:[0] [1275]OB_MALLOC COST TOO MUCH TIME, cost_time=1024445, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=52, WASH_BLOCK_END=106, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1024282], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:07.746678] EDIAG [RS.COMPACTION] try_idle (ob_major_merge_scheduler.cpp:226) [1275][T1001_MergeSche][T1001][YB42AC4D0007-00063C4963E99CBE-0-0] [lt=12286][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1024473, owner: ObLog, click_count: 5, time dist:[FORMAT_END=14, ObMalloc#start=1, ObMalloc#end=1024456, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1125d502 0x1125cf81 0x1125a4c3 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:04.961320] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=11][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:05:07.745177] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:07.745916] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:07.755265] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:07.914461] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=840483] PNIO [ratelimit] time: 1755162307914459, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007614, add_bytes: 0 [2025-08-14 09:05:08.733293] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9470] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:08.744140] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3772] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:08.745123] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3118] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:08.747249] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=7] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:08.757292] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:08.748476] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [878][T1002_ArcSender][T1002][YB42AC4D0007-00063C495EA99C16-0-0] [lt=730346] ObArchiveSender is running(thread_index=4) [2025-08-14 09:05:03.802253] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=198][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4963798, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=2, FORMAT_END=4963783, ObMalloc#start=0, ObMalloc#end=3, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:03.769504] EDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 2001836, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2001824, ObMalloc#start=1, ObMalloc#end=4, ALLOC_END=1, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x96ecc48 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:04.899494] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3868403, owner: ObLog, click_count: 5, time dist:[FORMAT_END=3868396, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:03.802253] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=198] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:08.773590] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4963798][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:08.773624] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:08.773638] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:04.899494] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:04.728947] EDIAG [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=22][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4039851, owner: ObLog, click_count: 5, time dist:[FORMAT_END=4039847, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x12377491 0x1236e8e9 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [396]OB_MALLOC COST TOO MUCH TIME, cost_time=1031692, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=966780], seq:[0, 1] [2025-08-14 09:05:07.747011] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1031722, owner: ObLog, click_count: 7, time dist:[FORMAT_END=7, ObMalloc#start=1, ObMalloc#end=1031711, ALLOC_END=2, APPEND_END=0, ALLOC_BLOCK#start=16, ALLOC_BLOCK#end=5], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:04.728947] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=22] finish to resize log external storage handler(current_ts=1755162302769670, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-08-14 09:05:07.747011] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:08.780384] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1031722][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:08.780413] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:08.781452] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=340] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:06.737385] EDIAG [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2017454, owner: ObLog, click_count: 6, time dist:[FORMAT_END=2017240, ObMalloc#start=1, ObMalloc#end=40, ObMalloc#start=19, ObMalloc#end=37, FREE_END=117], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x260b4cc1 0x260b9960 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:08.743244] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1002196, owner: ObLog, click_count: 8, time dist:[FORMAT_END=1001451, ObMalloc#start=1, ObMalloc#end=80, ObMalloc#start=23, ObMalloc#end=50, ObMalloc#start=318, ObMalloc#end=53, FREE_END=220], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:08.752298] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1000691, owner: ObLog, click_count: 8, time dist:[FORMAT_END=992778, ObMalloc#start=0, ObMalloc#end=47, ObMalloc#start=22, ObMalloc#end=46, ObMalloc#start=91, ObMalloc#end=50, FREE_END=7657], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:08.781472] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1968521, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1968519, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [864]OB_MALLOC COST TOO MUCH TIME, cost_time=1969929, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1969921], seq:[0] [864]OB_MALLOC COST TOO MUCH TIME, cost_time=1970098, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=43, WASH_BLOCK_END=94, ALLOC_BLOCK#start=7, ALLOC_BLOCK#end=1969949], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:08.781280] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=177][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1970122, owner: ObLog, click_count: 5, time dist:[FORMAT_END=13, ObMalloc#start=0, ObMalloc#end=1970108, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:08.788069] EDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] LOGGER COST TOO MUCH TIME, cost: 1963608, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1963607, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x13b06ffb 0x958a39e 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:08.781781] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=163] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:08.781856] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=435] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:08.803602] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=206] gc stale ls task succ [2025-08-14 09:05:09.733261] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=978] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:09.733298] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:09.744592] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=7890] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:08.773782] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=3868403][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:08.774479] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:09.746740] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:08.772752] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=2003695] blacklist_loop exec finished(cost_time=17, is_enabled=true, send_cnt=0) [2025-08-14 09:05:09.756826] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=30] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:09.761086] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=665] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:09.772670] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=1143] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:09.781905] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=20] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:09.784296] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=8] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:09.786313] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:09.806257] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=12] gc stale ls task succ [2025-08-14 09:05:10.727502] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:10.735424] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:10.746261] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:08.781472] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:10.753015] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1968521][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:08.781280] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=177] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:05:08.788069] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:10.756106] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:08.780957] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1974219, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1974213, ObMalloc#start=0, ObMalloc#end=2, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:08.791889] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1963386, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1963380, ObMalloc#start=0, ObMalloc#end=2, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:10.758080] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=365] gc stale ls task succ [2025-08-14 09:05:08.780957] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:10.758190] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1974219][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:08.791889] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:10.758266] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1963386][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:10.758278] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task use too much time(end=1755162310758278, start=1755162308791887, delta=1966391) [2025-08-14 09:05:10.743256] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=93] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:10.758314] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=15034][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:10.745880] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=344] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:10.758588] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12687][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:09.795115] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=186] gc stale ls task succ [2025-08-14 09:05:10.760959] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:09.754896] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3234][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1006792, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:09.754896] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3234] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:10.763246] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1006792][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:10.763265] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:09.745203] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=1004528][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1018332, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1018326, ObMalloc#start=0, ObMalloc#end=2, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:09.745203] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=1004528] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:05:10.727483] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:10.721950] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=163] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:10.753572] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [339][EventTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:10.722599] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=2009857] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:09.755835] EDIAG [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=2982541][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1009711, owner: ObLog, click_count: 6, time dist:[FORMAT_END=999712, ObMalloc#start=1, ObMalloc#end=75, ObMalloc#start=23, ObMalloc#end=58, FREE_END=9841], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x18bdb084 0x18bda6b1 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:10.767359] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=974332][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:10.762077] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:233) [529][T1_RFLWorker][T1][YB42AC4D0007-00063C495B199C16-0-0] [lt=15384] ObRemoteFetchWorker is running(thread_index=0) [2025-08-14 09:05:09.745287] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=1958859][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1021491, , BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:09.745287] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=1958859] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:05:10.772733] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=17] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:10.775263] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=204] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:10.781455] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=220] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:10.781491] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=209] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:10.782005] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=13] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:10.784393] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:10.790450] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:10.795178] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=964193] gc stale ls task succ [2025-08-14 09:05:10.800158] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:10.804581] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:10.806327] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=13] gc stale ls task succ [2025-08-14 09:05:10.848042] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:10.855512] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=198] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:10.937321] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=220] PNIO [ratelimit] time: 1755162310937318, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007626, add_bytes: 0 [2025-08-14 09:05:11.011263] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:11.262912] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=265][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:11.724713] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3052] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:11.726980] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:10.754683] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=1963608][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:02.726798] EDIAG [RS] get_valid_sts_after (ob_tenant_info_loader.cpp:454) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 8026122, owner: ObLog, click_count: 5, time dist:[FORMAT_END=8026096, ObMalloc#start=1, ObMalloc#end=18, ALLOC_END=0, APPEND_END=7], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1001d942 0x996d843 0x996d15a 0x996cfd8 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:10.721932] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=2968162] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:05:10.721932] EDIAG [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=2968162][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1007011, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1007000, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=8], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0xfb90f6f 0xfb90ac1 0x9941516 0x993eecd 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:10.754745] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:05:11.732400] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:10.767386] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=4039851] snapshot_traffic_info once [2025-08-14 09:05:11.743345] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:11.746126] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=211] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:11.746163] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=271] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:11.746187] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=187] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:11.747279] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:11.749274] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:11.749302] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=204] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:08.714330] EDIAG [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3032598, owner: ObLog, click_count: 5, time dist:[FORMAT_END=3032589, ObMalloc#start=1, ObMalloc#end=4, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1af3d850 0x1af3c294 0x20071394 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:11.749758] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=188] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:08.714330] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] [COMPACTION DAG STATUS] (start_time=1755162293735896, end_time=1755162306737476, dag_status={}) [718]OB_MALLOC COST TOO MUCH TIME, cost_time=988749, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=988748], seq:[0] [2025-08-14 09:05:10.758361] EDIAG [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [726][T1001_ObXAHbWor][T1001][Y0-0000000000000000-0-0] [lt=2987714][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1005378, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=3, FORMAT_END=1005371, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b64d8ba 0x1b64d154 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [718]OB_MALLOC COST TOO MUCH TIME, cost_time=996728, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=97, WASH_BLOCK_END=71, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=988763], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:09.746172] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=1008121][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2017052, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1020275, ObMalloc#start=0, ObMalloc#end=996760, ALLOC_END=4, APPEND_END=13], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:00.745276] EDIAG [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=38648][errcode=0] LOGGER COST TOO MUCH TIME, cost: 11005565, owner: ObLog, click_count: 5, time dist:[FORMAT_END=10991407, ObMalloc#start=1, ObMalloc#end=14155, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x9c6174d 0xfd347d6 0xfd344b7 0xfd34254 0x1c029398 0x97a1ad0 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:11.754028] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:11.754502] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=1970122] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:05:11.731798] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=37593] temporary file module use no blocks [2025-08-14 09:05:10.758361] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [726][T1001_ObXAHbWor][T1001][Y0-0000000000000000-0-0] [lt=2987714] XA scheduler heartbeat task statistics(avg_time=2) [2025-08-14 09:05:09.746172] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=1008121] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:05:00.745276] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=38648] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=751990218, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=253247, succ_cluster_heartbeat_count=2213, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161000606550000, v:0}, local_cluster_delta=1299122184, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:05:09.755487] EDIAG [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [412][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=998413][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2011423, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a5bbf2 0x94da542 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:11.758952] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=2001836][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:11.746253] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:11.769910] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23620][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:09.755487] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [412][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=998413] [sql nio session](*s={this:0x7f98f733a030, session_id:3221590255, trace_id:YB42AC4D0007-00063C49651A3232-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:257}, err:0, last_decode_time:1755161952736711, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1716, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:05:11.770012] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=975560] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:11.770062] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:11.770103] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:11.770139] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:05:11.770155] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:11.770169] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:10.775570] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=364][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:11.766663] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=8026122] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:05:11.770331] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=3660] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:05:11.770350] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:11.773970] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=30] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:11.781451] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:11.782075] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:11.782507] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:11.784487] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:11.791249] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:11.795256] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=19] gc stale ls task succ [2025-08-14 09:05:11.806388] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=19] gc stale ls task succ [2025-08-14 09:05:11.806937] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:11.810273] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] [sql nio session](*s={this:0x7f98f72d0030, session_id:3221589415, trace_id:YB42AC4D0007-00063C4965FA43DC-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:245}, err:0, last_decode_time:1755161751714366, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1662, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:05:11.810302] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=29] [sql nio session](*s={this:0x7f98f72d5030, session_id:3221590200, trace_id:YB42AC4D0007-00063C4965FA43E0-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:248}, err:0, last_decode_time:1755161776777167, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1593, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:05:11.857526] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=18] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:11.944924] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=23] PNIO [ratelimit] time: 1755162311944922, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007604, add_bytes: 0 [2025-08-14 09:05:12.016032] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:12.230373] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:12.264630] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=40][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:11.775261] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:12.723644] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=946948][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:12.723713] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=29] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:11.763148] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=27173] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:12.726634] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:11.763137] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=27108] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:11.764418] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=26876] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:12.735986] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:12.737947] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [807][T1002_TimerWK3_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=1] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):258, sequence:1}) [2025-08-14 09:05:12.738001] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:97) [807][T1002_TimerWK3_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=0] BlockGCTimerTask success(ret=0, cost_time_us=966048, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.77.0.7:2882", log_dir:"/root/ob/observer/store/clog/tenant_1002", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):258, sequence:1}, log_alloc_mgr_:{flying_log_task:1, flying_meta_task:2}}) [2025-08-14 09:05:12.745072] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:12.745110] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:12.745136] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:12.747449] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:12.747575] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=30] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:12.748266] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:12.748313] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=11] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:12.750270] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:12.754609] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=14] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:05:12.747492] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:12.756545] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=333] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:12.734249] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=967067] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:12.758213] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=2526] gc stale ls task succ [2025-08-14 08:59:56.746665] EDIAG [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [14961][T1_TimerWK8_LSMetaCh][T1][YB42AC4D0007-00063C4979D99C24-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1027652, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2715, ObMalloc#start=1, ObMalloc#end=1024935, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1eecf890 0x1eecde9d 0x1eeb5094 0x1eed9495 0x1eed9090 0x99deeb9 0x12d3af34 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:12.761262] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4533] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:12.737905] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=968330] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:12.762743] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=24800] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623206181464, flush_io_finished_ret_:0, flush_io_finished_round_:26366, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:12.760646] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=3032598] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:12.763986] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=3334] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:12.763998] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:12.764008] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:12.764016] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:12.764023] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:12.764030] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:12.764039] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:12.764046] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:12.764445] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:05:12.764454] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:05:12.770923] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4076] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:12.758686] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=22593] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623210224033, flush_io_finished_ret_:0, flush_io_finished_round_:26390, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:6, normal_idle_loop_cnt_:6, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:12.757874] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=23588] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623210185805, flush_io_finished_ret_:0, flush_io_finished_round_:26351, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:6, normal_idle_loop_cnt_:6, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:12.773772] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=47] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:12.774061] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=23] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:12.781407] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:12.782182] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=14] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:12.782536] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:12.784593] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=17] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:12.791165] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:12.795316] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=19] gc stale ls task succ [2025-08-14 09:05:12.806452] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=21] gc stale ls task succ [2025-08-14 09:05:12.858160] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=19] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:12.952541] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=26] PNIO [ratelimit] time: 1755162312952539, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007617, add_bytes: 0 [2025-08-14 09:05:12.993465] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:13.024569] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:13.093250] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:13.235038] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:13.265241] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:13.636524] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=356][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:13.725049] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=68] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:13.729138] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=83] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:12.864052] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:05:13.732359] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=866951] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:13.732440] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=50] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:13.732486] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:13.732530] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:05:13.732550] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:13.732562] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:13.736281] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:13.740001] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:13.743993] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:13.744026] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:13.744711] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:13.745536] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:13.745902] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8493] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:13.747665] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=1018332] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:05:13.748288] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=13] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:13.748328] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:13.749916] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=2017052] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:05:13.749963] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=1021491] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:05:13.734899] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [807][T1002_TimerWK3_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=0] ====== check clog disk timer task ====== [2025-08-14 09:05:13.752780] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [807][T1002_TimerWK3_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=17843] get_disk_usage(ret=0, capacity(MB):=24883, used(MB):=8642) [2025-08-14 09:05:13.753527] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:13.755421] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=21] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:05:13.755645] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:13.758261] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=14] gc stale ls task succ [2025-08-14 09:05:13.759316] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=2783291][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:05:13.767784] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:13.767856] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:13.768257] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=1813816][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:05:13.772519] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4076] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:13.774142] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=16] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:13.779254] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:13.781255] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:13.781491] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:13.782256] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:13.784747] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=26] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:13.790276] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:13.795413] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=21] gc stale ls task succ [2025-08-14 09:05:13.802002] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=47][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:13.806513] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=21] gc stale ls task succ [2025-08-14 09:05:13.849285] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=993202][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:13.858257] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=48][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:13.858981] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=19] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:13.937303] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:05:13.937354] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:13.937398] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:13.937442] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:13.937485] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:05:13.937496] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:13.937507] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:13.960169] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=50] PNIO [ratelimit] time: 1755162313960167, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007628, add_bytes: 0 [2025-08-14 09:05:13.970317] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:13.999493] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:14.025362] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:14.098063] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:14.237064] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:14.272152] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:14.720280] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:13.761262] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:14.722436] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=959720][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:14.728033] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:14.736362] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=76] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:14.739955] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:14.743998] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:14.745523] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:14.746759] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:14.748355] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:14.749277] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=17] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:14.754271] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [396]OB_MALLOC COST TOO MUCH TIME, cost_time=974404, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=13571], seq:[0, 1] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=977240, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=16355], seq:[0, 1] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=1007904, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1007902], seq:[0] [2025-08-14 09:01:16.759879] [996][T1002_L0_G0][T1002][YB42AC4D0007-00063C4962EA8FFF-0-0] LOGGER COST TOO MUCH TIME, cost: 1017083, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1014329, ObMalloc#start=1, ObMalloc#end=2749, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x261b569c 0x261b52ae 0x93badc4 0x927f06e 0x926c154 0x12c5a0c9 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:32.764637] EDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [337][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1008120, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1008118, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x974f472 0x974e159 0x974e084 0x948f9df 0x97257c0 0x124e2fb7 0x97aa07c 0x124eac91 0x97a9944 0x261395f9 0x1f900517 0x1f8fe2bf 0x9a5f486 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 08:59:58.772911] EDIAG [SERVER] force_close (ob_inner_sql_result.cpp:199) [779][T1002_PLSSer][T1001][YB42AC4D0007-00063C4964B9FE3E-0-0] [lt=22238][errcode=-4283] LOGGER COST TOO MUCH TIME, cost: 2958469, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=5149, ALLOC_BLOCK#end=68, FORMAT_END=2953252, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x124fe24f 0x124dfe49 0x96a5a4d 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x966116d 0x9660b08 0x966002f 0x10041219 0x10040975 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [658]OB_MALLOC COST TOO MUCH TIME, cost_time=1040201, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=53, WASH_BLOCK_END=124, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1007924], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:13.871899] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1894866, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=8, ALLOC_BLOCK#end=3, FORMAT_END=854628, ObMalloc#start=1, ObMalloc#end=1040224, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [384]OB_MALLOC COST TOO MUCH TIME, cost_time=1020395, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1020388], seq:[0] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=1020418, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1020413], seq:[0, 1] [2025-08-14 09:05:14.755551] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1020435, owner: ObLog, click_count: 5, time dist:[FORMAT_END=8, ObMalloc#start=0, ObMalloc#end=1020425, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [864]OB_MALLOC COST TOO MUCH TIME, cost_time=1020475, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1020466, ALLOC_CHUNK_END=5], seq:[0, 1] [864]OB_MALLOC COST TOO MUCH TIME, cost_time=1020506, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1020482, WASH_BLOCK_END=24], seq:[0, 1, 2] [807]OB_MALLOC COST TOO MUCH TIME, cost_time=1019536, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1019527, ALLOC_CHUNK_END=4], seq:[0, 1] [807]OB_MALLOC COST TOO MUCH TIME, cost_time=1019550, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1019542, WASH_BLOCK_END=8], seq:[0, 1, 2] [2025-08-14 09:05:13.752840] EDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 2021900, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1002318, ObMalloc#start=1, ObMalloc#end=1019556, ALLOC_END=24, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x96ecc48 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:14.755591] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1020910, owner: ObLog, click_count: 6, time dist:[FORMAT_END=11, ObMalloc#start=0, ObMalloc#end=1020512, ObMalloc#start=24, ObMalloc#end=45, FREE_END=318], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:14.746807] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:15.722728] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=974445][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:14.744041] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:15.722775] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=977255][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:15.722981] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=1016957][errcode=-4015] clock out of order(cur_ts=1755162314748763, cur_ts_=1755162314723060, delta=25703) [2025-08-14 09:05:15.747329] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:15.747681] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:15.749881] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=18] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:13.871899] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:14.755551] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:15.777555] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1020435][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:15.777573] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:13.752840] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=1018878, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1018872], seq:[0] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=1018888, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1018887], seq:[0, 1] [2025-08-14 09:05:14.758882] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=24][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1018912, owner: ObLog, click_count: 5, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=1018895, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [477]OB_MALLOC COST TOO MUCH TIME, cost_time=1018898, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1018893], seq:[0] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=1018906, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1018905], seq:[0, 1] [2025-08-14 09:05:14.758934] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [477][T1_StandbySchem][T1][YB42AC4D0007-00063C495B6A03A3-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1018927, owner: ObLog, click_count: 5, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=1018912, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [395]OB_MALLOC COST TOO MUCH TIME, cost_time=1017312, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=17, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1017277], seq:[0, 1, 2, 3, 4] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=1017320, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1017319], seq:[0, 1] [2025-08-14 09:05:14.760566] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=53][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1017345, owner: ObLog, click_count: 5, time dist:[FORMAT_END=15, ObMalloc#start=1, ObMalloc#end=1017326, ALLOC_END=0, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [776]OB_MALLOC COST TOO MUCH TIME, cost_time=1011669, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=1011632], seq:[0, 1, 2, 3, 4] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=1011679, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1011678], seq:[0, 1] [2025-08-14 09:05:14.766286] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [776][T1002_StandbySc][T1002][YB42AC4D0007-00063C495DFA0395-0-0] [lt=20][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1011703, owner: ObLog, click_count: 5, time dist:[FORMAT_END=15, ObMalloc#start=1, ObMalloc#end=1011684, ALLOC_END=0, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [663]OB_MALLOC COST TOO MUCH TIME, cost_time=1011663, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1011642], seq:[0, 1, 2, 3, 4] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=1011672, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1011671], seq:[0, 1] [2025-08-14 09:05:14.766347] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [663][T1001_StandbySc][T1001][YB42AC4D0007-00063C495C7A0387-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1011690, owner: ObLog, click_count: 5, time dist:[FORMAT_END=7, ObMalloc#start=0, ObMalloc#end=1011678, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [800]OB_MALLOC COST TOO MUCH TIME, cost_time=1009785, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=18, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1009751], seq:[0, 1, 2, 3, 4] [800]OB_MALLOC COST TOO MUCH TIME, cost_time=1009794, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1009793], seq:[0, 1] [2025-08-14 09:05:14.768280] EDIAG [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1009820, owner: ObLog, click_count: 5, time dist:[FORMAT_END=15, ObMalloc#start=1, ObMalloc#end=1009800, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1af3d850 0x1af3c294 0x20071394 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [332]OB_MALLOC COST TOO MUCH TIME, cost_time=1003896, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=1003872], seq:[0, 1, 2, 3, 4] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=1003906, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1003905], seq:[0, 1] [2025-08-14 09:05:14.774224] EDIAG do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1003939, owner: ObLog, click_count: 5, time dist:[FORMAT_END=23, ObMalloc#start=0, ObMalloc#end=1003912, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x202c530b 0x202c6888 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [394]OB_MALLOC COST TOO MUCH TIME, cost_time=1002901, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1002881], seq:[0, 1, 2, 3, 4] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=1002911, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1002910], seq:[0, 1] [2025-08-14 09:05:14.775283] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1002928, owner: ObLog, click_count: 5, time dist:[FORMAT_END=8, ObMalloc#start=0, ObMalloc#end=1002916, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [856]OB_MALLOC COST TOO MUCH TIME, cost_time=1001499, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=1001478], seq:[0, 1, 2, 3, 4] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=1001510, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1001508], seq:[0, 1] [2025-08-14 09:05:14.776746] EDIAG [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=3][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1001540, owner: ObLog, click_count: 5, time dist:[FORMAT_END=21, ObMalloc#start=0, ObMalloc#end=1001515, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1b52e1b7 0x996d441 0x996cfd8 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [382]OB_MALLOC COST TOO MUCH TIME, cost_time=997099, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=997075], seq:[0, 1, 2, 3, 4] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=997109, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=997108], seq:[0, 1] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=996057, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=996032, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=996094, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=996066, WASH_BLOCK_END=28], seq:[0, 1, 2] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=996102, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=996078, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=996124, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=996112, WASH_BLOCK_END=12], seq:[0, 1, 2] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=993780, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=993754, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=993798, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=993789, WASH_BLOCK_END=9], seq:[0, 1, 2] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=983204, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=983180, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=983231, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=983220, WASH_BLOCK_END=10], seq:[0, 1, 2] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=972235, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=972207, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=972253, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=972243, WASH_BLOCK_END=10], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=919306, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=919277, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=919324, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=919315, WASH_BLOCK_END=9], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=811455, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=811418, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=811475, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=811466, WASH_BLOCK_END=9], seq:[0, 1, 2] [2025-08-14 09:05:14.758882] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=24] gc stale ls task succ [2025-08-14 09:05:15.779361] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=1018912] gc stale ls task succ [2025-08-14 09:05:14.760566] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=53] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:15.779479] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:14.768280] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10] [COMPACTION DAG STATUS] (start_time=1755162306737476, end_time=1755162314768278, dag_status={}) [2025-08-14 09:05:14.774224] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=17] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:14.775283] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:15.779745] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1002928][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:15.779762] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:14.776746] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=3] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:05:15.779827] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1001540] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:05:14.781225] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:15.779876] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=997142][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:15.779895] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:14.782354] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:14.784857] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:14.795477] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=18] gc stale ls task succ [2025-08-14 09:05:14.806587] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=18] gc stale ls task succ [2025-08-14 09:05:15.721396] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=40] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:15.780655] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=59167][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:15.725734] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:15.780749] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=54931][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:15.780791] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=996869][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:15.780814] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:14.791434] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:15.780879] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=987950][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:15.737007] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=33] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:15.780905] WDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=43831][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:15.737558] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12604] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [403]OB_MALLOC COST TOO MUCH TIME, cost_time=1965887, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1965871], seq:[0] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=2004179, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=38179, WASH_BLOCK_END=68, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1965927], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:16.567680] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1181080, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1181079, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [389]OB_MALLOC COST TOO MUCH TIME, cost_time=1996498, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1996460, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [389]OB_MALLOC COST TOO MUCH TIME, cost_time=1996558, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1996531, WASH_BLOCK_END=27], seq:[0, 1, 2] [376]OB_MALLOC COST TOO MUCH TIME, cost_time=1996862, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1996815, ALLOC_CHUNK_END=19], seq:[0, 1, 2, 3, 4, 5, 6, 7] [376]OB_MALLOC COST TOO MUCH TIME, cost_time=1996909, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1996898, WASH_BLOCK_END=11], seq:[0, 1, 2] [528]OB_MALLOC COST TOO MUCH TIME, cost_time=1978735, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1978684, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [528]OB_MALLOC COST TOO MUCH TIME, cost_time=1978760, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1978749, WASH_BLOCK_END=11], seq:[0, 1, 2] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=2005959, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=2005916, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=2006001, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2005979, WASH_BLOCK_END=22], seq:[0, 1, 2] [2025-08-14 09:05:15.743864] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2006045, owner: ObLog, click_count: 5, time dist:[FORMAT_END=9, ObMalloc#start=0, ObMalloc#end=2006010, ALLOC_END=25, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:15.738347] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13444] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:15.739666] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:15.781292] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=41562][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:15.880886] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=919713] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:16.655249] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=72][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:16.721975] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:16.740177] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:34.747992] WDIAG [STORAGE] runTimerTask (ob_tenant_tablet_stat_mgr.cpp:1206) [14905][T1001_TimerWK8_TabletStatRpt][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] tablet streams not refresh too long(interval_step=4) [2025-08-14 09:05:16.765518] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [726][T1001_ObXAHbWor][T1001][Y0-0000000000000000-0-0] [lt=1005378] XA scheduler heartbeat task statistics(avg_time=2) [2025-08-14 09:05:16.765886] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:16.766946] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:16.770427] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:16.776259] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:16.778158] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=1020910] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:05:16.781393] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:16.783298] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=35] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:16.803784] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:16.881635] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=9] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:16.995292] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:17.274059] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:17.721265] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:17.740257] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:15.743083] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:16.567680] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:15.743864] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:17.752962] WDIAG ev_invoke_pending (ev.c:2136) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2006045][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:17.752991] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:17.753004] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [397]OB_MALLOC COST TOO MUCH TIME, cost_time=1960862, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=17, ALLOC_CHUNK_END=1960816], seq:[0, 1, 2, 3, 4, 5, 6] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=1960876, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1960875], seq:[0, 1] [2025-08-14 09:05:15.789929] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1960898, owner: ObLog, click_count: 5, time dist:[FORMAT_END=9, ObMalloc#start=0, ObMalloc#end=1960883, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [730]OB_MALLOC COST TOO MUCH TIME, cost_time=1954637, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1954599], seq:[0, 1, 2, 3, 4, 5, 6] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=1954650, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1954649], seq:[0, 1] [2025-08-14 09:05:15.796255] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=983318][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1954674, owner: ObLog, click_count: 5, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=1954655, ALLOC_END=2, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [858]OB_MALLOC COST TOO MUCH TIME, cost_time=1944334, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1944298], seq:[0, 1, 2, 3, 4, 5, 6] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=1944346, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1944344], seq:[0, 1] [2025-08-14 09:05:15.806650] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=972292][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1944365, owner: ObLog, click_count: 5, time dist:[FORMAT_END=10, ObMalloc#start=0, ObMalloc#end=1944351, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [321]OB_MALLOC COST TOO MUCH TIME, cost_time=1775701, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1775663], seq:[0, 1, 2, 3, 4, 5, 6] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=1775715, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1775714], seq:[0, 1] [2025-08-14 09:05:15.975572] EDIAG eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=811855][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1775760, owner: ObLog, click_count: 5, time dist:[FORMAT_END=34, ObMalloc#start=1, ObMalloc#end=1775721, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x27a8a8c8 0x97b7e1a 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [387]OB_MALLOC COST TOO MUCH TIME, cost_time=1026418, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1026379], seq:[0, 1, 2, 3, 4, 5, 6] [387]OB_MALLOC COST TOO MUCH TIME, cost_time=1026430, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1026429], seq:[0, 1] [2025-08-14 09:05:16.726087] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1026454, owner: ObLog, click_count: 5, time dist:[FORMAT_END=12, ObMalloc#start=2, ObMalloc#end=1026436, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:15.789929] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:17.753817] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1960898][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:17.753839] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:15.796255] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=983318] gc stale ls task succ [2025-08-14 09:05:17.753912] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=1954674] gc stale ls task succ [2025-08-14 09:05:15.806650] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=972292] gc stale ls task succ [2025-08-14 09:05:17.753974] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=1944365] gc stale ls task succ [2025-08-14 09:05:15.975572] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=811855] PNIO [ratelimit] time: 1755162315975571, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007787, add_bytes: 0 [2025-08-14 09:05:17.754046] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=1775760] PNIO [ratelimit] time: 1755162317754045, bytes: 0, bw: 0.000000 MB/s, add_ts: 1778474, add_bytes: 0 [2025-08-14 09:05:16.726087] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:17.754113] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1026454][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:17.754132] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [392]OB_MALLOC COST TOO MUCH TIME, cost_time=1019716, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1019684], seq:[0, 1, 2, 3, 4, 5, 6] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=1019743, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1019742], seq:[0, 1] [2025-08-14 09:05:16.732905] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1019766, owner: ObLog, click_count: 5, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=1019749, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [385]OB_MALLOC COST TOO MUCH TIME, cost_time=1007384, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=1007347], seq:[0, 1, 2, 3, 4, 5, 6] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=1007394, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1007393], seq:[0, 1] [2025-08-14 09:05:16.745366] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1007420, owner: ObLog, click_count: 5, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=1007406, ALLOC_END=0, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [396]OB_MALLOC COST TOO MUCH TIME, cost_time=1006265, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1006228], seq:[0, 1, 2, 3, 4, 5, 6] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=1006274, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1006274], seq:[0, 1] [2025-08-14 09:05:16.746564] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1006291, owner: ObLog, click_count: 5, time dist:[FORMAT_END=8, ObMalloc#start=0, ObMalloc#end=1006279, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [401]OB_MALLOC COST TOO MUCH TIME, cost_time=1005685, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=15, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1005647], seq:[0, 1, 2, 3, 4, 5, 6] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=1005695, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1005695], seq:[0, 1] [2025-08-14 09:05:16.747250] EDIAG [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1005714, owner: ObLog, click_count: 5, time dist:[FORMAT_END=10, ObMalloc#start=0, ObMalloc#end=1005700, ALLOC_END=2, APPEND_END=2], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a1f867 0x27a135c3 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [384]OB_MALLOC COST TOO MUCH TIME, cost_time=997360, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=997328], seq:[0, 1, 2, 3, 4, 5, 6] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=997369, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=997369], seq:[0, 1] [822]OB_MALLOC COST TOO MUCH TIME, cost_time=992852, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=16, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=992812], seq:[0, 1, 2, 3, 4, 5, 6] [822]OB_MALLOC COST TOO MUCH TIME, cost_time=992870, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=992869], seq:[0, 1] [2025-08-14 09:05:12.734907] EDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1007011][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5012142, owner: ObLog, click_count: 5, time dist:[FORMAT_END=4019241, ObMalloc#start=2, ObMalloc#end=992885, ALLOC_END=9, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0xfc0e73d 0x993f560 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [660]OB_MALLOC COST TOO MUCH TIME, cost_time=980374, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=980326], seq:[0, 1, 2, 3, 4, 5, 6] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=980407, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=980406], seq:[0, 1] [2025-08-14 09:05:16.732905] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:17.754246] WDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1019766][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:17.754264] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:16.745366] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:17.754326] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1007420][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:17.754341] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:16.746564] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:17.754400] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1006291][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:17.754415] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:16.747250] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=11] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:17.754515] WDIAG ev_invoke_pending (ev.c:2136) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=1005714][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:17.754529] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=7] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:16.755670] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:12.734907] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1007011] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:8607757, max_lsn:1604090331, max_scn:{val:1755161756730847001, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:1556737504}, reuse_lsn:{lsn:1604090331}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:1562147291}}, last_submit_lsn:{lsn:1604090203}, last_submit_end_lsn:{lsn:1604090331}, last_submit_log_id:8607757, last_submit_log_pid:50, max_flushed_lsn:{lsn:1604090203}, max_flushed_end_lsn:{lsn:1604090331}, max_flushed_log_pid:50, committed_end_lsn:{lsn:1604090090}, last_slide_log_id:8607755, last_slide_scn:{val:1755161749741193000, v:0}, last_slide_lsn:{lsn:1604089968}, last_slide_end_lsn:{lsn:1604090090}, last_slide_log_pid:50, last_slide_log_accum_checksum:1446576759, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f990bfe4530}, [StateMgr]={this:0x7f990bfe89b0, self:"172.77.0.7:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"PENDING", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:51}, leader:"0.0.0.0:0", leader_epoch:-1, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:1604090331}, scan_disk_log_finished:true, last_check_start_id:8607756, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161901828039, palf_role_change_cb:0x7f990bfef470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:50, prev:{version:1, config:{config_version:{proposal_id:49, config_seq:45}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:50, config_seq:46}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_mode_pid:50}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_end_lsn:{lsn:1604088821}, prev_mode_pid:50}, reconfig_barrier:{prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_end_lsn:{lsn:1604088821}, prev_mode_pid:50}, persistent_config_version:{proposal_id:50, config_seq:46}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755161738854289, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755161738854289, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f990bfe5c70}, [ModeMgr]={palf_id:1, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:50, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:1543409664}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:51}, log_config_meta:{version:3, proposal_id:50, prev:{version:1, config:{config_version:{proposal_id:49, config_seq:45}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:50, config_seq:46}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_mode_pid:50}, log_snapshot_meta:{version:2, base_lsn:{lsn:1543409664}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:712704}, readable_log_tail:{lsn:712704}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:1610514432}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/meta", dir_fd:225, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66392064, block_header_serialize_buf_:0x7f990bfeabb8, flashback_version:0}, log_storage:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1755161901828095, this:0x7f990bfe90f0}, [Reconfirm]={palf_id:1, self:"172.77.0.7:2882", state:"RECONFIRM_MODE_META", new_proposal_id:51, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"172.77.0.7:2882", majority_max_accept_pid:50, majority_max_lsn:{lsn:1604090331}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:1755161901828373, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f990bfe8b30}) [2025-08-14 09:05:17.754664] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=5012142] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161974728707, last_flushed_end_lsn:{lsn:1604090331}}]) [2025-08-14 09:05:16.773336] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=990791, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=990760], seq:[0, 1, 2, 3, 4, 5, 6] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=990806, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=990805], seq:[0, 1] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=974854, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=974826], seq:[0, 1, 2, 3, 4, 5, 6] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=974864, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=974863], seq:[0, 1] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=974677, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=974642], seq:[0, 1, 2, 3, 4, 5, 6] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=974686, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=974685], seq:[0, 1] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=974984, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=974951], seq:[0, 1, 2, 3, 4, 5, 6] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=974353, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=974324], seq:[0, 1, 2, 3, 4, 5, 6] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=975005, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=975002], seq:[0, 1] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=974366, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=974366], seq:[0, 1] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=973204, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=973169], seq:[0, 1, 2, 3, 4, 5, 6] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=973216, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=973215], seq:[0, 1] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=973097, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=973068], seq:[0, 1, 2, 3, 4, 5, 6] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=973107, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=973105], seq:[0, 1] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=972651, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=972623], seq:[0, 1, 2, 3, 4, 5, 6] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=972662, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=972662], seq:[0, 1] [2025-08-14 09:05:15.742951] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14746][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2010145, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:14.757530] EDIAG [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=964310][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2994155, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1012045, ObMalloc#start=1, ObMalloc#end=1978767, ObMalloc#start=28, ObMalloc#end=102, FREE_END=3212], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x12377491 0x1236e8e9 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [574]OB_MALLOC COST TOO MUCH TIME, cost_time=973438, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=973405], seq:[0, 1, 2, 3, 4, 5, 6] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=973453, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=973451], seq:[0, 1] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=970680, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=970642, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=970721, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=970689, WASH_BLOCK_END=32], seq:[0, 1, 2] [2025-08-14 09:05:14.758921] EDIAG [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=982153][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2992923, owner: ObLog, click_count: 6, time dist:[FORMAT_END=992285, ObMalloc#start=0, ObMalloc#end=1996918, ObMalloc#start=23, ObMalloc#end=140, FREE_END=3557], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1fce4e91 0x1fce455d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [416]OB_MALLOC COST TOO MUCH TIME, cost_time=915653, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=915612, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [2025-08-14 09:05:15.752607] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2000822, owner: ObLog, click_count: 6, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=1996566, ObMalloc#start=259, ObMalloc#end=82, FREE_END=3904], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [416]OB_MALLOC COST TOO MUCH TIME, cost_time=915679, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=915664, WASH_BLOCK_END=15], seq:[0, 1, 2] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=886942, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=886898, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=886969, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=886958, WASH_BLOCK_END=11], seq:[0, 1, 2] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=498827, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=498785, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=498844, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=498837, WASH_BLOCK_END=7], seq:[0, 1, 2] [2025-08-14 09:05:16.763271] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:17.755611] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=990839][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:16.779711] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=1003939] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:16.779427] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=9] gc stale ls task succ [2025-08-14 09:05:16.780076] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=996136] start do ls ha handler(ls_id_array_=[{id:1}]) [276]OB_MALLOC COST TOO MUCH TIME, cost_time=365692, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=24, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=365640, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [2025-08-14 09:05:15.742951] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14746] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:17.756160] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2010145][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:17.756180] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:17.756196] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:16.781341] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=993841] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:17.756246] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=2994155] snapshot_traffic_info once [2025-08-14 09:05:16.784164] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:17.756474] WDIAG ev_invoke_pending (ev.c:2136) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2000822][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:17.756493] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:16.839404] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:05:17.756507] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:17.756551] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=915719] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:17.756605] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:17.756646] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:17.756684] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:05:17.756697] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:17.756709] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:16.868544] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:17.257335] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [276]OB_MALLOC COST TOO MUCH TIME, cost_time=365730, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=365707, WASH_BLOCK_END=23], seq:[0, 1, 2] [2025-08-14 09:05:17.391196] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:17.758675] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1008) [408][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:17.756402] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=1009711] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:05:17.760252] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=978449][errcode=-4076] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:02:34.747992] EDIAG [STORAGE] runTimerTask (ob_tenant_tablet_stat_mgr.cpp:1206) [14905][T1001_TimerWK8_TabletStatRpt][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] LOGGER COST TOO MUCH TIME, cost: 161768073, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b5724fe 0x96ee26d 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:15.738107] EDIAG [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2021261, , BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1d189030 0x1d18285e 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:17.761045] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:15.738107] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=40] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:17.754696] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=27] current log_task status(palf_id=1, self="172.77.0.7:2882", log_id=8607756, log_task={header:{begin_lsn:{lsn:1604090090}, end_lsn:{lsn:1604090203}, log_id:8607756, min_scn:{val:1755161749741193001, v:0}, max_scn:{val:1755161749741193001, v:0}, data_len:57, proposal_id:50, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:1604090090}, data_checksum:2031283194, accum_checksum:759186326, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1755161750716165, freeze_ts:1755161750716173, submit_ts:1755161750716183, flushed_ts:1755161853765770, gen_to_freeze cost time:8, gen_to_submit cost time:18, submit_to_flush cost time:103049587}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161974728707, last_flushed_end_lsn:{lsn:1604090331}}]) [2025-08-14 09:05:17.764843] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=2021261] temporary file module use no blocks [2025-08-14 09:05:17.764862] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=18] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:17.764879] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=15] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:17.764894] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623215202553, flush_io_finished_ret_:0, flush_io_finished_round_:26367, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:17.765821] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:17.765890] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=989840, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=989805], seq:[0, 1, 2, 3, 4, 5, 6] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=989861, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=989859], seq:[0, 1] [2025-08-14 09:05:17.775267] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:16.784083] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:17.775495] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] timer task use too much time(end=1755162317775495, start=1755162316784081, delta=991414) [2025-08-14 09:05:17.776871] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=42765] temporary file module use no blocks [2025-08-14 09:05:17.776896] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=23] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:17.776908] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:17.776929] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=21] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623215211107, flush_io_finished_ret_:0, flush_io_finished_round_:26352, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:17.776979] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:951) [779][T1002_PLSSer][T1001][YB42AC4D0007-00063C4964B9FE3E-0-0] [lt=2958469][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2025-08-14 09:05:15.766570] INFO [SERVER] check_report_replicas_ (ob_tenant_meta_checker.cpp:591) [14961][T1_TimerWK8_LSMetaCh][T1][YB42AC4D0007-00063C4979D99C24-0-0] [lt=1027652] add missing replica to ls meta table success(ret=0, ret="OB_SUCCESS", tenant_id=1, ls_id={id:1}) [2025-08-14 09:05:17.777641] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:15.748563] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2026237, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:15.748563] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:17.777893] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2026237][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:17.777913] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:17.777928] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:17.778849] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:17.782316] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:17.783264] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:17.789936] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:17.856453] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:17.856920] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:05:17.856980] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:17.857022] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:17.857068] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:17.857115] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:05:17.857132] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:17.857149] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:17.860331] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=0] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:05:17.976723] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:18.245650] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:18.257779] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:18.722315] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:18.726758] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:17.754567] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=997389][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:18.728315] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=43307] temporary file module use no blocks [2025-08-14 09:05:18.728342] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:18.728359] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=16] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:18.728375] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623215211078, flush_io_finished_ret_:0, flush_io_finished_round_:26391, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:17.779491] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=975034] gc stale ls task succ [2025-08-14 09:05:17.778243] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=14] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:05:15.743083] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2004203, owner: ObLog, click_count: 7, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=2004190, ALLOC_END=1, APPEND_END=1, ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=27340], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:17.882308] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=11] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:18.277526] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:18.740538] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:18.740616] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:18.728753] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [408][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=38] kill and revert session(conn.sessid_=3221590656, proxy_sessid=0, ret=0) [2025-08-14 09:05:13.777579] EDIAG [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4957696, owner: ObLog, click_count: 5, time dist:[FORMAT_END=4957685, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x989c577 0x989bc96 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:13.777579] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] [thread pool stat](pool_stat={pool:0x564d6e8922c0, last_check_time:1755162313777576, last_idle_time:102477764956, last_thread_count:12}, *pool={name:TimerWK, this:0x564d6e8922c0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:10, threads_idle_time:102477764956, tenant_id:500}, qsize=10, current_idle_time=102477764956) [2025-08-14 09:05:18.742858] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=4957696] [thread pool stat](pool_stat={pool:0x7f99667cbcf0, last_check_time:1755162318742857, last_idle_time:34927961689, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x7f99667cbcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:34927961689, tenant_id:500}, qsize=-1, current_idle_time=34927961689) [2025-08-14 09:05:18.742900] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=41] [thread pool stat](pool_stat={pool:0x7f99667ccaf0, last_check_time:1755162318742900, last_idle_time:26702380203, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x7f99667ccaf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:26702380203, tenant_id:500}, qsize=-1, current_idle_time=26702380203) [2025-08-14 09:05:18.742914] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] [thread pool stat](pool_stat={pool:0x7f9969de56b0, last_check_time:1755162318742913, last_idle_time:102938681542, last_thread_count:17}, *pool={name:TimerWK, this:0x7f9969de56b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:16, threads_idle_time:102938681542, tenant_id:1}, qsize=10, current_idle_time=102938681542) [2025-08-14 09:05:18.742926] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] [thread pool stat](pool_stat={pool:0x7f99356b0e70, last_check_time:1755162318742926, last_idle_time:102366343177, last_thread_count:15}, *pool={name:TimerWK, this:0x7f99356b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:12, threads_idle_time:102366343177, tenant_id:1001}, qsize=12, current_idle_time=102366343177) [2025-08-14 09:05:18.742947] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=20] [thread pool stat](pool_stat={pool:0x7f99230b0e70, last_check_time:1755162318742947, last_idle_time:102534969169, last_thread_count:17}, *pool={name:TimerWK, this:0x7f99230b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:16, threads_idle_time:102534969169, tenant_id:1002}, qsize=12, current_idle_time=102534969169) [2025-08-14 09:05:18.729327] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=2004203][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:18.744043] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=14678] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:18.744064] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:18.728424] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1079) [337][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=1008120][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2025-08-14 09:05:18.747584] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:18.747584] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1543) [337][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=19119][errcode=-4012] execute_write failed(ret=-4012, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1755161909716139), 'FAILURE_DETECTOR', 'clog disk hang event', 'FAILURE_MODULE', 'LOG', 'FAILURE_TYPE', 'PROCESS HANG', 'AUTO_RECOVER', 'False', '', '', '', '172.77.0.7', 2882), is_user_sql=false) [2025-08-14 09:05:18.747663] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:18.747734] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=8] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:18.749585] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:981) [779][T1002_PLSSer][T1001][YB42AC4D0007-00063C4964B9FE3E-0-0] [lt=22][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id"}, record_ret=-4283, ret=-4283) [2025-08-14 09:05:15.783367] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [339][EventTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2965138, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2942113, ObMalloc#start=1, ObMalloc#end=23022, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [384]OB_MALLOC COST TOO MUCH TIME, cost_time=969053, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=969044], seq:[0] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=969212, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=45, WASH_BLOCK_END=80, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=969081], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:15.783367] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [339][EventTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:18.753028] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [339][EventTimer][T0][Y0-0000000000000000-0-0] [lt=2965138][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:18.755265] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=9] gc stale ls task succ [2025-08-14 09:05:18.757114] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:18.757519] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:18.760804] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=10] PNIO [ratelimit] time: 1755162318760801, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006756, add_bytes: 0 [2025-08-14 09:05:18.764774] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=974389] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:18.764842] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=973473] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:18.765283] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:18.765575] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:18.765590] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] temporary file module use no blocks [2025-08-14 09:05:18.765597] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=5] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:18.764577] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=9] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:05:18.747620] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:159) [337][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4012] execute sql failed(ret=-4012, conn=0x7f9966b3d0f0, start=1755162022770470, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1755161909716139), 'FAILURE_DETECTOR', 'clog disk hang event', 'FAILURE_MODULE', 'LOG', 'FAILURE_TYPE', 'PROCESS HANG', 'AUTO_RECOVER', 'False', '', '', '', '172.77.0.7', 2882)) [2025-08-14 09:05:18.765616] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=18] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:18.790265] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:18.766833] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:2005) [807][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=2021900] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1002) [2025-08-14 09:05:18.728360] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=972307] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:19.722471] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=992607][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:19.722498] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:18.774592] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8961] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623220204691, flush_io_finished_ret_:0, flush_io_finished_round_:26368, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:19.725274] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:18.779554] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=947629] gc stale ls task succ [2025-08-14 09:05:18.728972] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=2017454] log compressor cycles once. (ret=0, cost_time=977534, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25879375872, fast_delete_log_mode=false) [2025-08-14 09:05:19.726659] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=23][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:05:19.314503] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [832]OB_MALLOC COST TOO MUCH TIME, cost_time=948734, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=948732, ALLOC_CHUNK_END=1], seq:[0, 1] [832]OB_MALLOC COST TOO MUCH TIME, cost_time=948751, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=948749], seq:[0, 1] [2025-08-14 09:05:18.776979] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=31] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:19.727209] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=948785] temporary file module use no blocks [2025-08-14 09:05:19.727818] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=28][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:05:19.727221] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:19.727914] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=689] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:19.727934] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=19] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623220216729, flush_io_finished_ret_:0, flush_io_finished_round_:26353, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:19.728808] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=949044] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:05:19.729662] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:19.729688] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=22] temporary file module use no blocks [2025-08-14 09:05:19.730396] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=953695][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:18.778907] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:19.730441] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=950097][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:19.733438] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1468] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:19.729697] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:19.735362] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=5654] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:19.735404] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=42] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623221168172, flush_io_finished_ret_:0, flush_io_finished_round_:26392, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:19.735443] INFO [STORAGE] refresh_queuing_mode (ob_tenant_tablet_stat_mgr.cpp:1161) [14905][T1001_TimerWK8_TabletStatRpt][T1001][Y0-0000000000000000-0-0] [lt=161768073] refresh queuing mode(ret=0, tenant_id=1001, stream_cnt=16, update_schema_cnt=16, extreme_tablet_cnt=0, cost_time=982216) [2025-08-14 09:05:19.736853] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [880][T1002_ArcSender][T1002][YB42AC4D0007-00063C495EB99C16-0-0] [lt=17] ObArchiveSender is running(thread_index=5) [2025-08-14 09:05:19.742274] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1409] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:19.744605] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:19.746748] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=56] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:19.747795] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=10] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:19.746318] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:19.736166] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1507) [807][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=949009] start lib cache evict(tenant_id=1002, mem_hold=211812352, mem_limit=429496725, cache_obj_num=1331, cache_node_num=1330) [2025-08-14 09:05:19.752492] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1524) [807][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=16297] end lib cache evict(tenant_id=1002, cache_evict_num=0, mem_hold=211812352, mem_limit=429496725, cache_obj_num=1331, cache_node_num=1330) [2025-08-14 09:05:19.754637] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:19.735507] INFO [STORAGE] runTimerTask (ob_tenant_tablet_stat_mgr.cpp:1211) [14905][T1001_TimerWK8_TabletStatRpt][T1001][Y0-0000000000000000-0-0] [lt=62] TenantTabletStatMgr refresh all tablet stream(MTL_ID()=1001, interval_step=4, global_iter_pool={is_inited:false, is_washing:false, is_disabled:false, get_cnt:0, tenant_id:1001, bucket_cnt:0, tenant_mem_user_limit:0, tenant_mem_user_hold:0, cached_node_array:0x7f99333dcff0}) [2025-08-14 09:05:19.760417] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [14905][T1001_TimerWK8][T1001][Y0-0000000000000000-0-0] [lt=24869][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:19.761261] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:19.764119] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:19.764174] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:19.764491] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=15] PNIO [ratelimit] time: 1755162319764488, bytes: 0, bw: 0.000000 MB/s, add_ts: 1003687, add_bytes: 0 [2025-08-14 09:05:19.764901] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=10] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:19.764965] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=6] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:19.765703] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=1026] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:19.774821] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:19.752514] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2792) [807][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=18] schedule next cache evict task(evict_interval=5000000) [2025-08-14 09:05:19.778484] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:19.779133] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4076] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:19.779481] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:19.779625] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=944769] gc stale ls task succ [2025-08-14 09:05:19.780487] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=253] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:19.782253] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=202] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:19.792253] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:19.809134] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:19.829478] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:19.864446] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:19.930657] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=234] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:19.964587] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=157] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:05:19.964652] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:19.964691] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:19.964730] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:19.964766] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:05:19.964778] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:19.964790] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:19.972794] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:19.986153] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:20.093437] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:20.256468] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:20.267145] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:20.327125] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=411618][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:20.721808] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2038] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:19.790875] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:20.723312] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:288) [884][T1002_ArcFetche][T1002][YB42AC4D0007-00063C495ED99C16-0-0] [lt=1718] ObArchiveFetcher is running(thread_index=1) [391]OB_MALLOC COST TOO MUCH TIME, cost_time=967103, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=967096], seq:[0] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=967213, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=46, WASH_BLOCK_END=43, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=967119], seq:[0, 1, 2, 3, 4] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=966374, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=966336], seq:[0, 1, 2, 3, 4, 5, 6] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=966396, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=966394], seq:[0, 1] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=968336, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=968303, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3, 4, 5, 6, 7] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=968417, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=968354, WASH_BLOCK_END=63], seq:[0, 1, 2] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=998341, tenant_id=500, label=TempVariables, ctx_id=0, prio=0, size=262176, owner: ObMalloc, click_count: 1, time dist:[ALLOC_CHUNK_END=27355], seq:[0] [2025-08-14 09:05:19.754752] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1444] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:20.723450] WDIAG ev_invoke_pending (ev.c:2136) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=967234][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:19.757406] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=11] gc stale ls task succ [2025-08-14 09:05:20.726331] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:20.727114] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=947615] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:20.727139] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=19] temporary file module use no blocks [2025-08-14 09:05:20.727150] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:20.727165] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:20.727190] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=24] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623222163411, flush_io_finished_ret_:0, flush_io_finished_round_:26369, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:20.745761] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5681] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:20.746033] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:20.746114] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:20.723301] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:233) [827][T1002_RFLWorker][T1002][YB42AC4D0007-00063C495D099C16-0-0] [lt=1303] ObRemoteFetchWorker is running(thread_index=0) [2025-08-14 09:05:20.747817] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=2068] temporary file module use no blocks [2025-08-14 09:05:20.747837] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=19] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:20.747851] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:20.747864] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623222175195, flush_io_finished_ret_:0, flush_io_finished_round_:26393, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:20.747872] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=4089] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:19.731416] EDIAG [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:582) [791][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1025249, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1025245, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x1c4371ab 0x1c43ae96 0x1c47767a 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:20.757474] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=966423] gc stale ls task succ [2025-08-14 09:05:20.757552] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=968588] gc stale ls task succ [2025-08-14 09:05:20.758330] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:20.758369] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:20.761313] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:20.765154] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=16] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:20.765212] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=10] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:20.765747] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [339][EventTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:20.765783] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=18] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:20.766175] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=27] PNIO [ratelimit] time: 1755162320766174, bytes: 0, bw: 0.000000 MB/s, add_ts: 1001686, add_bytes: 0 [2025-08-14 09:05:20.766214] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:20.729680] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:20.768398] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=38654] temporary file module use no blocks [2025-08-14 09:05:20.768410] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:20.768421] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:20.768437] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623222167730, flush_io_finished_ret_:0, flush_io_finished_round_:26354, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:20.768802] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:20.776337] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:20.779433] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:20.780065] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4076] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:20.780537] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:20.790429] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:20.790434] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:20.823351] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:20.880129] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=15] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:05:20.931358] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=22] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:20.980547] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:20.991344] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:21.026075] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:21.095631] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:21.268799] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:21.348304] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=450950][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:21.495374] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:21.726433] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:21.727456] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=31] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:21.727499] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=28] temporary file module use no blocks [2025-08-14 09:05:21.727514] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=14] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:21.727533] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=18] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:21.727551] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=18] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623223166991, flush_io_finished_ret_:0, flush_io_finished_round_:26370, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:21.722077] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:21.731827] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=768] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:05:21.734507] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=291] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:21.744820] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:21.746273] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=295] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:21.747886] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=18] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [568]OB_MALLOC COST TOO MUCH TIME, cost_time=968769, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=968766], seq:[0] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=968912, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=54, WASH_BLOCK_END=64, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=968789], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:20.779691] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=13] gc stale ls task succ [2025-08-14 09:05:20.968398] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:05:21.753688] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=784058] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:21.753747] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:21.753795] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:21.753841] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:05:21.753857] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:21.753869] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:21.757345] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:21.758099] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=25] gc stale ls task succ [2025-08-14 09:05:21.758143] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=14] gc stale ls task succ [2025-08-14 09:05:21.758593] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:21.758875] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:21.752251] INFO print_info (ob_kvcache_hazard_domain.cpp:138) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11005565] [KVCACHE-HAZARD-POINTER] retired memory size: 0, blocks size: 94, free list size: 1003, memory_efficiency: 0.897691 [2025-08-14 09:05:21.758921] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:1131) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6656][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:21.758948] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] can not find enough memory block to wash(ret=-4273, size_washed=0, size_to_wash=6291456) [2025-08-14 09:05:21.761645] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:21.762506] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:21.762543] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:21.762589] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:21.762610] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=15] temporary file module use no blocks [2025-08-14 09:05:21.762620] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:21.762645] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=24] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:21.762660] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=16] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623223187658, flush_io_finished_ret_:0, flush_io_finished_round_:26394, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:21.762703] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:21.765288] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:21.765342] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=10] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:21.767168] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=20] PNIO [ratelimit] time: 1755162321767166, bytes: 0, bw: 0.000000 MB/s, add_ts: 1000992, add_bytes: 0 [2025-08-14 09:05:21.769123] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:21.769148] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=21] temporary file module use no blocks [2025-08-14 09:05:21.769154] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=5] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:21.769167] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:21.769185] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=17] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623223208235, flush_io_finished_ret_:0, flush_io_finished_round_:26355, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:21.775384] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:21.779333] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:21.779757] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=968948] gc stale ls task succ [2025-08-14 09:05:21.780537] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:21.781734] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=215] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:21.784294] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4076] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:21.791255] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:21.794813] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:21.824415] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:21.884354] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:05:21.981243] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:22.004327] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:22.054229] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:05:22.054294] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=30] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:22.054373] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:22.054420] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:22.054456] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:05:22.054468] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:22.097271] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:22.269711] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:22.276291] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:22.349367] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=40][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:21.941434] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=20] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:22.720860] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8866] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:22.726463] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=29] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:22.727659] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=30] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:22.727697] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=33] temporary file module use no blocks [2025-08-14 09:05:22.727708] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:22.727722] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:22.727736] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623224167351, flush_io_finished_ret_:0, flush_io_finished_round_:26371, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:22.054479] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:21.768296] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [726][T1001_ObXAHbWor][T1001][Y0-0000000000000000-0-0] [lt=15] XA scheduler heartbeat task statistics(avg_time=2) [982]OB_MALLOC COST TOO MUCH TIME, cost_time=1000319, tenant_id=500, label=CommonArray, ctx_id=0, prio=0, size=7936, [2025-08-14 09:05:22.736756] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [477]OB_MALLOC COST TOO MUCH TIME, cost_time=972728, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=972726], seq:[0] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=972757, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=972745], seq:[0, 1] [2025-08-14 09:05:22.743306] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:22.770853] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:22.770879] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=20] temporary file module use no blocks [2025-08-14 09:05:22.770887] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:22.779569] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:22.770898] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:22.744952] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:22.790625] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=45602][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:22.732505] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:05:22.791500] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4076] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:22.760025] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=22] gc stale ls task succ [2025-08-14 09:05:22.801281] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:22.755262] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:22.801688] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=46355][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:22.696526] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=305][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [832]OB_MALLOC COST TOO MUCH TIME, cost_time=971825, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=971817], seq:[0] [832]OB_MALLOC COST TOO MUCH TIME, cost_time=971958, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=40, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=971874], seq:[0, 1, 2, 3, 4] [2025-08-14 09:01:21.739653] EDIAG [STORAGE.TRANS] print_stat_ (ob_trans_functor.h:106) [865][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1012219, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1012218, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1b8c0892 0x1b6ec341 0x971c64e 0x986bcba 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:22.791897] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:22.802399] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10486][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:22.802228] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:66) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=995417][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-08-14 09:05:22.802414] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:93) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=186][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-08-14 09:05:23.006456] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:23.270927] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:22.736840] WDIAG [SERVER] set_timeout (ob_inner_sql_connection.cpp:2049) [866][T1002_TransferS][T1002][YB42AC4D0007-00063C495F199DA3-0-0] [lt=2947388][errcode=-4012] already timeout(ret=-4012, ctx={abs_timeout_us:1755162266724461}, abs_timeout_us=1755162266724461) [2025-08-14 09:05:23.747519] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9129] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:23.758323] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2482] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:23.760162] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=40039] gc stale ls task succ [2025-08-14 09:05:23.762101] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=1785] gc stale ls task succ [2025-08-14 09:05:23.764033] INFO print_info (ob_kvcache_hazard_domain.cpp:138) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [KVCACHE-HAZARD-POINTER] retired memory size: 0, blocks size: 94, free list size: 1003, memory_efficiency: 0.897691 [2025-08-14 09:05:23.764046] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:1131) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:23.764059] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9] can not find enough memory block to wash(ret=-4273, size_washed=0, size_to_wash=6291456) [2025-08-14 09:05:22.788834] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=17908] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623224208982, flush_io_finished_ret_:0, flush_io_finished_round_:26356, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:22.772684] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:22.780014] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17813][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:23.780865] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=999311] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:23.783451] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:23.783486] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=387] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:23.784068] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=207] gc stale ls task succ [2025-08-14 09:05:23.784101] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=904] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:23.784127] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=507] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:22.779262] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:23.790919] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=58234] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:05:23.791407] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:23.792138] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4076] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:23.804917] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:23.830117] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:23.892314] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:05:23.992346] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:24.007448] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:24.019991] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=220] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:24.132534] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=148] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:05:24.132590] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:24.215850] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:24.353539] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=281][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:24.524944] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:24.721506] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=237] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:24.730249] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=209] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:24.730315] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=52] temporary file module use no blocks [2025-08-14 09:05:24.730341] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=25] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:24.730375] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=32] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:24.730395] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=20] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623226169919, flush_io_finished_ret_:0, flush_io_finished_round_:26373, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:24.732275] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4156] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:24.745265] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=766] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:24.745265] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=612] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:24.746481] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3001] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:22.779262] EDIAG [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1003686, , BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1d189030 0x1d18285e 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:22.772684] EDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 1003308, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1003306, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x9757ced 0x97569df 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:20.779395] EDIAG [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2510) [807][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=24194][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2949921, owner: ObLog, click_count: 6, time dist:[FORMAT_END=2949591, ObMalloc#start=1, ObMalloc#end=158, ObMalloc#start=24, ObMalloc#end=70, FREE_END=77], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1535e3fb 0x15365204 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:24.132630] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:24.751376] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=617769] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:24.751449] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=42] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:05:24.751463] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:24.751476] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:24.753429] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=177] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:24.755414] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=180] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:24.758741] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=342] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:24.762284] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=13] gc stale ls task succ [2025-08-14 09:05:24.766104] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=15] gc stale ls task succ [552]OB_MALLOC COST TOO MUCH TIME, cost_time=968924, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=968922], seq:[0] [387]OB_MALLOC COST TOO MUCH TIME, cost_time=971161, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=971120], seq:[0, 1, 2] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=971799, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=39, WASH_BLOCK_END=971760], seq:[0, 1, 2] [389]OB_MALLOC COST TOO MUCH TIME, cost_time=975654, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=46, WASH_BLOCK_END=975608], seq:[0, 1, 2] [552]OB_MALLOC COST TOO MUCH TIME, cost_time=1007615, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=968947], seq:[0, 1] [2025-08-14 09:05:24.751352] EDIAG [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=1003686][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1007643, owner: ObLog, click_count: 5, time dist:[FORMAT_END=7, ObMalloc#start=1, ObMalloc#end=1007633, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x1d0f5b0a 0x1d0f57c6 0x1d17ad7c 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [574]OB_MALLOC COST TOO MUCH TIME, cost_time=988382, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=988375], seq:[0] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=991424, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=991401, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=991466, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=991442, WASH_BLOCK_END=24], seq:[0, 1, 2] [832]OB_MALLOC COST TOO MUCH TIME, cost_time=987367, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=987343, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [832]OB_MALLOC COST TOO MUCH TIME, cost_time=987400, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=987377, WASH_BLOCK_END=23], seq:[0, 1, 2] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=983597, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=983572, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=983617, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=983606, WASH_BLOCK_END=10], seq:[0, 1, 2] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=985522, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=985495, ALLOC_CHUNK_END=12], seq:[0, 1, 2, 3] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=985559, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=985540, WASH_BLOCK_END=19], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=985118, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=985086, ALLOC_CHUNK_END=11], seq:[0, 1, 2, 3] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=985134, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=985126, WASH_BLOCK_END=8], seq:[0, 1, 2] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=1002288, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1002282], seq:[0] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=1018644, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=45, WASH_BLOCK_END=16291, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1002302], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:23.768928] EDIAG do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=352][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2000517, owner: ObLog, click_count: 5, time dist:[FORMAT_END=981863, ObMalloc#start=1, ObMalloc#end=1018652, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x202c530b 0x202c6888 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:24.751352] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=1003686] temporary file module use no blocks [2025-08-14 09:05:25.760544] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=1007643] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:25.760556] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:25.760566] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623224202456, flush_io_finished_ret_:0, flush_io_finished_round_:26395, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:24.775338] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=984925] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:25.764285] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=987447] temporary file module use no blocks [2025-08-14 09:05:25.764294] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:25.764305] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:25.764316] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623226215019, flush_io_finished_ret_:0, flush_io_finished_round_:26357, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:24.779264] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:25.764410] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=983657][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:23.768928] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=352] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:24.767631] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=2143][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1003759, owner: ObLog, click_count: 9, time dist:[ALLOC_BLOCK#start=993264, ALLOC_BLOCK#end=84, ALLOC_BLOCK#start=10351, ALLOC_BLOCK#end=4, FORMAT_END=45, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=7], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:24.767631] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=2143] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:05:24.734127] EDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4016] LOGGER COST TOO MUCH TIME, cost: 1037842, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=23, ALLOC_BLOCK#end=5, FORMAT_END=1037806, ObMalloc#start=0, ObMalloc#end=2, ALLOC_END=0, APPEND_END=6], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x13b06ffb 0x958a39e 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [382]OB_MALLOC COST TOO MUCH TIME, cost_time=990486, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=990468], seq:[0, 1, 2] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=990500, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=990498], seq:[0, 1] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=988526, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=988515], seq:[0, 1, 2] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=988535, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=988533], seq:[0, 1] [864]OB_MALLOC COST TOO MUCH TIME, cost_time=981807, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=981787, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=981677, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=981657, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=981698, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=981686, WASH_BLOCK_END=12], seq:[0, 1, 2] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=977427, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=977406, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=977445, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=977436, WASH_BLOCK_END=9], seq:[0, 1, 2] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=961723, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=961701, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=961742, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=961733, WASH_BLOCK_END=9], seq:[0, 1, 2] [2025-08-14 09:05:24.734127] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:25.773563] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=1037842][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:24.782260] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:25.774278] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=990519][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:24.784274] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=12] gc stale ls task succ [2025-08-14 09:05:24.791255] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:25.774481] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=981739][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:24.811414] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=1002607, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=988390], seq:[0, 1] [2025-08-14 09:05:24.770767] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=1669][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1002645, owner: ObLog, click_count: 5, time dist:[FORMAT_END=25, ObMalloc#start=0, ObMalloc#end=1002615, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:24.770767] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=1669] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:24.767650] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=8055][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1005893, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1005886, ObMalloc#start=0, ObMalloc#end=2, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:24.767650] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=8055] report RowHolderMapper summary info(count=2, bkt_cnt=248) [658]OB_MALLOC COST TOO MUCH TIME, cost_time=935883, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=935867], seq:[0, 1, 2] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=935916, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=7, ALLOC_BLOCK#end=935908], seq:[0, 1] [808]OB_MALLOC COST TOO MUCH TIME, cost_time=922780, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=922755], seq:[0, 1, 2] [2025-08-14 09:05:25.775442] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:24.838254] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [808]OB_MALLOC COST TOO MUCH TIME, cost_time=922789, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=922787], seq:[0, 1] [2025-08-14 09:05:24.851462] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=1003308][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=781524, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=781506], seq:[0, 1, 2] [2025-08-14 09:05:25.718551] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=3806] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [472]OB_MALLOC COST TOO MUCH TIME, cost_time=781533, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=781532], seq:[0, 1] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=757254, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=757237], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=757262, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=757261], seq:[0, 1] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=753963, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=753946], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=753969, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=753969], seq:[0, 1] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=620572, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=620555], seq:[0, 1, 2] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=620581, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=620579], seq:[0, 1] [2025-08-14 09:05:24.992969] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:25.017311] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:25.020677] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=21] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:25.154295] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:05:25.775882] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=620605] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:25.775929] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:25.775971] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:25.776012] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:05:25.776024] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:25.776035] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [772]OB_MALLOC COST TOO MUCH TIME, cost_time=510713, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=510697], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=510722, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=510721], seq:[0, 1] [764]OB_MALLOC COST TOO MUCH TIME, cost_time=169846, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=169825], seq:[0, 1, 2] [764]OB_MALLOC COST TOO MUCH TIME, cost_time=169854, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=169853], seq:[0, 1] [2025-08-14 09:05:22.717887] EDIAG dump_info (ob_timer_service.cpp:728) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=365799][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3053813, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=7, ALLOC_BLOCK#end=4, FORMAT_END=2996754, ObMalloc#start=1, ObMalloc#end=57044, ALLOC_END=1, APPEND_END=2], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x2618816f 0x9756b3d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:24.727352] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=240][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1047749, owner: ObLog, click_count: 8, time dist:[FORMAT_END=34483, ObMalloc#start=2, ObMalloc#end=971183, ObMalloc#start=68, ObMalloc#end=41955, FREE_END=58, ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=42], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [401]OB_MALLOC COST TOO MUCH TIME, cost_time=1023727, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=48, WASH_BLOCK_END=982784, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=40894], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:24.751607] EDIAG [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=200][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1023773, owner: ObLog, click_count: 5, time dist:[FORMAT_END=7, ObMalloc#start=0, ObMalloc#end=1023737, ALLOC_END=26, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a1f867 0x27a135c3 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:24.761510] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1013932, owner: ObLog, click_count: 6, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=971812, ObMalloc#start=22, ObMalloc#end=41789, FREE_END=298], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:23.764069] EDIAG [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:845) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2010164, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1977530, ObMalloc#start=1, ObMalloc#end=32609, ALLOC_END=21, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1d7e0 0x26a180ba 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x1fc9113e 0x1fc8e0b3 0x1fc7a380 0x96f94a6 0x96f8efa 0x93131eb 0x9329467 0x932778f 0x9315ac5 0x27aa2827 0x27a872f9 0x1c0470a6 0x1c0340a3 0x1c02b713 0x97a97cd 0x97a1ba3 0x97a1536 0x2668daa5 0x2668bbb2 [2025-08-14 09:05:24.758829] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1017335, owner: ObLog, click_count: 6, time dist:[FORMAT_END=8, ObMalloc#start=0, ObMalloc#end=975671, ObMalloc#start=25, ObMalloc#end=41047, FREE_END=583], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:25.264668] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:25.606112] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:24.746255] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:22.717887] INFO dump_info (ob_timer_service.cpp:728) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=365799] dump info [summary](this=0x564d6e891f40, this={this:0x564d6e891f40, tenant_id_:500, priority_task_queue_.size():20, running_task_set_.size():20, uncanceled_task_set_.size():0, worker_thread_pool_.get_queue_num():10}) [2025-08-14 09:05:23.764080] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1055) [866][T1002_TransferS][T1002][YB42AC4D0007-00063C495F199DA3-0-0] [lt=995499][errcode=-4012] set timeout failed(ret=-4012) [2025-08-14 09:05:25.722168] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=32] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:25.776379] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=54128][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:25.731476] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=52] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:25.776413] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=44864] temporary file module use no blocks [2025-08-14 09:05:25.776422] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:25.776434] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:25.776446] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623227170216, flush_io_finished_ret_:0, flush_io_finished_round_:26374, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:24.751607] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=200] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:25.776944] WDIAG ev_invoke_pending (ev.c:2136) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=1023773][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:25.776959] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=7] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:23.764069] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:845) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_to_wash=6291456) [2025-08-14 09:05:25.777302] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:66) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2010164][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-08-14 09:05:25.777313] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:93) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-08-14 09:05:25.746442] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:25.777381] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=30892][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:24.776326] EDIAG [RS] skip_check_member_list_change_ (ob_ls_recovery_stat_handler.cpp:109) [777][T1002_TeRec][T1002][YB42AC4D0007-00063C495DDA0084-0-0] [lt=17179][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1000431, , BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x1067fb2c 0x1067ecc2 0x1067e03d 0x1002f2a3 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:25.778258] WDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=45930][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:25.776760] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1047749][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:25.778295] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1520] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:25.778340] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1333] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:25.777725] WDIAG ev_invoke_pending (ev.c:2136) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1017335][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:25.778366] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=632] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:25.777829] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22378][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:25.777933] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=37033][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:25.778050] WDIAG ev_invoke_pending (ev.c:2136) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23589][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:25.745268] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:25.778953] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=33633][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:25.779553] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:25.780362] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:25.781065] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=475] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:25.789248] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=988553] gc stale ls task succ [2025-08-14 09:05:25.789430] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=985451] PNIO [ratelimit] time: 1755162325789429, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007636, add_bytes: 0 [2025-08-14 09:05:25.814960] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:25.871534] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=358][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:25.875693] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:26.178252] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:26.514880] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:26.722583] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [567]OB_MALLOC COST TOO MUCH TIME, cost_time=8039463, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16448, owner: ALLOC_BLOCK, click_count: 24, time dist:[ObMalloc#start=2018704, ObMalloc#end=998357, ObLog#start=1003400, ObLog#end=6656, ObLog#end=17, ObLog#start=10, ObLog#end=13, ObLog#end=995417, ObLog#start=46272, ObLog#end=186, ObMalloc#start=929939, ObMalloc#end=30151, ObLog#start=65, ObLog#end=12, ObLog#end=2010188, ObLog#end=12], seq:[1, 2, 3, 4, 6, 7, 8, 10, 11, 12, 17, 18, 19, 20, 22, 23] [567]OB_MALLOC COST TOO MUCH TIME, cost_time=8988575, tenant_id=1, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=8988573], seq:[0, 1] [2025-08-14 09:05:25.777269] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=980187] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:05:26.727792] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=949084][errcode=-4076] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:23.845400] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [741][T1001_MultiVers][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1929923, owner: ObLog, click_count: 8, time dist:[FORMAT_END=1929021, ObMalloc#start=1, ObMalloc#end=64, ObMalloc#start=22, ObMalloc#end=38, ObMalloc#start=352, ObMalloc#end=51, FREE_END=374], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0xf9fb357 0x97ae92e 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:22.730951] EDIAG [OCCAM] operator() (ob_occam_timer.h:85) [338][Occam][T0][Y0-0000000000000000-0-0] [lt=8104][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 1992603, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=25253, FORMAT_END=1967331, ObMalloc#start=1, ObMalloc#end=4, ALLOC_END=1, APPEND_END=8], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ffa3b 0x9835318 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [864]OB_MALLOC COST TOO MUCH TIME, cost_time=983018, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=983016], seq:[0] [864]OB_MALLOC COST TOO MUCH TIME, cost_time=1972274, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=981815, WASH_BLOCK_END=46, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=983033], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:24.791042] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=30] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:05:26.728439] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=22][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:05:26.756348] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:26.778490] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22106][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:26.760412] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [338][Occam][T0][Y0-0000000000000000-0-0] [lt=1992603][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:26.779695] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:26.076020] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=753988] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:26.728370] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [741][T1001_MultiVers][T1001][Y0-0000000000000000-0-0] [lt=1929923][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:26.740593] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=494] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:26.779851] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=39198][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:26.748271] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:26.779887] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=31568][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:26.756381] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=8] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:26.779915] WDIAG ev_invoke_pending (ev.c:2136) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=23497][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:26.762200] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=579] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:26.779962] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17734][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:25.775780] EDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 1002702, owner: ObLog, click_count: 9, time dist:[FORMAT_END=958445, ObMalloc#start=1, ObMalloc#end=43752, ObMalloc#start=17, ObMalloc#end=72, ObMalloc#start=410, ObMalloc#end=1, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1b53d414 0x97ad9aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:25.775780] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162325775779, start=1755162325017309, delta=758470) [2025-08-14 09:05:26.780028] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1002702][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:26.728420] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=23][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:05:24.780256] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1997516, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:24.748455] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2029367, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=8, ALLOC_BLOCK#end=4, FORMAT_END=2029351, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:25.790563] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:26.780194] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=988135][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:26.780297] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:26.732635] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9110][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:24.780256] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:26.780824] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1997516][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:26.780844] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:26.780858] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:24.748455] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:26.780913] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=2029367][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:25.783654] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [339][EventTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:26.781264] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:24.746255] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1028491, owner: ObLog, click_count: 5, time dist:[FORMAT_END=970854, ObMalloc#start=1, ObMalloc#end=57632, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:25.762529] EDIAG [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=40][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1017855, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1017850, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x12377491 0x1236e8e9 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:24.791042] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1972313, owner: ObLog, click_count: 5, time dist:[FORMAT_END=13, ObMalloc#start=1, ObMalloc#end=1972292, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:25.354139] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=39][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1425678, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1425675, ALLOC_END=0, APPEND_END=3], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:26.530637] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:26.734187] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:26.781296] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:26.781307] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=47047][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:26.734437] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:26.781336] WDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=46827][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:26.722699] INFO dump_info (ob_timer_service.cpp:731) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=3053813] dump queue token(this=0x564d6e891f40, token={this:0x7f9970fc4738, timer:0x7f9970f40f80, task:0x564d6cd5b940, task_type:"N9oceanbase3sql15ObSQLSessionMgrE", scheduled_time:1755161613781775, pushed_time:0, delay:5000000}) [2025-08-14 09:05:26.781351] INFO dump_info (ob_timer_service.cpp:731) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=58561] dump queue token(this=0x564d6e891f40, token={this:0x7f9970fc4ff8, timer:0x7f9970f40f80, task:0x564d6d07c170, task_type:"N9oceanbase8observer8ObServer25ObRefreshNetworkSpeedTaskE", scheduled_time:1755161615749643, pushed_time:0, delay:1000000}) [2025-08-14 09:05:26.781362] INFO dump_info (ob_timer_service.cpp:731) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=11] dump queue token(this=0x564d6e891f40, token={this:0x7f9970fc49d8, timer:0x7f9970f40f80, task:0x564d6ef827d0, task_type:"N9oceanbase7storage20ObDDLCtrlSpeedHandle22RefreshSpeedHandleTaskE", scheduled_time:1755161615757725, pushed_time:0, delay:1000000}) [2025-08-14 09:05:26.781372] INFO dump_info (ob_timer_service.cpp:731) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=10] dump queue token(this=0x564d6e891f40, token={this:0x7f9970fc4b98, timer:0x7f9970f40f80, task:0x564d6d07c068, task_type:"N9oceanbase8observer16ObTenantDutyTaskE", scheduled_time:1755161617719061, pushed_time:0, delay:10000000}) [2025-08-14 09:05:26.781382] INFO dump_info (ob_timer_service.cpp:731) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=10] dump queue token(this=0x564d6e891f40, token={this:0x7f9970fc50d8, timer:0x7f9970f40f80, task:0x564d6d07c190, task_type:"N9oceanbase8observer8ObServer24ObRefreshCpuFreqTimeTaskE", scheduled_time:1755161617737100, pushed_time:0, delay:10000000}) [2025-08-14 09:05:26.781391] INFO dump_info (ob_timer_service.cpp:731) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=9] dump queue token(this=0x564d6e891f40, token={this:0x7f9970fc43b8, timer:0x7f9970f40f80, task:0x564d6d07a448, task_type:"N9oceanbase5share17ObServerTraceTaskE", scheduled_time:1755161621733806, pushed_time:0, delay:8000000}) [2025-08-14 09:05:26.781408] INFO dump_info (ob_timer_service.cpp:731) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=17] dump queue token(this=0x564d6e891f40, token={this:0x7f9970fc56f8, timer:0x7f9970f40f80, task:0x564d6ef58bc8, task_type:"N9oceanbase7storage24ObPrintTenantMemoryUsageE", scheduled_time:1755161623782823, pushed_time:0, delay:10000000}) [2025-08-14 09:05:26.781418] INFO dump_info (ob_timer_service.cpp:731) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=9] dump queue token(this=0x564d6e891f40, token={this:0x7f9970fc4818, timer:0x7f9970f40f80, task:0x564d6eb457d0, task_type:"N9oceanbase3sql23ObDASTaskResultGCRunnerE", scheduled_time:1755161623784040, pushed_time:0, delay:10000000}) [2025-08-14 09:05:26.762170] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=386] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:26.781431] WDIAG ev_invoke_pending (ev.c:2136) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19231][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:26.781427] INFO dump_info (ob_timer_service.cpp:731) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=9] dump queue token(this=0x564d6e891f40, token={this:0x7f9970fc4e38, timer:0x7f9970f40f80, task:0x564d6d07c140, task_type:"N9oceanbase7storage25ObRedefTableHeartBeatTaskE", scheduled_time:1755161637759743, pushed_time:0, delay:30000000}) [2025-08-14 09:05:26.781441] INFO dump_info (ob_timer_service.cpp:731) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=14] dump queue token(this=0x564d6e891f40, token={this:0x7f9970fc65d8, timer:0x7f9970f8c420, task:0x564d6e8e3210, task_type:"N9oceanbase3omt19ObTenantTimezoneMgr18DeleteTenantTZTaskE", scheduled_time:1755161702763867, pushed_time:0, delay:5000000}) [2025-08-14 09:05:26.781451] INFO dump_info (ob_timer_service.cpp:731) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=10] dump queue token(this=0x564d6e891f40, token={this:0x7f9970fc5a78, timer:0x7f9970f8c420, task:0x564d6e8e31f8, task_type:"N9oceanbase3omt19ObTenantTimezoneMgr15AddTenantTZTaskE", scheduled_time:1755161702763912, pushed_time:0, delay:5000000}) [2025-08-14 09:05:26.781461] INFO dump_info (ob_timer_service.cpp:731) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=9] dump queue token(this=0x564d6e891f40, token={this:0x7f9970fc4658, timer:0x7f9970f40f80, task:0x564d6d07a468, task_type:"N9oceanbase8observer23ObSchemaReleaseTimeTaskE", scheduled_time:1755161716573087, pushed_time:0, delay:0}) [2025-08-14 09:05:26.781470] INFO dump_info (ob_timer_service.cpp:731) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=10] dump queue token(this=0x564d6e891f40, token={this:0x7f9970fc41f8, timer:0x7f9970f40f80, task:0x564d6d07c3b0, task_type:"N9oceanbase5share22ObDeviceCredentialTaskE", scheduled_time:1755162016246491, pushed_time:0, delay:1200000000}) [2025-08-14 09:05:26.781480] INFO dump_info (ob_timer_service.cpp:731) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=10] dump queue token(this=0x564d6e891f40, token={this:0x7f9970fc4578, timer:0x7f9970f40f80, task:0x564d6d07a498, task_type:"N9oceanbase8observer26ObRemoteMasterRsUpdateTaskE", scheduled_time:1755162028742204, pushed_time:0, delay:600000000}) [2025-08-14 09:05:26.781491] INFO dump_info (ob_timer_service.cpp:731) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=10] dump queue token(this=0x564d6e891f40, token={this:0x7f9970fc5378, timer:0x7f9970ecbf90, task:0x564d6cd25100, task_type:"N9oceanbase10rootserver23ObIngressBWAllocServiceE", scheduled_time:1755162167753034, pushed_time:0, delay:3000000}) [2025-08-14 09:05:26.781535] INFO dump_info (ob_timer_service.cpp:731) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=44] dump queue token(this=0x564d6e891f40, token={this:0x7f9970fc5618, timer:0x564d6efbde60, task:0x564d6efbde20, task_type:"N9oceanbase12blocksstable14ObBlockManager13MarkBlockTaskE", scheduled_time:1755162196749035, pushed_time:0, delay:30000000}) [2025-08-14 09:05:26.781543] INFO dump_info (ob_timer_service.cpp:731) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=8] dump queue token(this=0x564d6e891f40, token={this:0x7f9970fc4498, timer:0x7f9966a67ec0, task:0x564d6d07b140, task_type:"N9oceanbase8observer32ObServerTabletMetaTableCheckTaskE", scheduled_time:1755162634710735, pushed_time:0, delay:0}) [2025-08-14 09:05:26.781553] INFO dump_info (ob_timer_service.cpp:731) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=9] dump queue token(this=0x564d6e891f40, token={this:0x7f9970fc6bf8, timer:0x7f9970f40f80, task:0x564d6d083c48, task_type:"N9oceanbase3sql20ObConnectResourceMgr23ConnResourceCleanUpTaskE", scheduled_time:1755164416565289, pushed_time:0, delay:0}) [2025-08-14 09:05:26.781564] INFO dump_info (ob_timer_service.cpp:731) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=11] dump queue token(this=0x564d6e891f40, token={this:0x7f9970fc4f18, timer:0x7f9970f40f80, task:0x564d6d07c150, task_type:"N9oceanbase8observer8ObServer17ObRefreshTimeTaskE", scheduled_time:1755164416594592, pushed_time:0, delay:3600000000}) [2025-08-14 09:05:26.781573] INFO dump_info (ob_timer_service.cpp:731) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=9] dump queue token(this=0x564d6e891f40, token={this:0x7f9970fc48f8, timer:0x7f9970f40f80, task:0x564d6cd59ea0, task_type:"N9oceanbase5share29ObClearTabletLSCacheTimerTaskE", scheduled_time:1755178816580457, pushed_time:0, delay:21600000000}) [2025-08-14 09:05:26.781585] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:26.781643] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1028491][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:26.781671] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:26.781686] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:26.762271] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:25.895808] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=330][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:26.763306] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=23] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:25.762529] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=40] finish to resize log external storage handler(current_ts=1755162323783517, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-08-14 09:05:25.354139] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=39][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:26.765484] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=20] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:26.782069] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16554] temporary file module use no blocks [2025-08-14 09:05:26.782076] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:26.782106] WDIAG ev_invoke_pending (ev.c:2136) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19825][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:26.782094] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:24.878600] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_timer.cpp:125) [734][T1001_ArcTimer][T1001][YB42AC4D0007-00063C495CC99C16-0-0] [lt=906785][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1900723, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1884359, ObMalloc#start=1, ObMalloc#end=16359, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x11406c8f 0x114063aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:25.762672] EDIAG [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [412][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=2011423][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1018484, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1002076, ObMalloc#start=1, ObMalloc#end=16403, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a5bbf2 0x94da542 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:26.439740] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:26.765525] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=992122] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:24.878600] INFO [ARCHIVE] do_thread_task_ (ob_archive_timer.cpp:125) [734][T1001_ArcTimer][T1001][YB42AC4D0007-00063C495CC99C16-0-0] [lt=906785] archive round not in doing status, just skip(key={incarnation:-1, dest_id:-1, round:-1}, state={status:"INVALID"}) [2025-08-14 09:05:26.176410] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:05:26.782300] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=604928] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:26.782345] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:26.782387] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:26.782426] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:05:26.782439] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:26.782451] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:26.767067] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=305] gc stale ls task succ [2025-08-14 09:05:26.773250] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=2000517] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:26.774290] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:25.762672] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [412][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=2011423] [sql nio session](*s={this:0x7f98f733a030, session_id:3221590255, trace_id:YB42AC4D0007-00063C49651A3232-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:257}, err:0, last_decode_time:1755161952736711, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1716, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:05:26.776985] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=21] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:26.782761] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=5763] temporary file module use no blocks [2025-08-14 09:05:26.782770] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:26.782781] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:26.782794] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623228216236, flush_io_finished_ret_:0, flush_io_finished_round_:26375, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:23.781438] EDIAG [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=1009820][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2996944, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2992449, ObMalloc#start=1, ObMalloc#end=4490, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x99721a1 0x997a72e 0x2006ed17 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:23.781438] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=1009820] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:26.775383] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=1002645] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:26.781938] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19692][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:26.789461] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:05:26.790284] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:26.797022] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=16] PNIO [ratelimit] time: 1755162326797020, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007591, add_bytes: 0 [2025-08-14 09:05:26.820832] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:26.927932] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:05:26.982333] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:27.080022] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=702701] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:27.183372] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:27.482582] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=341892][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:27.593650] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:27.721588] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=31] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:26.781694] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19393][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:26.782120] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=26] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623228204105, flush_io_finished_ret_:0, flush_io_finished_round_:26358, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:26.781849] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=18500] temporary file module use no blocks [2025-08-14 09:05:27.726219] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=942941] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:27.726249] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=29] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:27.726265] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623228200358, flush_io_finished_ret_:0, flush_io_finished_round_:26396, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:27.726319] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:27.728426] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4076] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:26.782936] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=2996944] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:27.732331] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:27.741008] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:27.751857] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:27.729283] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=944914] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:27.754613] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=25290] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:27.754629] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:26.882246] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:26.723694] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2800) [807][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=2949921] schedule next cache evict task(evict_interval=5000000) [2025-08-14 09:05:27.761290] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] [sql nio session](*s={this:0x7f98f72d0030, session_id:3221589415, trace_id:YB42AC4D0007-00063C4965FA43DC-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:245}, err:0, last_decode_time:1755161751714366, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1662, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:05:27.761328] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=38] [sql nio session](*s={this:0x7f98f72d5030, session_id:3221590200, trace_id:YB42AC4D0007-00063C4965FA43E0-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:248}, err:0, last_decode_time:1755161776777167, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1593, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:05:27.762293] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:27.762637] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=9] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:27.763104] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:27.763189] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1044] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:26.723694] EDIAG [SQL.PC] runTimerTask (ob_plan_cache.cpp:2800) [807][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=2949921][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1031846, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1031834, ObMalloc#start=2, ObMalloc#end=4, ALLOC_END=0, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x15365cdf 0x15365234 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [416]OB_MALLOC COST TOO MUCH TIME, cost_time=1026712, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1026710], seq:[0] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=991150, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=991143], seq:[0] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=991170, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=991165], seq:[0, 1] [2025-08-14 09:05:27.766111] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=19527] gc stale ls task succ [2025-08-14 09:05:27.767488] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=15515] gc stale ls task succ [2025-08-14 09:05:27.782648] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=9364] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:27.784128] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:27.786333] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:27.787304] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:27.789522] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:05:27.791295] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:27.763148] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:28.755835] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=991186][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:28.755864] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:27.774340] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=19668] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:28.760301] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=984467] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:28.760323] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=22] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:28.765266] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:05:28.765297] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=28] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:05:28.767395] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=19] gc stale ls task succ [2025-08-14 09:05:28.768715] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=17] gc stale ls task succ [2025-08-14 09:05:28.770019] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=694] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:28.770056] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=413] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:28.772869] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:28.772927] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=58][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=1045686, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=82, WASH_BLOCK_END=160, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1026733], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:27.182813] EDIAG [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1588444, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=4, FORMAT_END=542723, ObMalloc#start=0, ObMalloc#end=1045706, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1f9d7fcf 0x99b8be2 0x1f981833 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:27.182813] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:05:28.773742] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=1588444] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:28.773792] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:28.773837] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:28.773877] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:05:28.773891] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:28.773902] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:28.779407] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3196] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:28.781311] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=255] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:28.782285] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:28.783316] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2860] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:27.731963] EDIAG [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [726][T1001_ObXAHbWor][T1001][Y0-0000000000000000-0-0] [lt=960346][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1053004, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1052994, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b64d8ba 0x1b64d154 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:27.731963] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [726][T1001_ObXAHbWor][T1001][Y0-0000000000000000-0-0] [lt=960346] XA scheduler heartbeat task statistics(avg_time=1) [2025-08-14 09:05:28.788027] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=944] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:27.773983] EDIAG [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=995941][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1019959, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1019954, ObMalloc#start=0, ObMalloc#end=2, ALLOC_END=0, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x260b4cc1 0x260b9960 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:28.795353] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=11] gc stale ls task succ [2025-08-14 09:05:27.773983] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=995941] log compressor cycles once. (ret=0, cost_time=2050136, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25877913600, fast_delete_log_mode=false) [2025-08-14 09:05:28.797919] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=202] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:28.797960] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=168] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:05:28.798025] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=10] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:28.798166] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=380] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:28.803379] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:28.813447] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=202] PNIO [ratelimit] time: 1755162328813445, bytes: 0, bw: 0.000000 MB/s, add_ts: 1008810, add_bytes: 0 [2025-08-14 09:05:28.824813] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=47713] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:05:28.833333] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=2] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:05:28.876642] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:05:28.876705] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:28.876745] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:28.886517] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:28.898161] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:29.088689] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=221] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:29.721114] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5751] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:29.732620] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=470] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:28.876783] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:29.732779] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=854664] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:05:29.732826] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=41] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:29.732839] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:29.733208] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2646] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:29.740404] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2440] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:29.747710] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=2274] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:29.748046] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=2136] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:29.744973] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=215] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:29.754007] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1640] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:29.738208] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=16999][errcode=-4015] clock out of order(cur_ts=1755162328838969, cur_ts_=1755162328717605, delta=121364) [2025-08-14 09:05:29.756036] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:29.759520] WDIAG [RPC.OBRPC] in_black (ob_net_keepalive.cpp:257) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:29.767610] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:29.767655] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:29.767690] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:29.769389] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=25] gc stale ls task succ [2025-08-14 09:05:29.769428] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=23] gc stale ls task succ [2025-08-14 09:05:29.776256] WDIAG [SHARE] process_task (ob_event_history_table_operator.cpp:328) [337][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=26897][errcode=-4012] execute sql failed(sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1755161909716139), 'FAILURE_DETECTOR', 'clog disk hang event', 'FAILURE_MODULE', 'LOG', 'FAILURE_TYPE', 'PROCESS HANG', 'AUTO_RECOVER', 'False', '', '', '', '172.77.0.7', 2882), ret=-4012) [2025-08-14 09:05:29.776298] WDIAG [SHARE] process (ob_event_history_table_operator.cpp:172) [337][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4012] process_task failed(ret=-4012, ret="OB_TIMEOUT", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1755161909716139), 'FAILURE_DETECTOR', 'clog disk hang event', 'FAILURE_MODULE', 'LOG', 'FAILURE_TYPE', 'PROCESS HANG', 'AUTO_RECOVER', 'False', '', '', '', '172.77.0.7', 2882), is_delete=false, create_time_=1755161911739994, exec_tenant_id=1) [2025-08-14 09:05:29.781095] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:29.781744] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:29.783163] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:29.783504] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:29.783652] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:29.784253] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:29.785272] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:29.785306] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=57076] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:05:29.785345] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:29.788142] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:29.789266] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:29.789623] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:29.791355] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:29.795418] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=18] gc stale ls task succ [2025-08-14 09:05:29.798322] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=13] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:29.807587] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:29.813530] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=67451] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:29.813564] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27] temporary file module use no blocks [2025-08-14 09:05:29.813581] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:29.813596] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:29.813612] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623231253263, flush_io_finished_ret_:0, flush_io_finished_round_:26360, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:29.819886] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=27] PNIO [ratelimit] time: 1755162329819884, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006439, add_bytes: 0 [2025-08-14 09:05:29.833006] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:05:29.833074] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:29.833124] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:29.833170] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:29.833223] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:05:29.833258] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=29] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:29.833275] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:29.854119] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:29.992389] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:29.999900] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:30.721248] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=36] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:30.725646] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:29.785375] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:30.733217] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:29.781196] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=55722] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:05:30.736244] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=14] [COMPACTION DAG STATUS] (start_time=1755162314768278, end_time=1755162330736241, dag_status={}) [2025-08-14 09:05:29.783766] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=1005893] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:05:29.783909] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=1003759] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:05:30.740432] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:29.798303] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=13] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:05:30.729886] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=943073][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=946996, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=946989, ALLOC_CHUNK_END=1], seq:[0, 1] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=947013, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=947011], seq:[0, 1] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=947006, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=946962, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [552]OB_MALLOC COST TOO MUCH TIME, cost_time=931808, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=931759, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [552]OB_MALLOC COST TOO MUCH TIME, cost_time=931931, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=931829, WASH_BLOCK_END=97, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=2], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:29.813654] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=67843] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:30.747037] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=931966] temporary file module use no blocks [2025-08-14 09:05:30.747049] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:30.747062] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:30.747076] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623231253284, flush_io_finished_ret_:0, flush_io_finished_round_:26398, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:30.748162] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=14] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:26.729150] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [865][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=1012219] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136805728713000, v:0}, clog_base_lsn:{lsn:7247314944}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1755161030946590000, v:0}, tablet_change_checkpoint_scn:{val:1755154712438549001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:49, palf_env_:0x7f991c5e8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-08-14 09:05:26.729150] EDIAG [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [865][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=1012219][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4013147, owner: ObLog, click_count: 5, time dist:[FORMAT_END=4003018, ObMalloc#start=1, ObMalloc#end=10120, ALLOC_END=1, APPEND_END=7], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b51df0f 0x986bcde 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [736]OB_MALLOC COST TOO MUCH TIME, cost_time=948717, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=947016, WASH_BLOCK_END=478, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1219], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:30.089943] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=30] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:30.506883] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:29.798409] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=11] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:30.745265] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=157] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:30.748616] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=3342] temporary file module use no blocks [2025-08-14 09:05:30.748627] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:30.748647] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=18] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:30.748661] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623231237880, flush_io_finished_ret_:0, flush_io_finished_round_:26378, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:30.745340] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8540] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:30.746185] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:29.753547] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1001, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:8571658, max_lsn:7323687369, max_scn:{val:1755161749741193002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:7278610047}, reuse_lsn:{lsn:7323686131}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:7281744329}}, last_submit_lsn:{lsn:7323686131}, last_submit_end_lsn:{lsn:7323687369}, last_submit_log_id:8571658, last_submit_log_pid:49, max_flushed_lsn:{lsn:7323686009}, max_flushed_end_lsn:{lsn:7323686131}, max_flushed_log_pid:49, committed_end_lsn:{lsn:7323686009}, last_slide_log_id:8571656, last_slide_scn:{val:1755161747742105001, v:0}, last_slide_lsn:{lsn:7323685887}, last_slide_end_lsn:{lsn:7323686009}, last_slide_log_pid:49, last_slide_log_accum_checksum:2505458373, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f990112a530}, [StateMgr]={this:0x7f990112e9b0, self:"172.77.0.7:2882", palf_id:1001, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"PENDING", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:49}, leader:"0.0.0.0:0", leader_epoch:-1, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:7323687369}, scan_disk_log_finished:true, last_check_start_id:8571657, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161767744317, palf_role_change_cb:0x7f9901135470, allow_vote:true}, [ConfigMgr]={palf_id:1001, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:49, prev:{version:1, config:{config_version:{proposal_id:48, config_seq:47}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:49, config_seq:48}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_mode_pid:49}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_end_lsn:{lsn:7323685240}, prev_mode_pid:49}, reconfig_barrier:{prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_end_lsn:{lsn:7323685240}, prev_mode_pid:49}, persistent_config_version:{proposal_id:49, config_seq:48}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755161739755312, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755161739755312, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f990112bc70}, [ModeMgr]={palf_id:1001, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, accepted_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1001, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1753069671378640001, v:0}, base_lsn_for_block_gc:{lsn:7247314944}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:49}, log_config_meta:{version:3, proposal_id:49, prev:{version:1, config:{config_version:{proposal_id:48, config_seq:47}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:49, config_seq:48}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_mode_pid:49}, log_snapshot_meta:{version:2, base_lsn:{lsn:7247314944}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1753069671378640000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}}, log_meta_storage:{log_tail:{lsn:765952}, readable_log_tail:{lsn:765952}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7381524480}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/meta", dir_fd:232, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66338816, block_header_serialize_buf_:0x7f9901130bb8, flashback_version:0}, log_storage:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, palf_epoch:2, last_purge_throttling_ts:1755161769777679, this:0x7f990112f0f0}, [Reconfirm]={palf_id:1001, self:"172.77.0.7:2882", state:"WAITING_LOG_FLUSHED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:1755161769777679, this:0x7f990112eb30}) [2025-08-14 09:05:30.748876] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=993812] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [2025-08-14 09:05:30.746466] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=1] get_min_unreplayed_log_info(lsn={lsn:1604088821}, scn={val:1755100948296325001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873479, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:12492179, ret_code_:0}}, next_to_submit_lsn_:{lsn:1604088821}, next_to_submit_scn_:{val:1755100948296325001, v:0}, base_lsn_:{lsn:1409200128}, base_scn_:{val:1753840802540914000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:1604088821}, end_lsn:{lsn:1604088821}, read_buf:{buf_len_:3747840, buf_:0x7f98f6a05000}, block_size:67104768, log_storage_:0x7f990bfebbf0, log_storage_:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:42, read_io_size:147709952, read_disk_cost_ts:1061450, start_lsn:{lsn:1409200128}}}}}}) [2025-08-14 09:05:30.748923] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1085) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=2450] replay status is not follower(min_unreplayed_lsn={lsn:1604088821}, base_lsn={lsn:1409200128}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873479, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:12492179, ret_code_:0}}, next_to_submit_lsn_:{lsn:1604088821}, next_to_submit_scn_:{val:1755100948296325001, v:0}, base_lsn_:{lsn:1409200128}, base_scn_:{val:1753840802540914000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:1604088821}, end_lsn:{lsn:1604088821}, read_buf:{buf_len_:3747840, buf_:0x7f98f6a05000}, block_size:67104768, log_storage_:0x7f990bfebbf0, log_storage_:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:42, read_io_size:147709952, read_disk_cost_ts:1061450, start_lsn:{lsn:1409200128}}}}}}) [2025-08-14 09:05:30.748985] INFO [CLOG] operator() (ob_log_replay_service.cpp:1678) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=61] get_replay_process success(id={id:1}, submitted_log_size=194888693, unsubmitted_log_size=0, replayed_log_size=194888693, unreplayed_log_size=0) [2025-08-14 09:05:30.749001] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=12] get_min_unreplayed_log_info(lsn={lsn:7323685240}, scn={val:1755100946796590002, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873485, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:8946764, ret_code_:0}}, next_to_submit_lsn_:{lsn:7323685240}, next_to_submit_scn_:{val:1755100946796590002, v:0}, base_lsn_:{lsn:7180210176}, base_scn_:{val:1753840803166611000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:7323685240}, end_lsn:{lsn:7323685240}, read_buf:{buf_len_:3747840, buf_:0x7f98f7a05000}, block_size:67104768, log_storage_:0x7f9901131bf0, log_storage_:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:28, read_io_size:98508800, read_disk_cost_ts:645339, start_lsn:{lsn:7180210176}}}}}}) [2025-08-14 09:05:30.749060] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1085) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=58] replay status is not follower(min_unreplayed_lsn={lsn:7323685240}, base_lsn={lsn:7180210176}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873485, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:8946764, ret_code_:0}}, next_to_submit_lsn_:{lsn:7323685240}, next_to_submit_scn_:{val:1755100946796590002, v:0}, base_lsn_:{lsn:7180210176}, base_scn_:{val:1753840803166611000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:7323685240}, end_lsn:{lsn:7323685240}, read_buf:{buf_len_:3747840, buf_:0x7f98f7a05000}, block_size:67104768, log_storage_:0x7f9901131bf0, log_storage_:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:28, read_io_size:98508800, read_disk_cost_ts:645339, start_lsn:{lsn:7180210176}}}}}}) [2025-08-14 09:05:30.749098] INFO [CLOG] operator() (ob_log_replay_service.cpp:1678) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=38] get_replay_process success(id={id:1001}, submitted_log_size=143475064, unsubmitted_log_size=0, replayed_log_size=143475064, unreplayed_log_size=0) [2025-08-14 09:05:30.754024] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:30.756012] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:30.730091] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:288) [732][T1001_ArcFetche][T1001][YB42AC4D0007-00063C495CA99C16-0-0] [lt=862544] ObArchiveFetcher is running(thread_index=0) [2025-08-14 09:05:30.746726] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:233) [529][T1_RFLWorker][T1][YB42AC4D0007-00063C495B199C16-0-0] [lt=5803] ObRemoteFetchWorker is running(thread_index=0) [2025-08-14 09:05:30.757393] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=29] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:30.757425] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:30.748899] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=19] current log_task status(palf_id=1001, self="172.77.0.7:2882", log_id=8571657, log_task={header:{begin_lsn:{lsn:7323686009}, end_lsn:{lsn:7323686131}, log_id:8571657, min_scn:{val:1755161749741193000, v:0}, max_scn:{val:1755161749741193000, v:0}, data_len:66, proposal_id:49, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:7323686009}, data_checksum:2222604892, accum_checksum:1531236791, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1755161749745909, freeze_ts:1755161749745914, submit_ts:1755161749745940, flushed_ts:1755161785758706, gen_to_freeze cost time:5, gen_to_submit cost time:31, submit_to_flush cost time:36012766}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [2025-08-14 09:05:30.749125] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:174) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=9] dump tenant replay process(tenant_id=1002, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=322, replayed_log_size(MB)=322, last_submitted_log_size(MB)=322, last_replayed_log_size(MB)=322, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-08-14 09:05:30.759186] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10044][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:30.761838] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:30.770345] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=25] gc stale ls task succ [2025-08-14 09:05:30.770391] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=12] gc stale ls task succ [2025-08-14 09:05:30.779256] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:30.780718] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16732] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:30.781316] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:30.782744] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:30.783254] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:30.783932] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:30.788261] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:30.790140] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:30.790503] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:30.791375] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:30.797142] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=19] gc stale ls task succ [2025-08-14 09:05:30.797312] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:30.798410] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=18] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:30.800742] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:30.808402] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:30.827503] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=26] PNIO [ratelimit] time: 1755162330827501, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007617, add_bytes: 0 [2025-08-14 09:05:30.834032] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:05:30.834100] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:30.834149] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:30.834203] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:30.834264] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:05:30.834282] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:30.834299] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:30.998512] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:31.018829] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:31.148976] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=657592] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:31.557331] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=241298][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:31.699513] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=5567][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:31.722266] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:31.725538] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:30.746703] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161101770923, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755161572801635) [2025-08-14 09:05:31.733097] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:30.797193] INFO [CLOG] update_upstream (ob_remote_location_adaptor.cpp:76) [829][T1002_LogRessvr][T1002][YB42AC4D0007-00063C495D199C16-0-0] [lt=4034473] primary tenant, just skip [2025-08-14 09:05:30.778322] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=942140] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:05:31.736898] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=957125][errcode=-4076] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:31.740411] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:31.745631] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3409] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:31.745668] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=2616] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:31.745702] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=944645] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:05:31.748066] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=19] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:31.748180] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=1019959] log compressor cycles once. (ret=0, cost_time=2240, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25879027712, fast_delete_log_mode=false) [2025-08-14 09:05:31.745847] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [872][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E799C16-0-0] [lt=944252] ObArchiveSender is running(thread_index=1) [2025-08-14 09:05:31.748856] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=948739] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:30.787759] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [339][EventTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:30.778254] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=23596] blacklist_loop exec finished(cost_time=29, is_enabled=true, send_cnt=0) [2025-08-14 09:05:31.753514] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=37] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:31.753550] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=22] temporary file module use no blocks [2025-08-14 09:05:31.753560] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:31.753572] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:31.753587] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623233186882, flush_io_finished_ret_:0, flush_io_finished_round_:26399, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:31.753631] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=38] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:31.753644] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] temporary file module use no blocks [2025-08-14 09:05:31.753651] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:31.753660] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=9] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:31.753670] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623233188467, flush_io_finished_ret_:0, flush_io_finished_round_:26379, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:31.753984] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:31.755978] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:31.757420] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:31.759470] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:31.763524] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:31.766788] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:31.766911] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [807][T1002_TimerWK3_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=0] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):258, sequence:1}) [2025-08-14 09:05:31.766950] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:97) [807][T1002_TimerWK3_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=30] BlockGCTimerTask success(ret=0, cost_time_us=65, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.77.0.7:2882", log_dir:"/root/ob/observer/store/clog/tenant_1002", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):258, sequence:1}, log_alloc_mgr_:{flying_log_task:1, flying_meta_task:2}}) [2025-08-14 09:05:31.771666] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=33] gc stale ls task succ [2025-08-14 09:05:31.771707] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=14] gc stale ls task succ [2025-08-14 09:05:31.726286] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [865][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=4013147] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161143777799003, v:0}", last_lsn={lsn:7323684499}, last_gts={val:1755161749741193000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:05:31.779252] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:31.781285] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:31.781736] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:31.782815] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:31.783352] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=50] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:31.787663] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:31.788385] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:31.791091] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:31.791357] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:31.792705] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:31.797226] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=21] gc stale ls task succ [2025-08-14 09:05:31.797523] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:31.798495] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=20] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:31.809023] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:31.813775] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=30] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:31.813807] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27] temporary file module use no blocks [2025-08-14 09:05:31.813849] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=40] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:31.813867] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:31.813882] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623233253471, flush_io_finished_ret_:0, flush_io_finished_round_:26362, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:31.814510] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:31.835102] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=26] PNIO [ratelimit] time: 1755162331835099, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007598, add_bytes: 0 [2025-08-14 09:05:31.836983] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:05:31.839273] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:05:31.839331] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:31.839373] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:31.839416] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:31.839456] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:05:31.839479] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:31.839492] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:31.878529] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=49882][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:05:31.880180] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=51586][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:05:31.889017] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=56][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:32.022514] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:32.091610] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:32.151500] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=35] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:32.247539] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:32.559470] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=41][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:32.721257] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:36.753315] INFO [MDS] flush (mds_table_mgr.cpp:184) [15295][T1002_TimerWK9_Flush][T1002][YB42AC4D0007-00063C497ED99C6C-0-0] [lt=78898715] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:60379}, rec_scn:{val:1755136836462665002, v:0}}, {tablet_id:{id:50380}, rec_scn:{val:1755136836462665003, v:0}}, {tablet_id:{id:545}, rec_scn:{val:1755136836462665004, v:0}}, {tablet_id:{id:233}, rec_scn:{val:1755136836462665005, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1755136836462665006, v:0}}, {tablet_id:{id:50192}, rec_scn:{val:1755136836462665007, v:0}}, {tablet_id:{id:60259}, rec_scn:{val:1755136836462665008, v:0}}, {tablet_id:{id:188}, rec_scn:{val:1755136836462665009, v:0}}, {tablet_id:{id:224}, rec_scn:{val:1755136836462665010, v:0}}, {tablet_id:{id:308}, rec_scn:{val:1755136836462665011, v:0}}, {tablet_id:{id:369}, rec_scn:{val:1755136836462665013, v:0}}, {tablet_id:{id:493}, rec_scn:{val:1755136836462665014, v:0}}, {tablet_id:{id:101011}, rec_scn:{val:1755136836462665015, v:0}}, {tablet_id:{id:521}, rec_scn:{val:1755136836462665016, v:0}}, {tablet_id:{id:101109}, rec_scn:{val:1755136836462665017, v:0}}, {tablet_id:{id:284}, rec_scn:{val:1755136836462665018, v:0}}, {tablet_id:{id:50149}, rec_scn:{val:1755136836462665019, v:0}}, {tablet_id:{id:307}, rec_scn:{val:1755136836462665020, v:0}}, {tablet_id:{id:50240}, rec_scn:{val:1755136836462665021, v:0}}, {tablet_id:{id:50355}, rec_scn:{val:1755136836462665022, v:0}}, {tablet_id:{id:182}, rec_scn:{val:1755136836462665023, v:0}}, {tablet_id:{id:101098}, rec_scn:{val:1755136836462665024, v:0}}, {tablet_id:{id:419}, rec_scn:{val:1755136836462665025, v:0}}, {tablet_id:{id:60297}, rec_scn:{val:1755136836462665026, v:0}}, {tablet_id:{id:60305}, rec_scn:{val:1755136836462665027, v:0}}, {tablet_id:{id:378}, rec_scn:{val:1755136836462665028, v:0}}, {tablet_id:{id:101036}, rec_scn:{val:1755136836462665029, v:0}}, {tablet_id:{id:60005}, rec_scn:{val:1755136836462665030, v:0}}, {tablet_id:{id:60130}, rec_scn:{val:1755136836462665031, v:0}}, {tablet_id:{id:101064}, rec_scn:{val:1755136836462665032, v:0}}, {tablet_id:{id:101080}, rec_scn:{val:1755136836462665033, v:0}}, {tablet_id:{id:50424}, rec_scn:{val:1755136836462665034, v:0}}, }, max_consequent_callbacked_scn={val:1755161749741193000, v:0}, *this={this:0x7f9900461150, is_inited:true, freezing_scn:{val:1755136801441250000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136836462665002, v:0}, clog_base_lsn:{lsn:1543409664}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1755161030946590000, v:0}, tablet_change_checkpoint_scn:{val:1753850176390756001, v:0}, all_id_meta:{id_meta:[{limited_id:1755161762748684000, latest_log_ts:{val:1755161746772281001, v:0}}, {limited_id:9000001, latest_log_ts:{val:1755135652743621004, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:50, palf_env_:0x7f991c5e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-08-14 09:05:32.727378] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:32.732945] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [865][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=45992] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1001}) [2025-08-14 09:05:32.732983] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [865][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=26] tx gc loop thread is running(MTL_ID()=1002) [2025-08-14 09:05:32.732994] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [865][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] try gc retain ctx [2025-08-14 09:05:32.733012] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [865][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=17] try to retry start_working [2025-08-14 09:05:32.733019] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:125) [865][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=7] try to adjust log cb pool [2025-08-14 09:05:32.739136] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4076] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:32.740639] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:32.745501] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9481] LogLoopThread round_cost_time(us)(round_cost_time=34000382) [2025-08-14 09:05:32.733053] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:32.763003] WDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=29903][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:32.746422] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:32.763027] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=16579][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:32.774273] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:05:32.774313] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=11] gc stale ls task succ [2025-08-14 09:05:32.746373] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:32.774423] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28006][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:32.779255] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:32.781328] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:32.781445] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:32.784275] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:32.784315] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:32.788343] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:32.788476] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:32.791347] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:32.791716] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:32.797298] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=38] gc stale ls task succ [2025-08-14 09:05:32.799033] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=18] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:32.800743] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:32.811897] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:32.813991] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:32.814021] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=26] temporary file module use no blocks [2025-08-14 09:05:32.814029] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:32.814047] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:32.814066] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=18] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623234253679, flush_io_finished_ret_:0, flush_io_finished_round_:26363, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:32.842717] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=25] PNIO [ratelimit] time: 1755162332842715, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007616, add_bytes: 0 [2025-08-14 09:05:32.843408] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:05:32.843463] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:32.843505] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:32.843547] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:32.843585] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:05:32.843597] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:32.843607] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:32.877464] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:33.023744] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:33.092559] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:33.152193] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=27] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:33.254869] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:33.293843] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:33.563575] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=49][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:33.721266] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:33.731024] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:33.734490] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:32.763147] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=980781] get_max_applied_scn(scn={val:1755161547738516001, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:50, palf_committed_end_lsn_:{lsn:5282737655}, palf_committed_end_scn_:{val:1755161547738516001, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161547738516001, v:0}}) [2025-08-14 09:05:33.734558] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=969929] get_min_unreplayed_log_info(lsn={lsn:5282737655}, scn={val:1755100947825033001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161624848121, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:13063726, ret_code_:0}}, next_to_submit_lsn_:{lsn:5282737655}, next_to_submit_scn_:{val:1755100947825033001, v:0}, base_lsn_:{lsn:4831543296}, base_scn_:{val:1755063445733337001, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7605000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:5282737655}, end_lsn:{lsn:5282737655}, read_buf:{buf_len_:3747840, buf_:0x7f98f7605000}, block_size:67104768, log_storage_:0x7f993e529bf0, log_storage_:{log_tail:{lsn:5282737655}, readable_log_tail:{lsn:5282737655}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:5234171904}, min_scn:{val:1755152244802535004, v:0}, curr_block_id:78, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:206, min_block_id:42, max_block_id:79, curr_writable_block_id:78}, logical_block_size_:67104768, curr_block_writable_size_:18539017, block_header_serialize_buf_:0x7f993e52a8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:85, read_io_size:301367296, read_disk_cost_ts:2364604, start_lsn:{lsn:4831543296}}}}}}) [2025-08-14 09:05:32.733029] INFO [STORAGE.TRANS] refresh_elr_tenant_config_ (ob_tx_elr_util.cpp:42) [865][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=7] refresh tenant config success(tenant_id=1002, *this={last_refresh_ts:1755162332730753, can_tenant_elr:true}) [2025-08-14 09:05:33.745257] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=10212] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:33.748106] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=8041] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:32.772289] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [807][T1002_TimerWK3_TabletShell][T1002][Y0-0000000000000000-0-0] [lt=0] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-08-14 09:05:33.758616] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=410] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:33.758659] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=32] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:33.758682] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=16] temporary file module use no blocks [2025-08-14 09:05:33.758692] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:33.758704] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:33.758730] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=25] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623235193499, flush_io_finished_ret_:0, flush_io_finished_round_:26401, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:33.759441] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1124] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:33.734976] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:33.759823] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=1048] temporary file module use no blocks [2025-08-14 09:05:33.759836] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:33.759849] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:33.759862] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623235193523, flush_io_finished_ret_:0, flush_io_finished_round_:26381, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:33.761418] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=202] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:33.761455] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=55] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:05:32.733029] EDIAG [STORAGE.TRANS] refresh_elr_tenant_config_ (ob_tx_elr_util.cpp:42) [865][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1003842, owner: ObLog, click_count: 7, time dist:[FORMAT_END=1003827, ObMalloc#start=2, ObMalloc#end=4, ALLOC_END=0, APPEND_END=8, ALLOC_BLOCK#start=6, ALLOC_BLOCK#end=23602], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b878a47 0x1b8787a8 0x1b878197 0x986bac9 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:33.762098] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [865][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=1003842] get_max_applied_scn(scn={val:1755161749741193000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:50, palf_committed_end_lsn_:{lsn:1604090090}, palf_committed_end_scn_:{val:1755161749741193000, v:0}, last_check_scn_:{val:1755161749741193001, v:0}, max_applied_cb_scn_:{val:1755161749741193000, v:0}}) [2025-08-14 09:05:33.769372] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [807][T1002_TimerWK3_TabletShell][T1002][Y0-0000000000000000-0-0] [lt=975005] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=2, times=1262) [2025-08-14 09:05:33.769431] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=54][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:33.769475] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [807][T1002_TimerWK3_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=0] ====== check clog disk timer task ====== [2025-08-14 09:05:33.769494] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [807][T1002_TimerWK3_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=10] get_disk_usage(ret=0, capacity(MB):=24883, used(MB):=8642) [2025-08-14 09:05:33.775336] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=28] gc stale ls task succ [2025-08-14 09:05:33.779085] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:33.781118] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:33.782472] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:33.784613] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:33.788590] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:33.792325] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=60] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:33.799510] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=25] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:33.843378] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=2539] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:05:33.850333] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=24] PNIO [ratelimit] time: 1755162333850332, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007617, add_bytes: 0 [2025-08-14 09:05:34.155686] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=23] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:34.626611] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=58][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:34.721319] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:34.726257] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:33.747331] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=29] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:33.798502] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=22] gc stale ls task succ [2025-08-14 09:05:34.732257] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [858]OB_MALLOC COST TOO MUCH TIME, cost_time=967352, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=967350], seq:[0] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=967494, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=42, WASH_BLOCK_END=73, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=967374], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:34.744172] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=383] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:34.744310] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=220] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:33.775366] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=10] gc stale ls task succ [2025-08-14 09:05:34.747251] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=17] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:34.760436] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=277] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:34.740829] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=16941][errcode=-4015] clock out of order(cur_ts=1755162333778797, cur_ts_=1755162333750315, delta=28482) [2025-08-14 09:05:33.771281] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [865][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=20] finish update min start scn(max_decided_scn={val:1755161749741193000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1755161277810819002, v:0}, keep_alive_scn_:{val:1755161277810819002, v:0}, update_ts_:1755162333750315}) [2025-08-14 09:05:34.775797] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=967524] gc stale ls task succ [2025-08-14 09:05:34.777553] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=87] get wrs ts(ls_id={id:1}, delta=786011799, log_service_ts=1755161547738516, min_tx_service_ts=4611686018427387, timestamp={val:1755161547738516001, v:0}) [2025-08-14 09:05:34.777592] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161493768447002, v:0}", last_lsn={lsn:5282736757}, last_gts={val:1755161547738516000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:05:34.778302] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:34.745259] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:33.782313] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:34.778385] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=994566][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:34.754896] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:34.778416] WDIAG ev_invoke_pending (ev.c:2136) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23484][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:34.760370] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=722] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:34.778445] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18047][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:34.760458] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=283] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:34.778474] WDIAG ev_invoke_pending (ev.c:2136) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17988][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:34.761220] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=1691] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:05:34.753894] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=9405] finish to resize log external storage handler(current_ts=1755162333784695, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-08-14 09:05:34.778767] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=24830] snapshot_traffic_info once [2025-08-14 09:05:34.729588] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=980773] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:34.778841] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=49175] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:34.778859] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=18] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:34.778868] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:34.778877] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:34.778886] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:34.778897] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:34.778905] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:34.763266] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:34.779536] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16245][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:34.781266] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:34.782257] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:34.782267] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:34.783255] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:34.787543] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:34.789285] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:34.795215] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:34.797540] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:34.804356] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=22] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:34.805980] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:34.815680] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:34.819297] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=31] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:34.819333] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=29] temporary file module use no blocks [2025-08-14 09:05:34.819345] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:34.819365] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=18] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:34.819386] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=21] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623236253917, flush_io_finished_ret_:0, flush_io_finished_round_:26365, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:34.834529] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:34.848749] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:05:34.848777] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:34.857961] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=28] PNIO [ratelimit] time: 1755162334857959, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007627, add_bytes: 0 [2025-08-14 09:05:34.860783] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:05:34.860843] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:34.860899] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:34.899135] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:34.994737] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:35.010321] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:35.158704] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=19] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:35.568597] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:35.726911] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:35.727221] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:34.860942] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:35.732490] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:05:34.899727] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=954574, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=954572], seq:[0] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=958667, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=47, WASH_BLOCK_END=61, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=954594], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:34.791272] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:35.747395] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=953801] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:05:35.748247] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=953911] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:05:35.740227] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:35.754385] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14135][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:34.804265] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=931461] gc stale ls task succ [2025-08-14 09:05:34.780244] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:140) [807][T1002_TimerWK3_CompactionRefre][T1002][YB42AC4D0007-00063C495F8A196A-0-0] [lt=0] InfoPoolResizeTask(cost_ts=21623) [2025-08-14 09:05:35.757591] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25120] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:35.757610] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:35.734556] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162331728970}, version_delta=787996038) [2025-08-14 08:59:35.768588] EDIAG [STORAGE] init (ob_multiple_scan_merge.cpp:72) [15228][T1_TimerWK11_MediumLoop][T1][YB42AC4D0007-00063C497E399C25-0-0] [lt=12][errcode=-4012] LOGGER COST TOO MUCH TIME, cost: 1955061, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1955051, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1d7f3a76 0x94b4a18 0x9766028 0x9765ae5 0x976579e 0x9764de3 0x975ab8f 0x97f7c68 0x97f78b3 0x9800989 0x980049e 0x97fe9af 0x1c353d12 0x1aee74b9 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:35.751429] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=958705][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:35.757528] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][YB42AC4D0007-00063C495F8A196A-0-0] [lt=975805][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:35.758364] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:35.758423] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:35.758467] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:35.758495] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:35.734339] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:35.758538] WDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24161][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:34.780128] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [1026][TabletLSAUp1][T0][YB42AC4D0007-00063C496DC99CA7-0-0] [lt=86870857][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 08:58:29.748157] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:943) [15308][T1002_TimerWK11_LSMetaCh][T1001][YB42AC4D0007-00063C497F399C2E-0-0] [lt=45][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f98a69b1a98, id:3221590844, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1001, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f98db469c60}) [2025-08-14 09:05:35.759203] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13924][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:35.759525] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [567][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23112] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1755161547738516001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=788020998, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=253248, succ_cluster_heartbeat_count=2213, cluster_heartbeat_interval=50000, local_cluster_version={val:1755161000606550000, v:0}, local_cluster_delta=1335152964, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:05:35.759328] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=14016][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:00:07.855374] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_tenant_weak_read_service.cpp:468) [960][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8EFD-0-0] [lt=0][errcode=-4341] process cluster heartbeat rpc: self is not in cluster service(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id_=1002, svr="172.77.0.7:2882", version={val:1755161758221798000, v:0}, valid_part_count=0, total_part_count=2, generate_timestamp=1755162007855251) [2025-08-14 09:05:35.750470] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=11] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:35.761671] WDIAG ev_invoke_pending (ev.c:2136) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=11183][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:35.747401] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=953501] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:05:35.763702] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:35.763742] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=35] temporary file module use no blocks [2025-08-14 09:05:35.763759] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=16] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:35.763772] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:35.763787] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623237200275, flush_io_finished_ret_:0, flush_io_finished_round_:26403, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:35.763852] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=23] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:35.763866] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] temporary file module use no blocks [2025-08-14 09:05:35.763873] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:35.763884] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:35.763895] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623237200382, flush_io_finished_ret_:0, flush_io_finished_round_:26383, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:35.775863] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=14] gc stale ls task succ [2025-08-14 09:05:35.777154] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=419] gc stale ls task succ [2025-08-14 09:05:35.778830] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=17496] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:05:35.778898] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=18222] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:35.779241] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:35.781262] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:35.782228] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:35.783258] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:35.789853] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6339] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:35.795982] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:35.806293] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=20] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:35.808258] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=950215] gc stale ls task succ [2025-08-14 09:05:35.808290] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:35.812743] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=928719, tenant_id=500, label=CommonArray, ctx_id=0, prio=0, size=7936, [2025-08-14 08:58:29.748157] EDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:943) [15308][T1002_TimerWK11_LSMetaCh][T1001][YB42AC4D0007-00063C497F399C2E-0-0] [lt=45][errcode=-4283] LOGGER COST TOO MUCH TIME, cost: 425367829, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=2, ALLOC_BLOCK#end=1, FORMAT_END=1975042, ObMalloc#start=1, ObMalloc#end=9657, ALLOC_END=423383122, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x148dcafc 0x93403a6 0x933e2ee 0x93363f3 0x932e8ab 0x96ac524 0x96a672d 0x96a5033 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x1eeb57d8 0x1eeb4f81 0x1eed9495 0x1eed9090 0x99deeb9 0x12d3af34 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [399]OB_MALLOC COST TOO MUCH TIME, cost_time=972157, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=972150], seq:[0] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=972277, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=41, WASH_BLOCK_END=55, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=972176], seq:[0, 1, 2, 3, 4] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=935513, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=17, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=935460, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3, 4, 5, 6, 7] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=898272, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=898214, ALLOC_CHUNK_END=19], seq:[0, 1, 2, 3, 4, 5, 6, 7] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=898341, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=898289, WASH_BLOCK_END=52], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=891416, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=891373, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3, 4, 5, 6, 7] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=891438, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=891429, WASH_BLOCK_END=9], seq:[0, 1, 2] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=799365, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=799321, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3, 4, 5, 6, 7] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=799425, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=799417, WASH_BLOCK_END=8], seq:[0, 1, 2] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=759842, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=759788, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=759862, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=759852, WASH_BLOCK_END=10], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=744798, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=744761, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3, 4, 5, 6, 7] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=744823, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=744813, WASH_BLOCK_END=10], seq:[0, 1, 2] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=730420, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=730370, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=730439, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=730431, WASH_BLOCK_END=8], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=596856, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=596813, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=596874, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=596866, WASH_BLOCK_END=8], seq:[0, 1, 2] [2025-08-14 09:05:35.783285] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:36.757060] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=972301][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:36.161722] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=32] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:36.747302] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:36.766076] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:36.751775] WDIAG [STORAGE] init_and_open_multiple_scan_merge (ob_mds_row_iterator.cpp:315) [15228][T1_TimerWK11_MediumLoop][T1][YB42AC4D0007-00063C497E399C25-0-0] [lt=1955061][errcode=-4012] fail to init multiple scan merge(ret=-4012) [2025-08-14 09:05:36.769470] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=25652][errcode=-4015] clock out of order(cur_ts=1755162336769469, cur_ts_=1755162336735177, delta=34292) [2025-08-14 09:05:36.775939] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=25] gc stale ls task succ [2025-08-14 09:05:36.777177] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=18] gc stale ls task succ [2025-08-14 09:05:36.778953] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=16] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:36.779260] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:36.780606] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:36.781158] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:36.781979] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=55] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [397]OB_MALLOC COST TOO MUCH TIME, cost_time=934295, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=42, WASH_BLOCK_END=934253], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=632589, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=66, WASH_BLOCK_END=632523], seq:[0, 1, 2] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=930911, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=34, WASH_BLOCK_END=930877], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=466226, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=466185], seq:[0, 1, 2] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=451900, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=51, WASH_BLOCK_END=451848], seq:[0, 1, 2] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=917229, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=71, WASH_BLOCK_END=917158], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=916008, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=43, WASH_BLOCK_END=915965], seq:[0, 1, 2] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=911438, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=50, WASH_BLOCK_END=911388], seq:[0, 1, 2] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=865496, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=49, WASH_BLOCK_END=865446], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=852729, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=60, WASH_BLOCK_END=852669], seq:[0, 1, 2] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=429468, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=50, WASH_BLOCK_END=429418], seq:[0, 1, 2] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=917211, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=46, WASH_BLOCK_END=917164], seq:[0, 1, 2] [2025-08-14 09:05:35.776534] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [339][EventTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1948805, owner: ObLog, click_count: 8, time dist:[FORMAT_END=1937030, ObMalloc#start=1, ObMalloc#end=10913, ObMalloc#start=18, ObMalloc#end=35, ObMalloc#start=477, ObMalloc#end=29, FREE_END=302], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [400]OB_MALLOC COST TOO MUCH TIME, cost_time=106423, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=41, WASH_BLOCK_END=106382], seq:[0, 1, 2] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=941371, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=46, WASH_BLOCK_END=941324], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=994748, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=994746], seq:[0] [2025-08-14 09:05:34.744253] EDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3017403, , BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0xfc0e73d 0x993f560 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:36.755439] EDIAG [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=971237][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1018175, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1017494, ObMalloc#start=1, ObMalloc#end=66, ObMalloc#start=23, ObMalloc#end=44, FREE_END=547], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1fce4e91 0x1fce455d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [472]OB_MALLOC COST TOO MUCH TIME, cost_time=1964780, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=935523, WASH_BLOCK_END=107, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=994763], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:35.820155] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1964831, owner: ObLog, click_count: 5, time dist:[FORMAT_END=23, ObMalloc#start=0, ObMalloc#end=1964800, ALLOC_END=1, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:35.820155] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:37.787986] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1964831][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=1941878, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=51, WASH_BLOCK_END=929784, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=73849], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:36.786494] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [477][T1_StandbySchem][T1][YB42AC4D0007-00063C495B6A03B4-0-0] [lt=25][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1941927, owner: ObLog, click_count: 5, time dist:[FORMAT_END=14, ObMalloc#start=1, ObMalloc#end=1941903, ALLOC_END=1, APPEND_END=8], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:38.732391] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9082] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:35.819497] EDIAG [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2911356, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=6, ALLOC_BLOCK#end=2, FORMAT_END=2911338, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1d189030 0x1d18285e 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:35.934247] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2797278, owner: ObLog, click_count: 8, time dist:[FORMAT_END=2796954, ObMalloc#start=0, ObMalloc#end=67, ObMalloc#start=21, ObMalloc#end=43, ObMalloc#start=70, ObMalloc#end=47, FREE_END=76], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:35.819497] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=35] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:38.735291] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=2911356] temporary file module use no blocks [2025-08-14 09:05:38.735311] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=19] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:38.735324] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:38.735339] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623237259190, flush_io_finished_ret_:0, flush_io_finished_round_:26366, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [398]OB_MALLOC COST TOO MUCH TIME, cost_time=947920, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=947918], seq:[0] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=948023, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=47, WASH_BLOCK_END=34, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=947937], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:38.740850] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8032] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:37.779262] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:37.244013] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1495139, owner: ObLog, click_count: 9, time dist:[FORMAT_END=1494815, ObMalloc#start=0, ObMalloc#end=61, ObMalloc#start=21, ObMalloc#end=57, ObMalloc#start=179, ObMalloc#end=1, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:37.244013] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:38.741440] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1495139][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:36.726256] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2012217, owner: ObLog, click_count: 8, time dist:[ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=71, FORMAT_END=2011874, ObMalloc#start=1, ObMalloc#end=55, ObMalloc#start=21, ObMalloc#end=56, FREE_END=134], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:38.741536] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2012217][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:38.741557] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:38.741572] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:36.783292] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1955346, owner: ObLog, click_count: 8, time dist:[ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=2, FORMAT_END=1954643, ObMalloc#start=1, ObMalloc#end=74, ObMalloc#start=24, ObMalloc#end=57, FREE_END=542], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:38.741614] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1955346][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:38.741633] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:37.728360] EDIAG [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=762119][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1023529, , BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1f9ebc06 0x97c9052 0x1f9819be 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:38.745361] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1550] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:38.746107] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [807][T1002_TimerWK3_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=1] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):258, sequence:1}) [2025-08-14 09:05:37.727810] EDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=935502][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1034044, , BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:37.777268] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=31] gc stale ls task succ [2025-08-14 09:05:38.773202] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:38.776078] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=225] gc stale ls task succ [2025-08-14 09:05:38.777341] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=988586] gc stale ls task succ [2025-08-14 09:05:38.779338] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=215] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:38.780673] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=274] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:05:38.781254] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1793] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:38.782339] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9593] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:38.782418] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=265] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:38.783434] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:38.788663] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:38.789861] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=327] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:38.808471] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=190] gc stale ls task succ [2025-08-14 09:05:38.847708] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:38.854804] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=157] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:05:38.888427] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=257] PNIO [ratelimit] time: 1755162338888426, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007624, add_bytes: 0 [2025-08-14 09:05:38.930897] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=-4076] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:39.217059] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:39.307659] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:39.337487] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=224] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:39.637587] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=329][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:39.720733] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:39.721677] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=231] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [574]OB_MALLOC COST TOO MUCH TIME, cost_time=949631, tenant_id=500, label=CommonArray, ctx_id=0, prio=0, size=7936, [2025-08-14 09:05:39.724348] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=320] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:38.772825] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=278] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:39.732332] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:39.735445] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=35] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:39.716618] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=28426] current log_task status(palf_id=1, self="172.77.0.7:2882", log_id=8607756, log_task={header:{begin_lsn:{lsn:1604090090}, end_lsn:{lsn:1604090203}, log_id:8607756, min_scn:{val:1755161749741193001, v:0}, max_scn:{val:1755161749741193001, v:0}, data_len:57, proposal_id:50, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:1604090090}, data_checksum:2031283194, accum_checksum:759186326, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1755161750716165, freeze_ts:1755161750716173, submit_ts:1755161750716183, flushed_ts:1755161853765770, gen_to_freeze cost time:8, gen_to_submit cost time:18, submit_to_flush cost time:103049587}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161974728707, last_flushed_end_lsn:{lsn:1604090331}}]) [2025-08-14 09:05:38.792272] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [726][T1001_ObXAHbWor][T1001][Y0-0000000000000000-0-0] [lt=1053004] XA scheduler heartbeat task statistics(avg_time=0) [2025-08-14 09:05:38.854870] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=29] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:39.744460] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=888203] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:39.744564] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=47] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:39.744606] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:05:39.744628] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:39.744641] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:38.740907] EDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=960190][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1004830, , BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:38.740907] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=960190][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:39.747304] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1004830] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:39.748307] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:39.749529] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:39.755056] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=543] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:39.755345] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=365] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:39.755480] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:39.755513] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:39.755541] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=327] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:39.755564] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=330] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:34.780200] EDIAG [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [865][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=997883][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4968808, owner: ObLog, click_count: 5, time dist:[FORMAT_END=4968797, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b51df0f 0x986bcde 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:34.780200] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [865][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=997883] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136836462665002, v:0}, clog_base_lsn:{lsn:1543409664}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1755161030946590000, v:0}, tablet_change_checkpoint_scn:{val:1753850176390756001, v:0}, all_id_meta:{id_meta:[{limited_id:1755161762748684000, latest_log_ts:{val:1755161746772281001, v:0}}, {limited_id:9000001, latest_log_ts:{val:1755135652743621004, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:50, palf_env_:0x7f991c5e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-08-14 09:05:39.757670] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=248] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:38.772890] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=182] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:39.762891] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=988499] temporary file module use no blocks [2025-08-14 09:05:39.762906] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:39.762923] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=15] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:39.762940] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=16] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623240212582, flush_io_finished_ret_:0, flush_io_finished_round_:26386, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:39.724449] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=950179] temporary file module use no blocks [2025-08-14 09:05:39.763412] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=38903] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:39.763422] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:39.763433] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623240212527, flush_io_finished_ret_:0, flush_io_finished_round_:26406, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:39.779520] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:39.781307] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:39.789283] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:39.792515] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:39.794264] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:39.808544] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:05:39.836803] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:39.852676] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:39.862502] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:39.896038] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=21] PNIO [ratelimit] time: 1755162339896035, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007609, add_bytes: 0 [2025-08-14 09:05:39.935047] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=2] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:05:39.935090] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=-4076] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:39.944926] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:05:39.944991] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:39.945034] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:39.945077] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:39.945124] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:05:39.945137] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:39.945149] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:40.260852] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:40.340577] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=18] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:40.525166] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:40.667712] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=193111][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:40.679790] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:40.721709] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:40.735532] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=18] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:40.735559] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=20] temporary file module use no blocks [2025-08-14 09:05:40.735570] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:40.735583] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:40.735598] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623242175235, flush_io_finished_ret_:0, flush_io_finished_round_:26368, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:40.735726] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=1127] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:40.740267] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:40.740329] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:40.740993] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:40.741059] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:40.234620] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:40.744287] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:40.747528] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:51.765521] EDIAG print_current_status (ob_kvcache_hazard_version.cpp:328) [14881][TimerWK4_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 168739332, owner: ObLog, click_count: 7, time dist:[FORMAT_END=955798, ObMalloc#start=2, ObMalloc#end=24453, ObMalloc#start=167742402, ObMalloc#end=16672, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x1fc99db6 0x1fca0c5e 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:40.749406] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=18] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:51.765521] INFO print_current_status (ob_kvcache_hazard_version.cpp:328) [14881][TimerWK4_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] [KVCACHE-HAZARD-VERSION] hazard version status info: current_version: 3074300 | min_version= 3074300 | total_nodes_count: 9 | [KVCACHE-HAZARD] i= 798 | acquire_version=18446744073709551615 | waiting_nodes_count= 9 | last_retire_version= 3074291 | [2025-08-14 09:05:37.736405] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:647) [15308][T1002_TimerWK11_LSMetaCh][T1001][YB42AC4D0007-00063C497F399C2E-0-0] [lt=425367829][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs, exec_ctx)(ret=-4283, session_id=3221590844, *tx_desc={this:0x7f98db469c60, tx_id:{txid:0}, state:1, addr:"172.77.0.7:2882", tenant_id:1001, session_id:3221590844, assoc_session_id:3221590844, client_sid:3221590844, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1755161755759017, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180067074, seq_base:1755161755752409, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}) [2025-08-14 09:05:40.761033] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7601] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:40.762449] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=13037][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:40.762510] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [865][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8103] get_max_applied_scn(scn={val:1755161744727824001, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:49, palf_committed_end_lsn_:{lsn:7323686009}, palf_committed_end_scn_:{val:1755161747742105001, v:0}, last_check_scn_:{val:1755161749741193000, v:0}, max_applied_cb_scn_:{val:1755161744727824001, v:0}}) [2025-08-14 09:05:40.744546] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=3389] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:05:40.763784] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:40.767789] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:40.767817] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=22] temporary file module use no blocks [2025-08-14 09:05:40.767827] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:40.767840] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:40.767855] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623242202738, flush_io_finished_ret_:0, flush_io_finished_round_:26387, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:40.767892] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=18] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:40.767906] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] temporary file module use no blocks [2025-08-14 09:05:40.767913] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:40.767923] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:40.767934] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623242203220, flush_io_finished_ret_:0, flush_io_finished_round_:26407, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:40.776208] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=261] gc stale ls task succ [2025-08-14 09:05:40.779414] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:40.782631] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:40.783254] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:40.783357] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=446] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:40.792264] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:40.793033] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:40.808618] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=25] gc stale ls task succ [2025-08-14 09:05:40.825374] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=371][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=886601, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=886600], seq:[0] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=912054, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=52, WASH_BLOCK_END=82, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=886621], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:40.837700] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=866941, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=866940], seq:[0] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=866959, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=866954], seq:[0, 1] [2025-08-14 09:05:40.883270] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=347][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:41.745285] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:41.777925] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=32589][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=956876, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=956868], seq:[0] [864]OB_MALLOC COST TOO MUCH TIME, cost_time=941224, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=59, WASH_BLOCK_END=941165], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=811776, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=61, WASH_BLOCK_END=811715], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=376909, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=63, WASH_BLOCK_END=376846], seq:[0, 1, 2] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=941376, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=49, WASH_BLOCK_END=941327], seq:[0, 1, 2] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=942209, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=63, WASH_BLOCK_END=942146], seq:[0, 1, 2] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=940623, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=50, WASH_BLOCK_END=940572], seq:[0, 1, 2] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=930904, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=64, WASH_BLOCK_END=930839], seq:[0, 1, 2] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=914574, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=60, WASH_BLOCK_END=914514], seq:[0, 1, 2] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=960342, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=65, WASH_BLOCK_END=941980, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=7679], seq:[0, 1, 2, 3, 4] [2025-08-14 09:01:38.779968] EDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:662) [740][T1001_Occam][T1001][YB42AC4D0007-00063C4966399E1E-0-0] [lt=12][errcode=-4283] LOGGER COST TOO MUCH TIME, cost: 9965243, owner: ObLog, click_count: 5, time dist:[FORMAT_END=9965235, ObMalloc#start=0, ObMalloc#end=4, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b864fb7 0x1b864817 0x93a2971 0x934006e 0x933e2ee 0x93363f3 0x932e8ab 0x96ac524 0x96a672d 0x96a5033 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x1b488f1f 0x1b4ca7cf 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:42.721593] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=201] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:42.741987] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20363][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:41.780638] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:42.742467] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=960375][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:42.740661] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [880][T1002_ArcSender][T1002][YB42AC4D0007-00063C495EB99C16-0-0] [lt=3465] ObArchiveSender is running(thread_index=5) [2025-08-14 09:05:42.744664] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=985775][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:42.744700] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=963847][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:42.750926] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1282] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:42.751003] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:42.751038] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=228] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:42.751140] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=160] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:42.751370] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=18369] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25878560768, fast_delete_log_mode=false) [2025-08-14 09:05:42.751739] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=1063] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:42.730797] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4015] clock out of order(cur_ts=1755162341778491, cur_ts_=1755162341750338, delta=28153) [2025-08-14 09:05:41.243689] EDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] LOGGER COST TOO MUCH TIME, cost: 1510121, , BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x13b06ffb 0x958a39e 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:41.243689] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:42.759573] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=251] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:42.759626] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=463] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:42.761452] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=1384] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:42.761506] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:42.761554] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:42.759706] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:42.760549] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:41.782253] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1000145, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:41.782253] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=26] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:42.783943] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1000145][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:42.783963] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:42.770589] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=24] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:42.784701] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=14083] temporary file module use no blocks [2025-08-14 09:05:42.784719] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=17] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:42.784734] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:42.784749] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623244208651, flush_io_finished_ret_:0, flush_io_finished_round_:26389, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:42.770656] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=16] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:42.784816] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14137] temporary file module use no blocks [2025-08-14 09:05:42.772965] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:42.761102] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:42.774034] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:42.776377] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=253] gc stale ls task succ [2025-08-14 09:05:42.759656] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=264] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:42.784919] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25223][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:42.761594] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:05:42.778795] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:42.784974] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23330] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:42.784988] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:42.732752] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=948050][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:42.785513] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=52670] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:42.791279] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=407] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:42.797266] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:42.815305] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=922125] gc stale ls task succ [2025-08-14 09:05:42.846843] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=20015][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:05:42.846879] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=19849][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:05:42.918877] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=819680] PNIO [ratelimit] time: 1755162342918876, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007615, add_bytes: 0 [2025-08-14 09:05:43.256536] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:43.374492] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:43.446287] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=384745] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:43.675029] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:43.697667] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:43.721008] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:01:48.746719] EDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [1000][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8F01-0-0] [lt=24][errcode=-4341] LOGGER COST TOO MUCH TIME, cost: 2002146, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1994305, ALLOC_END=7840, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1c03240b 0x1c031da7 0x1c03844f 0x984d055 0x9280839 0x926c154 0x12c5a0c9 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:42.784825] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:43.726089] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=939840] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:43.726158] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=67] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623244208678, flush_io_finished_ret_:0, flush_io_finished_round_:26409, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:43.732242] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=949866] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:43.732529] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=949533] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:05:43.742724] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:43.742909] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:43.744742] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:43.745049] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:43.748111] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=11] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:43.757735] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=676] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:43.761138] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:43.779288] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6166] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:43.781439] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2053] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:43.782628] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=8486] gc stale ls task succ [2025-08-14 09:05:43.782720] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=22896][errcode=-4015] clock out of order(cur_ts=1755162343782719, cur_ts_=1755162343737525, delta=45194) [807]OB_MALLOC COST TOO MUCH TIME, cost_time=989931, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=989929], seq:[0] [807]OB_MALLOC COST TOO MUCH TIME, cost_time=990150, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=84, WASH_BLOCK_END=93, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=989957], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:42.776516] EDIAG [SQL.PC] cache_evict (ob_plan_cache.cpp:1507) [807][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=956320][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1942426, owner: ObLog, click_count: 5, time dist:[FORMAT_END=952255, ObMalloc#start=3, ObMalloc#end=990160, ALLOC_END=2, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1535920e 0x153639eb 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [574]OB_MALLOC COST TOO MUCH TIME, cost_time=937198, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=49, WASH_BLOCK_END=937149], seq:[0, 1, 2] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=936394, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=36, WASH_BLOCK_END=936358], seq:[0, 1, 2] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=936409, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=35, WASH_BLOCK_END=936374], seq:[0, 1, 2] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=935619, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=42, WASH_BLOCK_END=935576], seq:[0, 1, 2] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=929857, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=53, WASH_BLOCK_END=929804], seq:[0, 1, 2] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=926330, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=42, WASH_BLOCK_END=926287], seq:[0, 1, 2] [2025-08-14 09:05:42.776516] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1507) [807][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=956320] start lib cache evict(tenant_id=1002, mem_hold=211812352, mem_limit=429496725, cache_obj_num=1331, cache_node_num=1330) [472]OB_MALLOC COST TOO MUCH TIME, cost_time=922548, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=35, WASH_BLOCK_END=922513], seq:[0, 1, 2] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=916490, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=42, WASH_BLOCK_END=916447], seq:[0, 1, 2] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=906144, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=50, WASH_BLOCK_END=906093], seq:[0, 1, 2] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=856138, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=856097], seq:[0, 1, 2] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=847300, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=43, WASH_BLOCK_END=847257], seq:[0, 1, 2] [466]OB_MALLOC COST TOO MUCH TIME, cost_time=847263, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=37, WASH_BLOCK_END=847226], seq:[0, 1, 2] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=835289, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=35, WASH_BLOCK_END=835254], seq:[0, 1, 2] [2025-08-14 09:05:43.798756] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:43.804915] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:43.874478] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:43.886606] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:05:43.794868] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:43.865526] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:44.723470] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=937611][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:43.874600] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:44.723288] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=930605][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=796040, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=35, WASH_BLOCK_END=796005], seq:[0, 1, 2] [2025-08-14 09:05:43.926494] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=27] PNIO [ratelimit] time: 1755162343926491, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007615, add_bytes: 0 [770]OB_MALLOC COST TOO MUCH TIME, cost_time=736250, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=34, WASH_BLOCK_END=736206], seq:[0, 1, 2] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=723279, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=51, WASH_BLOCK_END=723228], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=705194, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=44, WASH_BLOCK_END=705150], seq:[0, 1, 2] [2025-08-14 09:05:43.986534] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:43.999519] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:44.724345] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=1036] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:44.724401] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:44.724438] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:05:44.724451] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:44.724462] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [372]OB_MALLOC COST TOO MUCH TIME, cost_time=465933, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=44, WASH_BLOCK_END=465889], seq:[0, 1, 2] [2025-08-14 09:05:44.258600] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:44.726259] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:44.726286] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=22] temporary file module use no blocks [2025-08-14 09:05:44.726297] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:44.726312] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:44.726327] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623246165954, flush_io_finished_ret_:0, flush_io_finished_round_:26410, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [419]OB_MALLOC COST TOO MUCH TIME, cost_time=275528, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=49, WASH_BLOCK_END=275479], seq:[0, 1, 2] [2025-08-14 09:05:44.451297] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=32] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:44.732368] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=29] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:41.758010] EDIAG [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [865][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2997709, owner: ObLog, click_count: 6, time dist:[FORMAT_END=2997531, ObMalloc#start=1, ObMalloc#end=37, ObMalloc#start=22, ObMalloc#end=36, FREE_END=82], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xff22be4 0x97a9cd7 0x97a9238 0x97a9307 0x97a8fe2 0x986bc46 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:42.849325] EDIAG [RS] get_valid_sts_after (ob_tenant_info_loader.cpp:454) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1913492, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1913137, ObMalloc#start=1, ObMalloc#end=64, ObMalloc#start=25, ObMalloc#end=35, FREE_END=230], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1001d942 0x996d843 0x996d15a 0x996cfd8 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [394]OB_MALLOC COST TOO MUCH TIME, cost_time=1934178, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=56, WASH_BLOCK_END=937038, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=45357], seq:[0, 1, 2, 3, 4] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=970849, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=970847], seq:[0] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=983171, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=43, WASH_BLOCK_END=12247, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=970875], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:43.783348] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1934225, owner: ObLog, click_count: 5, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=1934207, ALLOC_END=1, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:43.783348] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:45.720546] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1934225][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:45.720571] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:45.722633] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6099] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:45.722990] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=937908] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:43.782902] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3059][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1937784, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=6, ALLOC_BLOCK#end=4, FORMAT_END=1937757, ObMalloc#start=0, ObMalloc#end=3, ALLOC_END=10, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:44.722266] EDIAG [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1000910, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1000905, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a5bbf2 0x94da542 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:44.721881] EDIAG [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=22412][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1001378, owner: ObLog, click_count: 6, time dist:[FORMAT_END=996895, ObMalloc#start=1, ObMalloc#end=61, ObMalloc#start=28, ObMalloc#end=47, FREE_END=4346], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x99721a1 0x997a72e 0x2006ed17 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:44.733541] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:05:43.782902] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3059] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:45.723646] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1937784][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:45.723665] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:44.722266] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] [sql nio session](*s={this:0x7f98f72d0030, session_id:3221589415, trace_id:YB42AC4D0007-00063C4965FA43DC-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:245}, err:0, last_decode_time:1755161751714366, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1662, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:05:45.724717] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=1000910] [sql nio session](*s={this:0x7f98f72d5030, session_id:3221590200, trace_id:YB42AC4D0007-00063C4965FA43E0-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:248}, err:0, last_decode_time:1755161776777167, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1593, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:05:45.527817] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=275583] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:45.727514] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:44.924740] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:05:45.773340] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=847271] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:45.773389] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:45.773430] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:45.773466] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:05:45.773477] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:45.773487] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:45.720274] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=931165] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:45.777055] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=56693] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623246226328, flush_io_finished_ret_:0, flush_io_finished_round_:26391, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [800]OB_MALLOC COST TOO MUCH TIME, cost_time=964424, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=964422], seq:[0] [800]OB_MALLOC COST TOO MUCH TIME, cost_time=964582, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=59, WASH_BLOCK_END=67, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=964449], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:45.767851] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=42963] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:46.744187] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=974859] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:46.744201] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:46.744209] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:46.748910] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=385] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [398]OB_MALLOC COST TOO MUCH TIME, cost_time=981530, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=981522], seq:[0] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=981964, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=981551], seq:[0, 1] [2025-08-14 09:05:45.779650] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=218] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:46.763128] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=981990][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=1022832, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1022801, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3, 4, 5] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=1022881, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1022852, WASH_BLOCK_END=29], seq:[0, 1, 2] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=1022254, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1022224, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=1022283, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1022273, WASH_BLOCK_END=10], seq:[0, 1, 2] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=1021390, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1021354, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=1021409, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1021400, WASH_BLOCK_END=9], seq:[0, 1, 2] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=1020807, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1020775, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=1020860, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1020830, WASH_BLOCK_END=30], seq:[0, 1, 2] [2025-08-14 09:05:45.781665] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1022523, owner: ObLog, click_count: 6, time dist:[FORMAT_END=6, ObMalloc#start=0, ObMalloc#end=1022290, ObMalloc#start=21, ObMalloc#end=57, FREE_END=149], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [730]OB_MALLOC COST TOO MUCH TIME, cost_time=1019551, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=15, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1019509, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=1019570, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1019562, WASH_BLOCK_END=8], seq:[0, 1, 2] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=1004548, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=15, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1004505, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=586402, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=586369, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=1004743, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1004728, WASH_BLOCK_END=15], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=586420, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=586412, WASH_BLOCK_END=8], seq:[0, 1, 2] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=550787, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=550748, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=550805, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=550797, WASH_BLOCK_END=8], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=981028, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=980997, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=981063, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=981051, WASH_BLOCK_END=12], seq:[0, 1, 2] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=830997, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=830971, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=831344, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=831336, WASH_BLOCK_END=8], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=274556, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=274523, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=274590, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=274581, WASH_BLOCK_END=9], seq:[0, 1, 2] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=775390, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=775357, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3, 4, 5] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=863414, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=863380, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=737582, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=737550, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3, 4, 5] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=863446, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=863432, WASH_BLOCK_END=14], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=738219, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=738207, WASH_BLOCK_END=12], seq:[0, 1, 2] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=1014146, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1014113, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=1014990, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1014981, WASH_BLOCK_END=9], seq:[0, 1, 2] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=988974, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=988941, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=989905, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=989897, WASH_BLOCK_END=8], seq:[0, 1, 2] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=775814, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=775805, WASH_BLOCK_END=9], seq:[0, 1, 2] [2025-08-14 09:05:45.784722] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=931895][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1020834, owner: ObLog, click_count: 6, time dist:[FORMAT_END=8, ObMalloc#start=1, ObMalloc#end=1019584, ObMalloc#start=19, ObMalloc#end=53, FREE_END=1169], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [399]OB_MALLOC COST TOO MUCH TIME, cost_time=1025367, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1025337, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=1025393, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1025382, WASH_BLOCK_END=11], seq:[0, 1, 2] [2025-08-14 09:05:45.783283] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=230][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1022919, owner: ObLog, click_count: 6, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=1020870, ObMalloc#start=25, ObMalloc#end=59, FREE_END=1955], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:46.743301] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=18033] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:05:46.733311] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=336] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:46.741056] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=16] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:46.746537] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=261] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:46.807397] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=60767][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:46.765764] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2162] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:46.807594] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=41765][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:46.807770] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=1022919] gc stale ls task succ [2025-08-14 09:05:46.807614] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=49405][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:45.783258] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [339][EventTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1023072, owner: ObLog, click_count: 5, time dist:[FORMAT_END=981184, ObMalloc#start=1, ObMalloc#end=41883, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [764]OB_MALLOC COST TOO MUCH TIME, cost_time=202812, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=202772, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3, 4, 5] [2025-08-14 09:05:46.595839] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:45.783258] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [339][EventTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [764]OB_MALLOC COST TOO MUCH TIME, cost_time=202833, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=202824, WASH_BLOCK_END=9], seq:[0, 1, 2] [2025-08-14 09:05:46.807553] WDIAG ev_invoke_pending (ev.c:2136) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=49058][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:46.807934] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=63592] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:46.604729] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:46.807951] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:46.807970] WDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=75211][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:45.753839] EDIAG [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1053333, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1009826, ObMalloc#start=0, ObMalloc#end=43481, ALLOC_END=22, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x12377491 0x1236e8e9 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:46.728302] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:46.067331] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:46.029449] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:46.808045] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162346808045, start=1755162346067329, delta=740716) [2025-08-14 09:05:46.750461] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=453] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:45.753839] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=11] finish to resize log external storage handler(current_ts=1755162344786333, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-08-14 09:05:46.808824] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=52981][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:45.780752] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1026683, owner: ObLog, click_count: 6, time dist:[FORMAT_END=8, ObMalloc#start=1, ObMalloc#end=1025400, ObMalloc#start=20, ObMalloc#end=51, FREE_END=1203], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:44.765747] EDIAG [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1913492][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2040241, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1993383, ObMalloc#start=2, ObMalloc#end=45375, ObMalloc#start=20, ObMalloc#end=1227, FREE_END=234], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1b52e1b7 0x996d441 0x996cfd8 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:45.782671] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=174][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1025007, owner: ObLog, click_count: 6, time dist:[FORMAT_END=7, ObMalloc#start=1, ObMalloc#end=1021416, ObMalloc#start=29, ObMalloc#end=56, FREE_END=3498], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:46.808864] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=26115] temporary file module use no blocks [2025-08-14 09:05:46.808681] WDIAG ev_invoke_pending (ev.c:2136) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=54434][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:46.808887] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=22] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:46.808904] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=15] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:46.808919] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623248216856, flush_io_finished_ret_:0, flush_io_finished_round_:26392, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:46.807491] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=66271][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:46.809008] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1026683][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:46.809027] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:45.823679] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:46.808472] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=80464] temporary file module use no blocks [2025-08-14 09:05:46.809388] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=914] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:45.780889] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=249][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1026974, owner: ObLog, click_count: 6, time dist:[FORMAT_END=8, ObMalloc#start=1, ObMalloc#end=1022892, ObMalloc#start=26, ObMalloc#end=58, FREE_END=3989], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:46.809435] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1026974][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:46.809457] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:46.808877] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=87061][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:45.790264] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=330][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1015938, owner: ObLog, click_count: 6, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=1015007, ObMalloc#start=20, ObMalloc#end=57, FREE_END=843], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:45.799812] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1028538, owner: ObLog, click_count: 8, time dist:[FORMAT_END=34, ObMalloc#start=1, ObMalloc#end=1004758, ObMalloc#start=21, ObMalloc#end=47, ObMalloc#start=3322, ObMalloc#end=11856, FREE_END=8499], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [553]OB_MALLOC COST TOO MUCH TIME, cost_time=894354, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=894345], seq:[0] [552]OB_MALLOC COST TOO MUCH TIME, cost_time=895708, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=895673, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [552]OB_MALLOC COST TOO MUCH TIME, cost_time=895764, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=895730, WASH_BLOCK_END=34], seq:[0, 1, 2] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=895693, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=895668, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=895740, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=895711, WASH_BLOCK_END=29], seq:[0, 1, 2] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=895771, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=895735, ALLOC_CHUNK_END=14], seq:[0, 1, 2, 3, 4, 5] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=895790, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=895781, WASH_BLOCK_END=9], seq:[0, 1, 2] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=896601, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=896577, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=896638, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=896616, WASH_BLOCK_END=22], seq:[0, 1, 2] [864]OB_MALLOC COST TOO MUCH TIME, cost_time=896626, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=896597, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [864]OB_MALLOC COST TOO MUCH TIME, cost_time=896644, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=896635, WASH_BLOCK_END=9], seq:[0, 1, 2] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=896622, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=896598, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=896640, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=896631, WASH_BLOCK_END=7], seq:[0, 1, 2] [276]OB_MALLOC COST TOO MUCH TIME, cost_time=897628, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=897606, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [276]OB_MALLOC COST TOO MUCH TIME, cost_time=897684, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=897652, WASH_BLOCK_END=32], seq:[0, 1, 2] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=897721, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=897701, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=897745, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=897731, WASH_BLOCK_END=13], seq:[0, 1, 2] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=910645, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=910634], seq:[0, 1, 2] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=910661, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=910656], seq:[0, 1] [466]OB_MALLOC COST TOO MUCH TIME, cost_time=892480, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=15, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=892437, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [466]OB_MALLOC COST TOO MUCH TIME, cost_time=892524, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=892498, WASH_BLOCK_END=26], seq:[0, 1, 2] [2025-08-14 09:05:46.808524] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=61950][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=894976, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=894947, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=895019, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=894992, WASH_BLOCK_END=27], seq:[0, 1, 2] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=895044, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=895019, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=895076, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=895055, WASH_BLOCK_END=21], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=815920, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=815886, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=815939, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=815929, WASH_BLOCK_END=10], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=775908, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=775879, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=775927, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=775918, WASH_BLOCK_END=9], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=609643, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=609607, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=609665, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=609655, WASH_BLOCK_END=10], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=459869, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=459841, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=459890, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=459879, WASH_BLOCK_END=10], seq:[0, 1, 2] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=449985, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=449953, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=450015, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=450004, WASH_BLOCK_END=11], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=119991, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=119964, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=120010, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=120001, WASH_BLOCK_END=9], seq:[0, 1, 2] [2025-08-14 09:05:47.724251] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:05:46.829820] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:47.727434] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=620] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:47.116404] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:47.727522] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162347727522, start=1755162347116402, delta=611120) [2025-08-14 09:05:46.909376] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:47.276530] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:47.266514] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:46.829946] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1028538][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:46.830010] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [553]OB_MALLOC COST TOO MUCH TIME, cost_time=973468, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40032, WASH_BLOCK_END=11713, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=894373], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:46.740728] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=18033] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:05:46.740728] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=18033][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1002249, owner: ObLog, click_count: 5, time dist:[FORMAT_END=28755, ObMalloc#start=1, ObMalloc#end=973486, ALLOC_END=1, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:47.761771] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=242] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:47.754398] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=21] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:47.774916] WDIAG ev_invoke_pending (ev.c:2136) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=20485][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:47.720317] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=909541] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623248167622, flush_io_finished_ret_:0, flush_io_finished_round_:26412, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:45.753456] EDIAG [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11366][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2030818, , BACKTRACE: 0x26a1806d 0x9826e08 0x98267df 0x982650b 0x98262e0 0x1d188ec1 0x1d17b8ec 0x1d17ad88 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:45.932249] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1853074, , BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:47.808982] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:47.956958] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=777082] PNIO [ratelimit] time: 1755162347956956, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007374, add_bytes: 0 [574]OB_MALLOC COST TOO MUCH TIME, cost_time=912902, tenant_id=500, label=CommonArray, ctx_id=0, prio=0, size=7936, [382]OB_MALLOC COST TOO MUCH TIME, cost_time=940767, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=940765], seq:[0] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=940886, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=39, WASH_BLOCK_END=45, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=940796], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:47.781438] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:48.723783] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=940920][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:46.009244] EDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 1711498, owner: ObLog, click_count: 8, time dist:[FORMAT_END=789807, ObMalloc#start=1, ObMalloc#end=21992, ObMalloc#start=20, ObMalloc#end=897694, ObMalloc#start=74, ObMalloc#end=42, FREE_END=1867], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x9757ced 0x97569df 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:46.809368] EDIAG [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=834310][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1921801, owner: ObLog, click_count: 8, time dist:[ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=2, FORMAT_END=1921489, ObMalloc#start=1, ObMalloc#end=75, ObMalloc#start=25, ObMalloc#end=45, FREE_END=161], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1f9ebc06 0x97c9052 0x1f9819be 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:46.811420] EDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1015938][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1920275, owner: ObLog, click_count: 8, time dist:[FORMAT_END=1919888, ObMalloc#start=0, ObMalloc#end=57, ObMalloc#start=20, ObMalloc#end=52, ObMalloc#start=129, ObMalloc#end=46, FREE_END=83], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:46.754337] EDIAG [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [865][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=2997709][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1981559, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1981295, ObMalloc#start=2, ObMalloc#end=61, ObMalloc#start=22, ObMalloc#end=43, FREE_END=135], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1bdd73e2 0x1bdd6e36 0x986bc6f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:48.744895] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=230] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:48.748981] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=15] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:42.758731] EDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=7270][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5983193, , BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0xfc0e73d 0x993f560 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:48.761204] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:47.722018] EDIAG [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2510) [807][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=946005][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1041560, owner: ObLog, click_count: 5, time dist:[FORMAT_END=24869, ObMalloc#start=2, ObMalloc#end=3, ALLOC_END=1016679, APPEND_END=7], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1535e3fb 0x15365204 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:47.722018] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2510) [807][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=946005] Dumping All Cache Objs(alloc_obj_list.count()=1331, alloc_obj_list=[{obj_id:1, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:49654}, {obj_id:2, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50591}, {obj_id:3, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49570}, {obj_id:4, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:158277}, {obj_id:5, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207030}, {obj_id:6, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:129654}, {obj_id:7, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:170276}, {obj_id:8, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:92905}, {obj_id:9, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55437}, {obj_id:11, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49896}, {obj_id:12, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54788}, {obj_id:13, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60105}, {obj_id:15, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:108509}, {obj_id:16, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54018}, {obj_id:17, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:107151}, {obj_id:18, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127908}, {obj_id:19, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52851}, {obj_id:20, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46488}, {obj_id:21, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62127}, {obj_id:22, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:112117}, {obj_id:23, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71118}, {obj_id:24, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47416}, {obj_id:25, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64955}, {obj_id:26, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:107240}, {obj_id:27, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203554}, {obj_id:28, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:149171}, {obj_id:29, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:99644}, {obj_id:30, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:109417}, {obj_id:31, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:104852}, {obj_id:32, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:81681}, {obj_id:33, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:86868}, {obj_id:34, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:77320}, {obj_id:35, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73401}, {obj_id:36, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67092}, {obj_id:37, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:90423}, {obj_id:38, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73995}, {obj_id:39, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:115258}, {obj_id:40, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78254}, {obj_id:41, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63461}, {obj_id:42, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60610}, {obj_id:43, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59274}, {obj_id:44, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74545}, {obj_id:45, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:68364}, {obj_id:46, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62191}, {obj_id:47, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59473}, {obj_id:48, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62548}, {obj_id:49, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59505}, {obj_id:50, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62549}, {obj_id:51, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59441}, {obj_id:52, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65989}, {obj_id:53, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:61026}, {obj_id:54, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57028}, {obj_id:55, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85137}, {obj_id:56, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56916}, {obj_id:57, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:61962}, {obj_id:58, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57316}, {obj_id:59, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65864}, {obj_id:60, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56996}, {obj_id:61, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58212}, {obj_id:62, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65512}, {obj_id:63, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60453}, {obj_id:64, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56884}, {obj_id:65, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59085}, {obj_id:66, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235524}, {obj_id:67, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:133191}, {obj_id:68, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:122973}, {obj_id:69, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222820}, {obj_id:70, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:119011}, {obj_id:71, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62425}, {obj_id:72, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63273}, {obj_id:73, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173058}, {obj_id:74, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93811}, {obj_id:75, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:90631}, {obj_id:76, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:81787}, {obj_id:77, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71611}, {obj_id:78, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65300}, {obj_id:79, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51380}, {obj_id:80, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:81, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48067}, {obj_id:82, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:83, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52125}, {obj_id:84, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:85, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120160}, {obj_id:86, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:87, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97242}, {obj_id:88, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:90, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:91, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:46504}, {obj_id:92, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:68970}, {obj_id:93, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59166}, {obj_id:94, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60338}, {obj_id:95, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51478}, {obj_id:96, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:109830}, {obj_id:97, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48968}, {obj_id:98, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:112041}, {obj_id:99, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46440}, {obj_id:100, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69133}, {obj_id:101, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:102, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54372}, {obj_id:103, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:104, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48280}, {obj_id:105, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:116242}, {obj_id:106, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53185}, {obj_id:107, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:48908}, {obj_id:108, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51274}, {obj_id:109, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:108920}, {obj_id:110, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57540}, {obj_id:111, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54911}, {obj_id:112, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66049}, {obj_id:113, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:114, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:107403}, {obj_id:115, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59412}, {obj_id:116, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59924}, {obj_id:117, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64784}, {obj_id:118, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:119, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:120, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:121, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:30464}, {obj_id:122, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:123, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:124, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64774}, {obj_id:125, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63004}, {obj_id:126, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:127, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:30482}, {obj_id:133, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59119}, {obj_id:134, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:109167}, {obj_id:140, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57392}, {obj_id:142, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72205}, {obj_id:143, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60318}, {obj_id:144, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59063}, {obj_id:145, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52621}, {obj_id:146, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60640}, {obj_id:147, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:110708}, {obj_id:148, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:79964}, {obj_id:149, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105305}, {obj_id:150, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64637}, {obj_id:152, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:107581}, {obj_id:153, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56979}, {obj_id:154, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:68485}, {obj_id:155, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51889}, {obj_id:157, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50408}, {obj_id:158, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47936}, {obj_id:160, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:112534}, {obj_id:161, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69948}, {obj_id:162, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60075}, {obj_id:163, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50629}, {obj_id:164, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50612}, {obj_id:165, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:33277}, {obj_id:167, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78418}, {obj_id:168, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50072}, {obj_id:169, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51380}, {obj_id:170, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:172, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55901}, {obj_id:173, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57684}, {obj_id:175, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55665}, {obj_id:177, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49689}, {obj_id:178, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62390}, {obj_id:179, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:111173}, {obj_id:181, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67757}, {obj_id:187, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:30668}, {obj_id:188, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67536}, {obj_id:189, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:113436}, {obj_id:191, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67820}, {obj_id:192, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52042}, {obj_id:193, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46840}, {obj_id:194, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46856}, {obj_id:195, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193886}, {obj_id:197, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:128794}, {obj_id:198, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69301}, {obj_id:199, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173059}, {obj_id:201, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:145729}, {obj_id:202, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:203, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:98424}, {obj_id:205, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:137715}, {obj_id:206, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59503}, {obj_id:207, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192000}, {obj_id:210, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47408}, {obj_id:211, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59513}, {obj_id:213, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59188}, {obj_id:215, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46848}, {obj_id:218, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54467}, {obj_id:219, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62656}, {obj_id:222, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57417}, {obj_id:223, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57076}, {obj_id:225, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:80665}, {obj_id:226, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:68110}, {obj_id:227, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59478}, {obj_id:228, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54200}, {obj_id:229, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71579}, {obj_id:230, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:79425}, {obj_id:231, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53027}, {obj_id:232, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50327}, {obj_id:233, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52701}, {obj_id:234, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53495}, {obj_id:235, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54912}, {obj_id:236, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:115782}, {obj_id:237, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49369}, {obj_id:239, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71297}, {obj_id:240, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:45664}, {obj_id:241, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:242, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:29563}, {obj_id:243, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:29547}, {obj_id:244, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72439}, {obj_id:245, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:82003}, {obj_id:246, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60693}, {obj_id:247, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:248, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:249, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:68970}, {obj_id:253, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:13745}, {obj_id:254, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53850}, {obj_id:255, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1244037}, {obj_id:256, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53353}, {obj_id:257, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:979432}, {obj_id:258, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:259, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:441763}, {obj_id:260, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:131652}, {obj_id:261, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102208}, {obj_id:262, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174688}, {obj_id:263, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:68064}, {obj_id:264, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:124055}, {obj_id:265, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51856}, {obj_id:266, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55892}, {obj_id:267, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53476}, {obj_id:268, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248928}, {obj_id:270, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:111648}, {obj_id:272, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55724}, {obj_id:273, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:139752}, {obj_id:274, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65613}, {obj_id:275, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74271}, {obj_id:276, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189587}, {obj_id:277, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46437}, {obj_id:278, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47893}, {obj_id:279, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:150376}, {obj_id:280, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49545}, {obj_id:281, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175345}, {obj_id:282, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:131048}, {obj_id:283, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72474}, {obj_id:284, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:285, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:286, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240919}, {obj_id:288, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55756}, {obj_id:291, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:81957}, {obj_id:292, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268057}, {obj_id:297, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85376}, {obj_id:298, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85602}, {obj_id:299, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:111929}, {obj_id:300, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:286644}, {obj_id:301, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56303}, {obj_id:302, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182524}, {obj_id:303, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:137836}, {obj_id:304, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:216847}, {obj_id:307, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55868}, {obj_id:308, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72832}, {obj_id:309, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102124}, {obj_id:310, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225211}, {obj_id:313, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:75504}, {obj_id:314, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:147275}, {obj_id:315, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204179}, {obj_id:316, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:588871}, {obj_id:319, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69223}, {obj_id:320, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206652}, {obj_id:321, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:324, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74659}, {obj_id:325, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:232949}, {obj_id:326, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53958}, {obj_id:327, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180179}, {obj_id:328, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:135494}, {obj_id:332, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:80174}, {obj_id:333, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:259278}, {obj_id:334, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55154}, {obj_id:335, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181355}, {obj_id:336, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:136712}, {obj_id:342, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67384}, {obj_id:343, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197887}, {obj_id:345, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53618}, {obj_id:346, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179771}, {obj_id:347, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:135134}, {obj_id:348, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:350, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:146442}, {obj_id:364, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:75480}, {obj_id:368, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50876}, {obj_id:369, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74415}, {obj_id:370, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51322}, {obj_id:371, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51084}, {obj_id:372, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51442}, {obj_id:374, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51202}, {obj_id:375, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52790}, {obj_id:376, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52854}, {obj_id:377, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54387}, {obj_id:378, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52854}, {obj_id:379, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51586}, {obj_id:380, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54872}, {obj_id:381, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51378}, {obj_id:382, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53030}, {obj_id:383, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55599}, {obj_id:384, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51378}, {obj_id:385, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51314}, {obj_id:386, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51250}, {obj_id:387, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53046}, {obj_id:388, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54815}, {obj_id:389, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51218}, {obj_id:390, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51218}, {obj_id:391, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51250}, {obj_id:392, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52870}, {obj_id:393, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51266}, {obj_id:394, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51362}, {obj_id:395, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51250}, {obj_id:396, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59119}, {obj_id:397, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51362}, {obj_id:398, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53062}, {obj_id:399, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51362}, {obj_id:400, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51362}, {obj_id:401, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51250}, {obj_id:402, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51218}, {obj_id:409, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:149775}, {obj_id:410, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66617}, {obj_id:415, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:5265}, {obj_id:416, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:31009}, {obj_id:417, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:418, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48336}, {obj_id:419, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59108}, {obj_id:420, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50938}, {obj_id:421, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48671}, {obj_id:422, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56563}, {obj_id:423, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46872}, {obj_id:424, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:61808}, {obj_id:425, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59824}, {obj_id:427, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:618230}, {obj_id:428, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:153758}, {obj_id:429, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69455}, {obj_id:431, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55724}, {obj_id:433, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55756}, {obj_id:436, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85376}, {obj_id:437, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:108532}, {obj_id:439, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55868}, {obj_id:441, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:75504}, {obj_id:443, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:390100}, {obj_id:444, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:360217}, {obj_id:445, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:457, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55740}, {obj_id:458, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:96462}, {obj_id:459, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212819}, {obj_id:460, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:340001}, {obj_id:461, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:473, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:154884}, {obj_id:476, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:375543}, {obj_id:477, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:116952}, {obj_id:478, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:123944}, {obj_id:479, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105139}, {obj_id:490, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50527}, {obj_id:491, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:151767}, {obj_id:494, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47368}, {obj_id:495, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48232}, {obj_id:496, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62071}, {obj_id:497, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:61352}, {obj_id:500, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52851}, {obj_id:504, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:333278}, {obj_id:505, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:129618}, {obj_id:507, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55724}, {obj_id:509, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55756}, {obj_id:512, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85376}, {obj_id:514, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55868}, {obj_id:516, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:75504}, {obj_id:525, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:549, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53306}, {obj_id:550, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62783}, {obj_id:551, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54109}, {obj_id:552, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:103697}, {obj_id:553, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120708}, {obj_id:554, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67452}, {obj_id:560, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:349340}, {obj_id:561, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127623}, {obj_id:563, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55724}, {obj_id:565, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55756}, {obj_id:568, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85376}, {obj_id:570, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55868}, {obj_id:572, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:75504}, {obj_id:581, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:605, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53171}, {obj_id:606, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55852}, {obj_id:607, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:116904}, {obj_id:608, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:258163}, {obj_id:609, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:132926}, {obj_id:610, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:80606}, {obj_id:611, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127311}, {obj_id:612, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:124808}, {obj_id:613, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54894}, {obj_id:614, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58088}, {obj_id:615, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:88608}, {obj_id:616, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:121604}, {obj_id:617, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63204}, {obj_id:618, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52886}, {obj_id:623, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:465799}, {obj_id:625, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55724}, {obj_id:627, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55756}, {obj_id:630, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85376}, {obj_id:632, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55868}, {obj_id:636, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:75504}, {obj_id:647, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:674, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:281099}, {obj_id:675, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:133643}, {obj_id:677, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55724}, {obj_id:679, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55756}, {obj_id:682, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85376}, {obj_id:684, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55868}, {obj_id:686, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:75504}, {obj_id:692, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196769}, {obj_id:700, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55740}, {obj_id:701, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:728, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:277051}, {obj_id:730, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55724}, {obj_id:732, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55756}, {obj_id:735, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85376}, {obj_id:737, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55868}, {obj_id:739, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:75504}, {obj_id:749, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:773, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:603737}, {obj_id:774, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58158}, {obj_id:775, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:84541}, {obj_id:776, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66315}, {obj_id:777, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200157}, {obj_id:778, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:277379}, {obj_id:780, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279299}, {obj_id:781, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54040}, {obj_id:782, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57275}, {obj_id:783, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71438}, {obj_id:785, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:108323}, {obj_id:786, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:145723}, {obj_id:787, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63771}, {obj_id:789, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:792, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:798, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:473748}, {obj_id:800, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55724}, {obj_id:802, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55756}, {obj_id:805, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85376}, {obj_id:807, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55868}, {obj_id:809, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:75504}, {obj_id:820, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:842, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46504}, {obj_id:843, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53171}, {obj_id:844, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55852}, {obj_id:845, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:116904}, {obj_id:846, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:258163}, {obj_id:847, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:80606}, {obj_id:848, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127311}, {obj_id:849, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54894}, {obj_id:855, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:81073}, {obj_id:856, tenant_id:1002, log_del_time:9223372036854775807 [2025-08-14 09:05:48.727636] INFO [COMMON] loop_finished_dag_net_list (ob_tenant_dag_scheduler.cpp:3868) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13499] loop finsihed dag net list(ret=0, finished_dag_net_list_size=0) [2025-08-14 09:05:48.765198] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2800) [807][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=1041560] schedule next cache evict task(evict_interval=5000000) [2025-08-14 09:05:48.767309] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=2105][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:48.733744] WDIAG [SHARE] batch_get_tablet_ls_cache (ob_tablet_to_ls_operator.cpp:656) [1026][TabletLSAUp1][T0][YB42AC4D0007-00063C496DC99CA7-0-0] [lt=976986][errcode=-4012] fail to inner batch get by sql(ret=-4012, ret="OB_TIMEOUT", tenant_id=1002, tablet_ids=[{id:1152921504606847305}, {id:200220}, {id:200225}, {id:1152921504606847222}], start_idx=0, end_idx=4) [2025-08-14 09:05:48.747253] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=1] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:48.777776] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=30476][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:48.766274] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=5983193] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [663]OB_MALLOC COST TOO MUCH TIME, cost_time=944776, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=41, WASH_BLOCK_END=944696, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=39], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:47.807527] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [663][T1001_StandbySc][T1001][YB42AC4D0007-00063C495C7A03A2-0-0] [lt=55080][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1915276, , BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [474]OB_MALLOC COST TOO MUCH TIME, cost_time=962504, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 15, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=962422], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=948002, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=35, WASH_BLOCK_END=947966], seq:[0, 1, 2] [279]OB_MALLOC COST TOO MUCH TIME, cost_time=942439, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=32896, owner: ALLOC_BLOCK, click_count: 16, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=942349, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=942310, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=23, WASH_BLOCK_END=942287], seq:[0, 1, 2] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=942262, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=20, WASH_BLOCK_END=942242], seq:[0, 1, 2] [387]OB_MALLOC COST TOO MUCH TIME, cost_time=942236, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 16, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=15, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=15, ALLOC_CHUNK_END=942126, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] [832]OB_MALLOC COST TOO MUCH TIME, cost_time=939420, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=42, WASH_BLOCK_END=939378], seq:[0, 1, 2] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=938914, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=938874], seq:[0, 1, 2] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=922738, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=34, WASH_BLOCK_END=922704], seq:[0, 1, 2] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=922932, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=922892], seq:[0, 1, 2] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=922279, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=33, WASH_BLOCK_END=922246], seq:[0, 1, 2] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=921614, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=921574], seq:[0, 1, 2] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=921596, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=30, WASH_BLOCK_END=921566], seq:[0, 1, 2] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=921574, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=30, WASH_BLOCK_END=921544], seq:[0, 1, 2] [717]OB_MALLOC COST TOO MUCH TIME, cost_time=921492, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=30, WASH_BLOCK_END=921462], seq:[0, 1, 2] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=912097, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=42, WASH_BLOCK_END=912055], seq:[0, 1, 2] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=908883, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=47, WASH_BLOCK_END=908835], seq:[0, 1, 2] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=902687, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=34, WASH_BLOCK_END=902653], seq:[0, 1, 2] [808]OB_MALLOC COST TOO MUCH TIME, cost_time=852488, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=35, WASH_BLOCK_END=852453], seq:[0, 1, 2] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=844725, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=35, WASH_BLOCK_END=844690], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=593807, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=34, WASH_BLOCK_END=593773], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=141860, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=48, WASH_BLOCK_END=141812], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=802889, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=34, WASH_BLOCK_END=802855], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=766960, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=50, WASH_BLOCK_END=766910], seq:[0, 1, 2] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=426982, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=42, WASH_BLOCK_END=426940], seq:[0, 1, 2] [552]OB_MALLOC COST TOO MUCH TIME, cost_time=962081, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=962080], seq:[0] [552]OB_MALLOC COST TOO MUCH TIME, cost_time=963948, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=56, WASH_BLOCK_END=48, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=962094], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:48.778494] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=57908] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:49.743943] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=963983] temporary file module use no blocks [2025-08-14 09:05:49.743956] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:49.743973] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=15] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:49.743991] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=18] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623250218082, flush_io_finished_ret_:0, flush_io_finished_round_:26413, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:49.736729] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=1438] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [398]OB_MALLOC COST TOO MUCH TIME, cost_time=964390, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=948502, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=15843], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:48.779326] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=569] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:49.745211] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=964424][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:47.720075] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=909424][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2029240, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:47.807614] EDIAG [RS] skip_check_member_list_change_ (ob_ls_recovery_stat_handler.cpp:109) [777][T1002_TeRec][T1002][YB42AC4D0007-00063C495DDA0097-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1941969, , BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x1067fb2c 0x1067ecc2 0x1067e03d 0x1002f2a3 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:47.256213] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2493041, , BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:49.638766] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=216] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:49.743538] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=202] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:49.753603] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10049][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:48.777370] WDIAG [SHARE.LOCATION] batch_renew_tablet_ls_cache (ob_tablet_ls_service.cpp:545) [1026][TabletLSAUp1][T0][YB42AC4D0007-00063C496DC99CA7-0-0] [lt=43549][errcode=-4664] the sql used to get tablets locations error, treat as timeout(ret=-4664, ret="OB_GET_LOCATION_TIME_OUT", previous_ret=-4012, tablet_list=[{id:1152921504606847305}{id:200220}{id:200225}{id:1152921504606847222}]) [2025-08-14 09:05:49.745448] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=964109][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:49.753015] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=942510] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [399]OB_MALLOC COST TOO MUCH TIME, cost_time=965996, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=965967], seq:[0, 1, 2, 3, 4] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=966034, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=966021], seq:[0, 1] [2025-08-14 09:05:49.756531] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=969339][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:50.724256] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=966248] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:50.724757] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:603) [1026][TabletLSAUp1][T0][YB42AC4D0007-00063C496DC99CA7-0-0] [lt=1517][errcode=-4664] fail to batch process task(ret=-4664) [2025-08-14 09:05:50.724777] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:502) [1026][TabletLSAUp1][T0][YB42AC4D0007-00063C496DC99CA7-0-0] [lt=18][errcode=-4664] fail to batch execute task(ret=-4664, tasks.count()=4) [2025-08-14 09:05:49.736160] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=273] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:50.724964] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=987310][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:50.725768] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=679] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:50.727083] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:288) [732][T1001_ArcFetche][T1001][YB42AC4D0007-00063C495CA99C16-0-0] [lt=11306] ObArchiveFetcher is running(thread_index=0) [2025-08-14 09:05:49.725788] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=888153] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:50.727308] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=999922] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:50.727357] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:50.727400] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:05:50.727413] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:50.727426] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:49.807339] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=944477] gc stale ls task succ [2025-08-14 09:05:49.756412] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=3390] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:50.736432] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=978538] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623250248774, flush_io_finished_ret_:0, flush_io_finished_round_:26394, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:48.755383] EDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12357][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1978698, , BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:48.755383] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12357][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:50.737113] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1978698] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:50.737539] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=20] log compressor cycles once. (ret=0, cost_time=2950375, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25879289856, fast_delete_log_mode=false) [2025-08-14 09:00:23.782485] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:951) [767][T1002_Occam][T1][YB42AC4D0007-00063C495F4A12FA-0-0] [lt=8][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2025-08-14 09:00:23.782485] EDIAG [SERVER] query (ob_inner_sql_connection.cpp:951) [767][T1002_Occam][T1][YB42AC4D0007-00063C495F4A12FA-0-0] [lt=8][errcode=-4283] LOGGER COST TOO MUCH TIME, cost: 326461481, owner: ObLog, click_count: 5, time dist:[FORMAT_END=326461470, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=7], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x124dff1f 0x96a5a77 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x9c34403 0x9c31cbc 0x122def07 0x123237cf 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:50.742202] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:50.743272] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:50.744081] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=30] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:50.744886] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=0] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:50.744667] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=796] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:50.750554] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=9532] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:50.753052] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=24390] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:05:50.753127] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=942698] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:50.753257] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=944539] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:50.754110] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=290] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:50.757079] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1170] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:50.757439] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=741] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:50.757474] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=625] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:50.757863] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:50.757905] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=624] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:50.759254] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1527] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:50.761340] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=200] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:50.750674] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:04:55.768264] WDIAG [SQL.DAS] execute_all_task (ob_das_ref.cpp:428) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=109842074][errcode=-4038] fail to retry das tasks(ret=-4038, failed_tasks=[{parent:{tenant_id:1, task_id:50056313, op_type:2, errcode:-4038, can_part_retry:0, task_started:1, in_part_retry:0, in_stmt_retry:0, need_switch_param:0, trans_desc:{this:0x7f98c907e020, tx_id:{txid:21195872}, state:1, addr:"172.77.0.7:2882", tenant_id:1, session_id:1, assoc_session_id:1, client_sid:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1755162170771423, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999987, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1753069619, cluster_version:17180067074, seq_base:1755162170724889, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:2}, snapshot:{this:0x7f9956cd2798, valid:true, source:5, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:21195872}, scn:{branch:0, seq:2}}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, tablet_id:{id:154}, ls_id:{id:1}, tablet_loc:{tablet_id:{id:154}, ls_id:{id:1}, server:"172.77.0.7:2882", loc_meta:{table_loc_id:18446744073709551614, ref_table_id:154, related_table_ids:[101015, 101016], use_dist_das:0, select_leader:1, is_dup_table:0, is_weak_read:0, unuse_related_pruning:0, is_external_table:0, route_policy:0}, in_retry:0, partition_id:18446744073709551615, first_level_part_id:18446744073709551615}, related_ctdefs:[{ObDASBaseCtDef:{ObDASBaseDef:{op_type:2, children_cnt:0}, table_id:18446744073709551614, index_tid:101015, schema_version:1753069637599528, rowkey_cnt:4, spk_cnt:0, column_ids:[19, 16, 17, 18], column_types:[{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}], column_accuracys:[{length:64, precision:-1, scale:-1}, {length:-1, precision:25, scale:6}, {length:46, precision:-1, scale:-1}, {length:-1, precision:20, scale:0}], old_row_projector:[], new_row_projector:[3, 0, 1, 2], is_total_quantity_log:1, is_ignore:0, is_batch_stmt:0, is_insert_up:0, is_table_api:0, is_main_table_in_fts_ddl:0, tz_info:{id:-1, offset:28800, error_on_overlap_time:false}, table_param:{tenant_schema_version:1753073979411448, data_table:{table_id:101015, schema_version:1753069637599528, table_type:5, index_type:1, index_status:2, shadow_rowkey_column_num:0, doc_id_col_id:18446744073709551615, fulltext_col_id:18446744073709551615, index_name:"idx_server_module", fts_parser_name:"", fts_parser_properties:"", pk_name:"", columns:[{column_id:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:64, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:25, scale:6}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"EXT":"CURRENT_TIMESTAMP"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}, {column_id:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:46, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}, {column_id:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}], read_info:{schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, trans_col_index:-1, mview_old_new_col_index:-1, group_idx_col_index:-1, seq_read_column_count:4, max_col_index:3, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 2, 3], data:0x7f9827a3ae60, count:4, allocator:0x7f9827a1e6f0, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 3], data:0x7f9827a3ae80, count:4, allocator:0x7f9827a1e6f0, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9827a3ae20, count:4, allocator:0x7f9827a1e6f0, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, has_all_column_group:true, need_truncate_filter:false}, lob_inrow_threshold:4096}, col_descs:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], col_map:{is_inited_:true, use_array_:true, shadow_use_array_:false, has_:true, has_shadow_:false}}, encrypt_meta:[]}, table_rowkey_cids:[], table_rowkey_types:[]}, {ObDASBaseCtDef:{ObDASBaseDef:{op_type:2, children_cnt:0}, table_id:18446744073709551614, index_tid:101016, schema_version:1753069637600200, rowkey_cnt:4, spk_cnt:0, column_ids:[20, 16, 17, 18], column_types:[{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}], column_accuracys:[{length:64, precision:-1, scale:-1}, {length:-1, precision:25, scale:6}, {length:46, precision:-1, scale:-1}, {length:-1, precision:20, scale:0}], old_row_projector:[], new_row_projector:[4, 0, 1, 2], is_total_quantity_log:1, is_ignore:0, is_batch_stmt:0, is_insert_up:0, is_table_api:0, is_main_table_in_fts_ddl:0, tz_info:{id:-1, offset:28800, error_on_overlap_time:false}, table_param:{tenant_schema_version:1753073979411448, data_table:{table_id:101016, schema_version:1753069637600200, table_type:5, index_type:1, index_status:2, shadow_rowkey_column_num:0, doc_id_col_id:18446744073709551615, fulltext_col_id:18446744073709551615, index_name:"idx_server_event", fts_parser_name:"", fts_parser_properties:"", pk_name:"", columns:[{column_id:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:64, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:25, scale:6}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"EXT":"CURRENT_TIMESTAMP"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}, {column_id:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:46, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}, {column_id:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}], read_info:{schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, trans_col_index:-1, mview_old_new_col_index:-1, group_idx_col_index:-1, seq_read_column_count:4, max_col_index:3, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 2, 3], data:0x7f9827a3baf0, count:4, allocator:0x7f9827a1e6f0, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 3], data:0x7f9827a3bb10, count:4, allocator:0x7f9827a1e6f0, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9827a3bab0, count:4, allocator:0x7f9827a1e6f0, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, has_all_column_group:true, need_truncate_filter:false}, lob_inrow_threshold:4096}, col_descs:[column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], col_map:{is_inited_:true, use_array_:true, shadow_use_array_:false, has_:true, has_shadow_:false}}, encrypt_meta:[]}, table_rowkey_cids:[], table_rowkey_types:[]}], related_rtdefs:[{ObDASBaseRtDef:{ObDASBaseRtDef:{op_type:2, children_cnt:0}, timeout_ts:1755162198653066, sql_mode:4194304, prelock:false, tenant_schema_version:1753073979411448, is_for_foreign_key_check:false, affected_rows:0, is_immediate_row_conflict_check:true}, need_fetch_conflict:false, is_duplicated:false, direct_insert_task_id:0, use_put:false, ddl_task_id:0}, {ObDASBaseRtDef:{ObDASBaseRtDef:{op_type:2, children_cnt:0}, timeout_ts:1755162198653066, sql_mode:4194304, prelock:false, tenant_schema_version:1753073979411448, is_for_foreign_key_check:false, affected_rows:0, is_immediate_row_conflict_check:true}, need_fetch_conflict:false, is_duplicated:false, direct_insert_task_id:0, use_put:false, ddl_task_id:0}], task_status:1, related_tablet_ids:[{id:101015}, {id:101016}], das_task_node:, plan_line_id:0}, ins_ctdef:{ObDASBaseCtDef:{ObDASBaseDef:{op_type:2, children_cnt:0}, table_id:18446744073709551614, index_tid:154, schema_version:1753069637597680, rowkey_cnt:3, spk_cnt:0, column_ids:[16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33], column_types:[{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"LONGTEXT", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}], column_accuracys:[{length:-1, precision:25, scale:6}, {length:46, precision:-1, scale:-1}, {length:-1, precision:20, scale:0}, {length:64, precision:-1, scale:-1}, {length:64, precision:-1, scale:-1}, {length:256, precision:-1, scale:-1}, {length:512, precision:-1, scale:-1}, {length:256, precision:-1, scale:-1}, {length:536870911, precision:-1, scale:-1}, {length:256, precision:-1, scale:-1}, {length:512, precision:-1, scale:-1}, {length:256, precision:-1, scale:-1}, {length:512, precision:-1, scale:-1}, {length:256, precision:-1, scale:-1}, {length:512, precision:-1, scale:-1}, {length:256, precision:-1, scale:-1}, {length:512, precision:-1, scale:-1}, {length:512, precision:-1, scale:-1}], old_row_projector:[], new_row_projector:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17], is_total_quantity_log:1, is_ignore:0, is_batch_stmt:0, is_insert_up:0, is_table_api:0, is_main_table_in_fts_ddl:0, tz_info:{id:-1, offset:28800, error_on_overlap_time:false}, table_param:{tenant_schema_version:1753073979411448, data_table:{table_id:154, schema_version:1753069637597680, table_type:0, index_type:0, index_status:0, shadow_rowkey_column_num:0, doc_id_col_id:18446744073709551615, fulltext_col_id:18446744073709551615, index_name:"", fts_parser_name:"", fts_parser_properties:"", pk_name:"", columns:[{column_id:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:25, scale:6}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"EXT":"CURRENT_TIMESTAMP"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}, {column_id:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:46, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}, {column_id:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}, {column_id:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:64, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:64, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:256, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:512, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:256, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:24, meta_type:{type:"LONGTEXT", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:536870911, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:25, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:256, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:26, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:512, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:27, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:256, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:28, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:512, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:29, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:256, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:30, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:512, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:31, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:256, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:32, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:512, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:33, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:512, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}], read_info:{schema_column_count:18, schema_rowkey_cnt:3, rowkey_cnt:5, trans_col_index:-1, mview_old_new_col_index:-1, group_idx_col_index:-1, seq_read_column_count:3, max_col_index:19, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:18, 0, 1, 2, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19], data:0x7f9827a39e20, count:18, allocator:0x7f9827a1e6f0, capacity:18, init_cnt:18}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:18, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17], data:0x7f9827a39e70, count:18, allocator:0x7f9827a1e6f0, capacity:18, init_cnt:18}}, cols_desc:{ObIArray:[cnt:18, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=23 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=24 {type:"LONGTEXT", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=25 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=26 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=27 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=28 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=29 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=30 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=31 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=32 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=33 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0], data:0x7f9827a39d40, count:18, allocator:0x7f9827a1e6f0, capacity:18, init_cnt:18}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, has_all_column_group:true, need_truncate_filter:false}, lob_inrow_threshold:4096}, col_descs:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=23 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=24 {type:"LONGTEXT", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=25 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=26 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=27 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=28 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=29 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=30 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=31 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=32 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=33 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0], col_map:{is_inited_:true, use_array_:true, shadow_use_array_:false, has_:true, has_shadow_:false}}, encrypt_meta:[]}, table_rowkey_cids:[], table_rowkey_types:[]}, ins_rtdef:{ObDASBaseRtDef:{ObDASBaseRtDef:{op_type:2, children_cnt:0}, timeout_ts:1755162198653066, sql_mode:4194304, prelock:false, tenant_schema_version:1753073979411448, is_for_foreign_key_check:false, affected_rows:0, is_immediate_row_conflict_check:true}, need_fetch_conflict:false, is_duplicated:false, direct_insert_task_id:0, use_put:false, ddl_task_id:0}, insert_buffer:{mem_attr:tenant_id=1, label=DASInsertBuffer, ctx_id=0, prio=0, buffer_memory:427, buffer_size:1, datum_store:NULL}}]) [2025-08-14 09:05:50.778170] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:50.780536] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=201] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:50.782327] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:50.807406] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=918895] gc stale ls task succ [2025-08-14 09:05:50.808934] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=184] gc stale ls task succ [2025-08-14 09:05:50.824258] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=901669] gc stale ls task succ [2025-08-14 09:05:50.827670] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:05:50.827723] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:50.827774] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:50.827816] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:50.829252] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=79506][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:50.955373] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:50.979805] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=222] PNIO [ratelimit] time: 1755162350979803, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007614, add_bytes: 0 [2025-08-14 09:05:51.159603] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:51.258794] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:51.651428] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=224] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:51.712519] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:51.721479] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3196] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:51.725889] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:04:55.768264] EDIAG [SQL.DAS] execute_all_task (ob_das_ref.cpp:428) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=109842074][errcode=-4038] LOGGER COST TOO MUCH TIME, cost: 54925797, owner: ObLog, click_count: 5, time dist:[FORMAT_END=54910024, ObMalloc#start=2, ObMalloc#end=26, ALLOC_END=15740, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x15ebc895 0x9344421 0x9615b91 0x9613406 0x9613912 0x92df373 0x932ff51 0x932e8be 0x96ac524 0x96a672d 0x96a5033 0x97aa16f 0x124eac91 0x97a9944 0x261395f9 0xfa0e430 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:50.827872] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:05:51.729056] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=899813] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:51.729131] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=73] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:50.853055] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:51.732952] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=217] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:51.735255] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=318] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:51.740256] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:50.774513] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16533] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:50.781997] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=172] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:51.741971] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=958523][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:50.749159] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [712][T1001_GCCollect][T1][YB42AC4D0007-00063C495FA9A613-0-0] [lt=4739][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:51.742361] WDIAG [SQL.DAS] execute_all_task (ob_das_ref.cpp:320) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=54925797][errcode=-4038] fail to execute all agg_tasks(ret=-4038) [2025-08-14 09:05:51.744179] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=17] temporary file module use no blocks [2025-08-14 09:05:51.744199] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=19] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:51.744212] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:51.744227] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623253183871, flush_io_finished_ret_:0, flush_io_finished_round_:26415, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:51.749822] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:51.757626] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:51.758414] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:51.759255] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:51.761304] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:50.762005] EDIAG [RS] get_valid_sts_after (ob_tenant_info_loader.cpp:454) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1002720, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1002693, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=1, APPEND_END=22], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1001d942 0x996d843 0x996d15a 0x996cfd8 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:51.766274] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1002720] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:05:51.766300] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:50.761343] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=38536] [COMPACTION DAG STATUS] (start_time=1755162339744361, end_time=1755162350757569, dag_status={}) [2025-08-14 09:05:50.761343] EDIAG [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=38536][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1003488, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1003466, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=19], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1af3d850 0x1af3c294 0x20071394 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:51.745283] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2456] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:51.766782] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21464][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:51.767342] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=35] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:51.767370] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=24] temporary file module use no blocks [2025-08-14 09:05:51.767380] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:51.767403] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=21] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:51.767416] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623253176237, flush_io_finished_ret_:0, flush_io_finished_round_:26395, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:51.742811] WDIAG [SQL.ENG] submit_all_dml_task (ob_table_modify_op.cpp:1190) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=445][errcode=-4038] execute all dml das task failed(ret=-4038) [2025-08-14 09:05:51.767492] WDIAG [SQL.ENG] inner_get_next_row (ob_table_modify_op.cpp:1279) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=24637][errcode=-4038] failed to submit the remaining dml tasks(ret=-4038) [2025-08-14 09:05:51.768453] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=965770] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:51.768482] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623252197605, flush_io_finished_ret_:0, flush_io_finished_round_:26375, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:51.775923] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:51.778939] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:51.780382] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:51.781059] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:51.782338] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:51.807470] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:05:51.808999] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=14] gc stale ls task succ [2025-08-14 09:05:51.824315] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=23] gc stale ls task succ [2025-08-14 09:05:51.829333] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:05:51.829389] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:51.829434] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:51.829477] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:51.829515] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:05:51.829532] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:51.829548] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:51.929731] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:51.955871] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:51.987423] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=24] PNIO [ratelimit] time: 1755162351987421, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007618, add_bytes: 0 [2025-08-14 09:05:51.998016] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:52.160183] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:52.260140] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:52.629539] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:52.652184] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=25] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:52.719973] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [807][T1002_TimerWK3_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=16294] get_disk_usage(ret=0, capacity(MB):=24883, used(MB):=8642) [2025-08-14 09:05:52.720026] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:52.720726] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:52.726002] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:52.728369] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:51.767517] WDIAG [SQL.ENG] get_next_row (ob_operator.cpp:1305) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=17][errcode=-4038] inner get next row failed(ret=-4038, type=24, op="PHY_INSERT", op_id=0) [2025-08-14 09:05:52.733467] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:52.737275] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:52.738608] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=34035] LogLoopThread round_cost_time(us)(round_cost_time=19975478) [2025-08-14 09:05:52.740516] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:52.742280] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2563] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 08:58:47.789107] EDIAG [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [15402][T1001_TimerWK10_CKClogDisk][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1966175, owner: ObLog, click_count: 7, time dist:[FORMAT_END=5, ObMalloc#start=1, ObMalloc#end=995148, ObMalloc#start=971016, ObMalloc#end=3, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x9870878 0x1c3fed66 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:56.780444] EDIAG [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:147) [14963][T1_TimerWK9_][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1984219, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1984205, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=10], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1c450593 0x1c44d3f9 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:52.743169] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [15402][T1001_TimerWK10_CKClogDisk][T1001][Y0-0000000000000000-0-0] [lt=1966175] get_disk_usage(ret=0, capacity(MB):=2764, used(MB):=2181) [2025-08-14 09:05:52.743198] WDIAG alarm_if_necessary (ob_timer_service.cpp:143) [15402][T1001_TimerWK10][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:52.744386] WDIAG [SQL.EXE] get_next_row (ob_execute_result.cpp:145) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=962955][errcode=-4038] get next row from operator failed(ret=-4038) [2025-08-14 09:05:52.744418] WDIAG [SQL] inner_get_next_row (ob_result_set.cpp:453) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=27][errcode=-4038] get next row from exec result failed(ret=-4038) [2025-08-14 09:05:52.745261] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=9904] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:52.745301] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:52.748247] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=236] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:52.750755] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=41] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:52.750805] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=45] temporary file module use no blocks [2025-08-14 09:05:52.750816] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:52.750831] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:52.750856] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=24] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623254184036, flush_io_finished_ret_:0, flush_io_finished_round_:26416, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:52.753520] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=300] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:05:52.754568] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=206] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:52.755326] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=292] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:52.755364] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=446] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:52.757484] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:52.757996] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:52.758459] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:52.760571] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:52.761587] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2304] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:52.767305] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4076] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:52.767510] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=24] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:52.769525] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:52.769552] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=23] temporary file module use no blocks [2025-08-14 09:02:58.897295] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [874][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E899C16-0-0] [lt=861935][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2846978, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2846976, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3c61 0x113d348b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [736]OB_MALLOC COST TOO MUCH TIME, cost_time=970818, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=970811], seq:[0] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=946728, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=16, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=946685, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=946805, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=946738, WASH_BLOCK_END=67], seq:[0, 1, 2] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=943968, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=30, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=943905, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=943985, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=943977, WASH_BLOCK_END=8], seq:[0, 1, 2] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=943153, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=943121, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=943221, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=943176, WASH_BLOCK_END=44], seq:[0, 1, 2] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=943030, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=943003, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=943064, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=943040, WASH_BLOCK_END=24], seq:[0, 1, 2] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=923045, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=923000, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=923066, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=923056, WASH_BLOCK_END=10], seq:[0, 1, 2] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=916625, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=916592, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=916644, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=916635, WASH_BLOCK_END=9], seq:[0, 1, 2] [808]OB_MALLOC COST TOO MUCH TIME, cost_time=904962, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=904920, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [808]OB_MALLOC COST TOO MUCH TIME, cost_time=904982, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=904972, WASH_BLOCK_END=10], seq:[0, 1, 2] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=901577, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=901543, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=901600, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=901591, WASH_BLOCK_END=9], seq:[0, 1, 2] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=895705, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=895667, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=895723, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=895715, WASH_BLOCK_END=8], seq:[0, 1, 2] [466]OB_MALLOC COST TOO MUCH TIME, cost_time=890629, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=890597, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [466]OB_MALLOC COST TOO MUCH TIME, cost_time=890652, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=890644, WASH_BLOCK_END=8], seq:[0, 1, 2] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=887700, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=887671, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=887715, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=887708, WASH_BLOCK_END=7], seq:[0, 1, 2] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=887786, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=887758, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=887802, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=887794, WASH_BLOCK_END=8], seq:[0, 1, 2] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=859148, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=15, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=859107, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=859163, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=859155, WASH_BLOCK_END=8], seq:[0, 1, 2] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=788092, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=788059, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=788108, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=788101, WASH_BLOCK_END=7], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=731906, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=731874, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=565826, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=565795, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=731933, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=731921, WASH_BLOCK_END=12], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=565849, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=565841, WASH_BLOCK_END=8], seq:[0, 1, 2] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=466272, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=466237, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=466291, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=466282, WASH_BLOCK_END=9], seq:[0, 1, 2] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=919408, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=919378, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=919425, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=919418, WASH_BLOCK_END=7], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=770782, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=770751, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=770798, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=770790, WASH_BLOCK_END=8], seq:[0, 1, 2] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=979958, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=65, WASH_BLOCK_END=103, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=970832], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:52.753932] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=217] start do ls ha handler(ls_id_array_=[{id:1}]) [832]OB_MALLOC COST TOO MUCH TIME, cost_time=964563, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=964562], seq:[0] [832]OB_MALLOC COST TOO MUCH TIME, cost_time=964575, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=964571], seq:[0, 1] [2025-08-14 09:05:52.769557] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=5] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:53.735613] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=964593] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:53.735632] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=18] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623254208279, flush_io_finished_ret_:0, flush_io_finished_round_:26376, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:53.735738] INFO [SQL.PC] need_check_schema_version (ob_plan_cache_value.cpp:2334) [337][EvtHisUpdTask][T1][YB42AC4D0007-00063C4960299C79-0-0] [lt=25] need check schema(new_schema_version=1753073979411448, cached_tenant_schema_version=1753073979411448, is_contain_synonym()=false, contain_sys_name_table_=true, is_contain_tmp_tbl()=false, is_contain_sys_pl_object()=false, need_check=true, constructed_sql_=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(?), ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)) [2025-08-14 09:05:53.737071] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:53.726966] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [14963][T1_TimerWK9_][T1][Y0-0000000000000000-0-0] [lt=1984219] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-08-14 09:05:53.737205] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:275) [14963][T1_TimerWK9_][T1][Y0-0000000000000000-0-0] [lt=10219] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=9) [2025-08-14 09:05:53.737224] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [14963][T1_TimerWK9][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:52.766935] WDIAG [SQL] drive_dml_query (ob_result_set.cpp:1557) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=12][errcode=-4038] do dml query failed(ret=-4038) [2025-08-14 09:05:53.737373] WDIAG [SQL] open_result (ob_result_set.cpp:193) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=968967][errcode=-4038] fail to drive dml query(ret=-4038) [2025-08-14 09:05:53.737389] WDIAG [SQL] open (ob_result_set.cpp:139) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=12][errcode=-4038] open result set failed(ret=-4038) [2025-08-14 09:05:53.740604] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:53.745309] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:53.745341] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:53.745425] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [880][T1002_ArcSender][T1002][YB42AC4D0007-00063C495EB99C16-0-0] [lt=2672] ObArchiveSender is running(thread_index=5) [2025-08-14 09:05:53.726295] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:53.748355] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=16] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:53.749312] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=31] log compressor cycles once. (ret=0, cost_time=962780, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25878777856, fast_delete_log_mode=false) [2025-08-14 09:05:53.751029] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:53.751058] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=23] temporary file module use no blocks [2025-08-14 09:05:53.751068] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:53.751082] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:53.751097] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623255190654, flush_io_finished_ret_:0, flush_io_finished_round_:26417, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:53.753634] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=12] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:05:53.754270] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:53.754650] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=15] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:53.757355] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:53.758258] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:53.758489] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:53.758607] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:53.759300] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:53.762683] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:53.763984] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:53.767577] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=19] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:53.767607] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=24] temporary file module use no blocks [382]OB_MALLOC COST TOO MUCH TIME, cost_time=936160, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 10, time dist:[ALLOC_CHUNK_END=936126, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=468267, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=39, WASH_BLOCK_END=468228], seq:[0, 1, 2] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=881986, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=33, WASH_BLOCK_END=881952], seq:[0, 1, 2] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=687517, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=42, WASH_BLOCK_END=687475], seq:[0, 1, 2] [733]OB_MALLOC COST TOO MUCH TIME, cost_time=873088, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=42, WASH_BLOCK_END=873045], seq:[0, 1, 2] [861]OB_MALLOC COST TOO MUCH TIME, cost_time=867324, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=49, WASH_BLOCK_END=867275], seq:[0, 1, 2] [400]OB_MALLOC COST TOO MUCH TIME, cost_time=835589, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=49, WASH_BLOCK_END=835540], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=714707, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=38, WASH_BLOCK_END=714669], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=687656, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=687616], seq:[0, 1, 2] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=891982, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=55, WASH_BLOCK_END=891925], seq:[0, 1, 2] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=937178, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=936190, WASH_BLOCK_END=987], seq:[0, 1, 2] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=908263, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=45, WASH_BLOCK_END=908218], seq:[0, 1, 2] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=888461, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=35, WASH_BLOCK_END=888425], seq:[0, 1, 2] [808]OB_MALLOC COST TOO MUCH TIME, cost_time=689324, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=43, WASH_BLOCK_END=689281], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=487750, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=49, WASH_BLOCK_END=487701], seq:[0, 1, 2] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=906326, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=46, WASH_BLOCK_END=906280], seq:[0, 1, 2] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=940416, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=32, WASH_BLOCK_END=935845, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=4538], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:53.781533] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7978] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:54.729530] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=946563][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=928560, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=43, WASH_BLOCK_END=908733, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=19778], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:53.807621] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=919905] gc stale ls task succ [2025-08-14 09:05:54.729855] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=76076] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:54.730473] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:54.733445] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=211] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:54.743425] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=230] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:54.745442] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:54.723513] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=954746] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:54.746178] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=22629] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:54.746192] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:54.746201] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:54.746210] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:54.746219] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:54.746228] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:54.746251] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=22] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:05:54.748258] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=20958] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:54.748371] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=14] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:54.751190] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=23] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:54.751212] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=17] temporary file module use no blocks [2025-08-14 09:05:54.751226] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:54.753830] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=12] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:05:54.751248] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=20] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:53.761248] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:54.754426] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=991668] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:05:54.755118] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=16] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:54.756721] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:54.756763] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=64] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:54.757485] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=62] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:54.758893] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:54.759134] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=33] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [800]OB_MALLOC COST TOO MUCH TIME, cost_time=975679, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=975677], seq:[0] [800]OB_MALLOC COST TOO MUCH TIME, cost_time=975899, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=55, WASH_BLOCK_END=138, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=975701], seq:[0, 1, 2, 3, 4] [2025-08-14 09:05:54.759928] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [777]OB_MALLOC COST TOO MUCH TIME, cost_time=973895, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=973889], seq:[0] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=975284, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=43, WASH_BLOCK_END=1326, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=973909], seq:[0, 1, 2, 3, 4] [2025-08-14 08:59:14.765614] EDIAG [SERVER] detect_master_rs_ls (ob_service.cpp:2601) [716][T1_L0_G9][T1][YB42AC4D0007-00063C4977899C56-0-0] [lt=7725][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1984643, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1982976, ALLOC_END=1663, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x13b5e532 0x13b5cb6b 0x1254a2a5 0x984d055 0x9280839 0x926c154 0x12c5a0c9 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [772]OB_MALLOC COST TOO MUCH TIME, cost_time=973853, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=973824, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=973913, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=973862, WASH_BLOCK_END=51], seq:[0, 1, 2] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=964411, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=964378, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=964478, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=964433, WASH_BLOCK_END=45], seq:[0, 1, 2] [2025-08-14 09:05:55.739849] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=223] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:55.724742] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [726][T1001_ObXAHbWor][T1001][Y0-0000000000000000-0-0] [lt=961782] XA scheduler heartbeat task statistics(avg_time=1) [2025-08-14 09:05:55.740033] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=975930] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [382]OB_MALLOC COST TOO MUCH TIME, cost_time=964313, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=964289], seq:[0, 1, 2, 3, 4] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=964326, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=964325], seq:[0, 1] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=963492, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=963463, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=937481, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=937440, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=963521, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=963503, WASH_BLOCK_END=18], seq:[0, 1, 2] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=937511, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=937497, WASH_BLOCK_END=14], seq:[0, 1, 2] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=936061, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=936032, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=936088, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=936071, WASH_BLOCK_END=17], seq:[0, 1, 2] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=920883, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=920849, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=920902, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=920893, WASH_BLOCK_END=9], seq:[0, 1, 2] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=819948, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=819918, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=819973, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=819961, WASH_BLOCK_END=12], seq:[0, 1, 2] [2025-08-14 09:05:55.746281] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=212] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:54.780552] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:55.746358] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=964346][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:54.781620] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:55.746643] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=963564][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [373]OB_MALLOC COST TOO MUCH TIME, cost_time=815367, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=815343], seq:[0, 1, 2, 3, 4] [373]OB_MALLOC COST TOO MUCH TIME, cost_time=815379, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=815376], seq:[0, 1] [2025-08-14 09:05:54.767418] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=963290][errcode=-4038] open result set failed(ret=-4038) [2025-08-14 09:05:55.748741] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=13] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [807]OB_MALLOC COST TOO MUCH TIME, cost_time=818158, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=818124, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=814618, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=814586, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=814670, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=814643, WASH_BLOCK_END=27], seq:[0, 1, 2] [807]OB_MALLOC COST TOO MUCH TIME, cost_time=818185, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=818171, WASH_BLOCK_END=14], seq:[0, 1, 2] [2025-08-14 09:05:55.736524] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2547] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:55.749156] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12612][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=738897, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=738873], seq:[0, 1, 2, 3, 4] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=738907, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=738906], seq:[0, 1] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=722989, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=722958, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=723012, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=722997, WASH_BLOCK_END=15], seq:[0, 1, 2] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=626232, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=626192, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=626248, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=626240, WASH_BLOCK_END=8], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=626274, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=626247, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=626290, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=626282, WASH_BLOCK_END=8], seq:[0, 1, 2] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=499944, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=499912, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=499961, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=499952, WASH_BLOCK_END=8], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=425670, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=425639, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=425685, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=425678, WASH_BLOCK_END=7], seq:[0, 1, 2] [764]OB_MALLOC COST TOO MUCH TIME, cost_time=116863, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=116819, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [764]OB_MALLOC COST TOO MUCH TIME, cost_time=116879, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=116871, WASH_BLOCK_END=8], seq:[0, 1, 2] [2025-08-14 09:05:55.010270] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=719389] PNIO [ratelimit] time: 1755162355010268, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007614, add_bytes: 0 [2025-08-14 09:05:55.324436] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:55.751010] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162355751010, start=1755162355324433, delta=426577) [2025-08-14 09:05:55.123335] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:55.123258] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:55.249940] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:55.026238] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:55.633813] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:55.720904] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2610] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:55.751631] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=30679][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:55.732373] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4854] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:55.751769] WDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19366][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:55.751808] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=27165] temporary file module use no blocks [2025-08-14 09:05:55.751821] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:55.751834] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:55.751847] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623257163847, flush_io_finished_ret_:0, flush_io_finished_round_:26398, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:54.824172] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:05:55.751917] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=926340][errcode=-4076] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:54.753951] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=2698] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623256190889, flush_io_finished_ret_:0, flush_io_finished_round_:26418, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:55.738883] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:55.752435] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=989525][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:55.752662] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:54.923481] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=840648][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:55.755207] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=13] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:55.756337] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:55.759584] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=23421] finish to resize log external storage handler(current_ts=1755162355752727, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-08-14 09:05:55.766396] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=6798] snapshot_traffic_info once [2025-08-14 09:05:55.748204] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=821281] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:55.766519] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18242] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:55.766564] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:55.766612] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:05:55.766632] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:55.766651] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:55.766803] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=21514][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:04.773820] EDIAG [SHARE.LOCATION] dump_statistic (ob_tablet_location_refresh_service.cpp:240) [365][TbltRefreshSer][T0][YB42AC4D0007-00063C4960B99C42-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2963663, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2963657, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x2023bce8 0x2023daf3 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:55.769423] WDIAG [SQL] open_plan (ob_result_set.cpp:104) [712][T1001_GCCollect][T1][YB42AC4D0007-00063C495FA9A613-0-0] [lt=991478][errcode=-4012] query is timeout(ret=-4012, timeout_ts=1755162036732199, start_time=1755162012762844) [2025-08-14 09:05:55.769455] WDIAG [SQL] open (ob_result_set.cpp:137) [712][T1001_GCCollect][T1][YB42AC4D0007-00063C495FA9A613-0-0] [lt=30][errcode=-4012] execute plan failed(ret=-4012) [2025-08-14 09:05:55.769465] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [712][T1001_GCCollect][T1][YB42AC4D0007-00063C495FA9A613-0-0] [lt=8][errcode=-4012] open result set failed(ret=-4012) [2025-08-14 09:05:55.776420] INFO print_info (ob_kvcache_hazard_domain.cpp:138) [981][T1002_L0_G0][T1002][YB42AC4D0007-00063C49634A0941-0-0] [lt=30] [KVCACHE-HAZARD-POINTER] retired memory size: 0, blocks size: 94, free list size: 1003, memory_efficiency: 0.897691 [2025-08-14 09:05:55.776441] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:1131) [981][T1002_L0_G0][T1002][YB42AC4D0007-00063C49634A0941-0-0] [lt=19][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:55.776453] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:826) [981][T1002_L0_G0][T1002][YB42AC4D0007-00063C49634A0941-0-0] [lt=9] can not find enough memory block to wash(ret=-4273, size_washed=0, size_to_wash=6291456) [2025-08-14 09:05:55.778082] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5382] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:55.780448] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:55.780724] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:55.781879] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=599] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:55.807750] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=937669] gc stale ls task succ [2025-08-14 09:05:55.809255] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=936361] gc stale ls task succ [2025-08-14 09:05:55.824580] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=922816] gc stale ls task succ [2025-08-14 09:05:55.832364] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:55.833227] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [466][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:55.931141] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=889317][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:55.952058] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:05:55.966892] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:05:55.966949] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:55.966995] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:55.967040] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:55.967088] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:05:55.967104] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:56.015479] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:56.017890] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=738934] PNIO [ratelimit] time: 1755162356017888, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007620, add_bytes: 0 [2025-08-14 09:05:56.151222] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:56.151305] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:56.253974] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:56.274105] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:56.351377] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:56.721262] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:55.967119] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:55.776470] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:845) [981][T1002_L0_G0][T1002][YB42AC4D0007-00063C49634A0941-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_to_wash=6291456) [2025-08-14 09:05:56.729822] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:66) [981][T1002_L0_G0][T1002][YB42AC4D0007-00063C49634A0941-0-0] [lt=951907][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-08-14 09:05:56.729863] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:93) [981][T1002_L0_G0][T1002][YB42AC4D0007-00063C49634A0941-0-0] [lt=40][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-08-14 09:05:55.761750] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:801) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=978088][errcode=-4038] result set open failed(ret=-4038, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1755162162745681), 'ELECTION', 'decentralized to be leader', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'server|lease:["172.77.0.7:2882"|2025-08-14 08:44:58.850]', 'HAPPENED_TIME', '2025-08-14 08:44:55.757588', '', '', '172.77.0.7', 2882)"}) [2025-08-14 09:05:56.732593] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:56.733525] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=982573][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:55.769476] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:801) [712][T1001_GCCollect][T1][YB42AC4D0007-00063C495FA9A613-0-0] [lt=10][errcode=-4012] result set open failed(ret=-4012, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1001"}) [2025-08-14 09:05:56.734619] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:946) [712][T1001_GCCollect][T1][YB42AC4D0007-00063C495FA9A613-0-0] [lt=963682][errcode=-4012] execute failed(ret=-4012, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1001"}, retry_cnt=2, local_sys_schema_version=1753073979411448, local_tenant_schema_version=1753073979411448) [2025-08-14 09:05:56.733598] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:946) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=967208][errcode=-4038] execute failed(ret=-4038, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1755162162745681), 'ELECTION', 'decentralized to be leader', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'server|lease:["172.77.0.7:2882"|2025-08-14 08:44:58.850]', 'HAPPENED_TIME', '2025-08-14 08:44:55.757588', '', '', '172.77.0.7', 2882)"}, retry_cnt=0, local_sys_schema_version=1753073979411448, local_tenant_schema_version=1753073979411448) [2025-08-14 09:05:56.736172] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:56.736453] INFO [SHARE.LOCATION] refresh_cache_ (ob_tablet_location_refresh_service.cpp:657) [365][TbltRefreshSer][T0][YB42AC4D0007-00063C4960B99C42-0-0] [lt=2963663] [REFRESH_TABLET_LOCATION] refresh cache start(tenant_id=1002) [981]OB_MALLOC COST TOO MUCH TIME, cost_time=996539, tenant_id=1002, label=SqlExecutor, ctx_id=0, prio=0, size=245760, owner: ObMalloc, click_count: 19, time dist:[ALLOC_CHUNK_END=9, ObMalloc#start=36262, ObMalloc#end=8, ObLog#start=44, ObLog#end=19, ObLog#end=9, ObLog#start=4, ObLog#end=16, ObLog#end=951907, ObLog#start=5, ObLog#end=40, ObLog#end=17, WASH_KVCACHE_END=1, ObLog#start=8183, ObLog#end=2, WASH_BLOCK_END=10], seq:[0, 1, 2, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15, 16, 17, 18] [2025-08-14 09:05:56.741266] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:56.743333] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=34] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:56.743365] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=26] temporary file module use no blocks [2025-08-14 09:05:56.743377] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:56.743391] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:56.743407] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623258179656, flush_io_finished_ret_:0, flush_io_finished_round_:26379, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:56.515732] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:56.744266] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=568] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:56.746788] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:56.747313] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=17] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:56.750345] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=1466] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:56.751953] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=39] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:56.751981] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=24] temporary file module use no blocks [2025-08-14 09:05:56.751988] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:56.752004] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=15] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:56.743488] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:56.766749] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23224][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:56.741897] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [412][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=18268] [sql nio session](*s={this:0x7f98f733a030, session_id:3221590255, trace_id:YB42AC4D0007-00063C49651A3232-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:257}, err:0, last_decode_time:1755161952736711, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1716, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:05:56.781418] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:56.781720] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:56.781896] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:56.807817] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=124] gc stale ls task succ [2025-08-14 09:05:56.809319] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=21] gc stale ls task succ [2025-08-14 09:05:56.824642] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=22] gc stale ls task succ [2025-08-14 09:05:56.933392] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=44][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:57.018202] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:57.025501] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=21] PNIO [ratelimit] time: 1755162357025500, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007612, add_bytes: 0 [2025-08-14 09:05:57.030011] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=761445] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:05:57.030076] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:57.030131] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:57.030181] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:57.030224] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:05:57.030254] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:57.030266] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:57.151763] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:57.151957] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:57.302062] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:57.351833] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:57.494174] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:57.716260] WARN [USING_LOG_PREFIX] print_alloc_failed_msg (alloc_failed_reason.cpp:141) [981][T1002_L0_G0][T1002][YB42AC4D0007-00063C49634A0941-0-0] [lt=976666][errcode=-4013] [oops]: alloc failed reason is that server memory has reached the upper limit(server_hold: 4293697536, server_limit: 4294967296, alloc_size: 2097152) [384]OB_MALLOC COST TOO MUCH TIME, cost_time=960004, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=14337, WASH_BLOCK_END=945635, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=31], seq:[0, 1, 2, 3, 4] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=958983, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=12893, WASH_BLOCK_END=946090], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=964912, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=17639, WASH_BLOCK_END=947273], seq:[0, 1, 2] [552]OB_MALLOC COST TOO MUCH TIME, cost_time=965640, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=18154, WASH_BLOCK_END=947383, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=103], seq:[0, 1, 2, 3, 4] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=970315, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=18021, WASH_BLOCK_END=952294], seq:[0, 1, 2] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=969847, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=15661, WASH_BLOCK_END=954185], seq:[0, 1, 2] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=972269, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=17991, WASH_BLOCK_END=954278], seq:[0, 1, 2] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=971938, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=17597, WASH_BLOCK_END=954341], seq:[0, 1, 2] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=967673, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=13384, WASH_BLOCK_END=954289], seq:[0, 1, 2] [389]OB_MALLOC COST TOO MUCH TIME, cost_time=967175, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=12902, WASH_BLOCK_END=954273], seq:[0, 1, 2] [864]OB_MALLOC COST TOO MUCH TIME, cost_time=965475, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=11185, WASH_BLOCK_END=954290], seq:[0, 1, 2] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=965520, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=11161, WASH_BLOCK_END=954359], seq:[0, 1, 2] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=965342, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=10921, WASH_BLOCK_END=954421], seq:[0, 1, 2] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=965012, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=10525, WASH_BLOCK_END=954486], seq:[0, 1, 2] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=960300, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=5741, WASH_BLOCK_END=954559], seq:[0, 1, 2] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=960250, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=5653, WASH_BLOCK_END=954597], seq:[0, 1, 2] [2025-08-14 09:05:56.752463] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=12020] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:56.765099] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=243] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:57.726902] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=960350][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:56.778986] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:57.734168] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=953739][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:57.740254] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:57.742538] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:57.749566] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:57.749601] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=30] temporary file module use no blocks [2025-08-14 09:05:57.749608] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:57.749620] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:57.749636] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623259183204, flush_io_finished_ret_:0, flush_io_finished_round_:26380, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:57.749709] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:57.750272] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:57.750312] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=18] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:57.750453] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:57.748227] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [807][T1002_TimerWK3_TabletShell][T1002][Y0-0000000000000000-0-0] [lt=963719] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=2, times=1263) [2025-08-14 09:05:57.752946] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=4706][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:57.753122] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=472] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:57.757644] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=251] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:57.757697] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=436] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:57.757862] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=635] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:08.763330] EDIAG [RS.COMPACTION] get_epoch_with_retry (ob_major_merge_scheduler.cpp:743) [1061][T1002_MergeSche][T1002][YB42AC4D0007-00063C4962499CBA-0-0] [lt=2835][errcode=-4012] LOGGER COST TOO MUCH TIME, cost: 1012506, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=8, ALLOC_BLOCK#end=2, FORMAT_END=1012494, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x1126f45c 0x11254f92 0x1125a47b 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:08.921610] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [882][T1002_ArcSender][T1002][YB42AC4D0007-00063C495EC99C16-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2834551, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3c61 0x113d348b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:57.758571] WDIAG [SQL] do_close (ob_result_set.cpp:935) [712][T1001_GCCollect][T1][YB42AC4D0007-00063C495FA9A613-0-0] [lt=5581][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:57.758614] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [712][T1001_GCCollect][T1][YB42AC4D0007-00063C495FA9A613-0-0] [lt=29][errcode=-4012] result set close failed(ret=-4012) [2025-08-14 09:05:57.758627] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:199) [712][T1001_GCCollect][T1][YB42AC4D0007-00063C495FA9A613-0-0] [lt=8][errcode=-4012] result set close failed(ret=-4012) [2025-08-14 09:05:57.758642] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:951) [712][T1001_GCCollect][T1][YB42AC4D0007-00063C495FA9A613-0-0] [lt=14][errcode=-4012] failed to close result(close_ret=-4012, ret=-4012) [2025-08-14 09:05:57.759991] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:57.754330] WDIAG [SERVER] set_timeout (ob_inner_sql_connection.cpp:2049) [1026][TabletLSAUp1][T0][YB42AC4D0007-00063C496DC99CA8-0-0] [lt=9][errcode=-4012] already timeout(ret=-4012, ctx={abs_timeout_us:1755162351742322}, abs_timeout_us=1755162351742322) [2025-08-14 09:05:57.767741] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:57.778256] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:57.781409] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:57.781497] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:57.781892] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:57.807876] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=25] gc stale ls task succ [2025-08-14 09:05:57.809383] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:05:57.819542] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=965069] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:57.824660] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=24] gc stale ls task succ [2025-08-14 09:05:57.826789] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=0] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:05:57.826820] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:57.853403] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:57.934201] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:58.033119] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=26] PNIO [ratelimit] time: 1755162358033118, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007618, add_bytes: 0 [2025-08-14 09:05:58.033480] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:05:58.033557] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:05:58.033604] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:05:58.033643] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:05:58.094441] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:58.154555] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:58.308195] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:58.354649] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:58.652906] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:58.227664] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:58.755512] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=229] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:58.727204] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=967855] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:58.718339] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=964787] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:58.756684] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=38283] temporary file module use no blocks [2025-08-14 09:05:58.756695] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:58.756718] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=20] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:58.756731] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623260158039, flush_io_finished_ret_:0, flush_io_finished_round_:26400, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:58.756830] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [807][T1002_TimerWK3_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=86] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):258, sequence:1}) [2025-08-14 09:05:58.756868] WDIAG [PALF] runTimerTask (block_gc_timer_task.cpp:94) [807][T1002_TimerWK3_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:58.756898] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:97) [807][T1002_TimerWK3_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=30] BlockGCTimerTask success(ret=0, cost_time_us=993744, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.77.0.7:2882", log_dir:"/root/ob/observer/store/clog/tenant_1002", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):258, sequence:1}, log_alloc_mgr_:{flying_log_task:1, flying_meta_task:2}}) [2025-08-14 09:05:58.758073] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:58.727257] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=966140] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:05:58.736102] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=418] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:58.758432] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22294][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:58.758580] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=723791] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:58.758597] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:05:58.758729] WDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25840][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:55.742079] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:8607757, max_lsn:1604090331, max_scn:{val:1755161756730847001, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:1556737504}, reuse_lsn:{lsn:1604090331}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:1562147291}}, last_submit_lsn:{lsn:1604090203}, last_submit_end_lsn:{lsn:1604090331}, last_submit_log_id:8607757, last_submit_log_pid:50, max_flushed_lsn:{lsn:1604090203}, max_flushed_end_lsn:{lsn:1604090331}, max_flushed_log_pid:50, committed_end_lsn:{lsn:1604090090}, last_slide_log_id:8607755, last_slide_scn:{val:1755161749741193000, v:0}, last_slide_lsn:{lsn:1604089968}, last_slide_end_lsn:{lsn:1604090090}, last_slide_log_pid:50, last_slide_log_accum_checksum:1446576759, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f990bfe4530}, [StateMgr]={this:0x7f990bfe89b0, self:"172.77.0.7:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"PENDING", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:51}, leader:"0.0.0.0:0", leader_epoch:-1, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:1604090331}, scan_disk_log_finished:true, last_check_start_id:8607756, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161901828039, palf_role_change_cb:0x7f990bfef470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:50, prev:{version:1, config:{config_version:{proposal_id:49, config_seq:45}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:50, config_seq:46}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_mode_pid:50}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_end_lsn:{lsn:1604088821}, prev_mode_pid:50}, reconfig_barrier:{prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_end_lsn:{lsn:1604088821}, prev_mode_pid:50}, persistent_config_version:{proposal_id:50, config_seq:46}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755161738854289, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755161738854289, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f990bfe5c70}, [ModeMgr]={palf_id:1, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:50, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:1543409664}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:51}, log_config_meta:{version:3, proposal_id:50, prev:{version:1, config:{config_version:{proposal_id:49, config_seq:45}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:50, config_seq:46}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_mode_pid:50}, log_snapshot_meta:{version:2, base_lsn:{lsn:1543409664}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:712704}, readable_log_tail:{lsn:712704}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:1610514432}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/meta", dir_fd:225, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66392064, block_header_serialize_buf_:0x7f990bfeabb8, flashback_version:0}, log_storage:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1755161901828095, this:0x7f990bfe90f0}, [Reconfirm]={palf_id:1, self:"172.77.0.7:2882", state:"RECONFIRM_MODE_META", new_proposal_id:51, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"172.77.0.7:2882", majority_max_accept_pid:50, majority_max_lsn:{lsn:1604090331}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:1755161901828373, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f990bfe8b30}) [2025-08-14 09:05:58.720335] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=190] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:58.759012] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=38614] temporary file module use no blocks [2025-08-14 09:05:58.759022] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:58.154533] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:11.728322] EDIAG [SHARE] refresh (ob_task_define.cpp:396) [280][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=29913][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1022731, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1022724, ObMalloc#start=2, ObMalloc#end=4, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1fd6cd17 0x96c37e0 0x1fda2168 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:58.759385] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=73] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:58.752988] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:981) [712][T1001_GCCollect][T1][YB42AC4D0007-00063C495FA9A613-0-0] [lt=15][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1001"}, record_ret=-4012, ret=-4012) [2025-08-14 09:05:58.768309] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:05:58.759034] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:58.769367] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10315] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623260160038, flush_io_finished_ret_:0, flush_io_finished_round_:26420, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [386]OB_MALLOC COST TOO MUCH TIME, cost_time=951023, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=951017], seq:[0] [386]OB_MALLOC COST TOO MUCH TIME, cost_time=951047, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=951042], seq:[0, 1] [2025-08-14 09:05:58.721267] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2924] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:59.722006] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=999225][errcode=0] epoll handler too much time :1000751 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:05:59.722030] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:55.742079] EDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3976401, owner: ObLog, click_count: 7, time dist:[FORMAT_END=2988038, ObMalloc#start=1, ObMalloc#end=24089, ObMalloc#start=22, ObMalloc#end=1, ALLOC_END=8, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0xfc0e73d 0x993f560 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:59.724535] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=3976401] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161974728707, last_flushed_end_lsn:{lsn:1604090331}}]) [2025-08-14 09:05:58.959212] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:05:58.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:05:59.255952] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:05:59.743463] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=486760][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:05:59.743499] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:05:58.820209] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=20] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:58.869446] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [280][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=1022731][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:15180, tid:374}, {errcode:-4016, dropped:4323, tid:372}, {errcode:-4076, dropped:333, tid:856}, {errcode:-4389, dropped:433, tid:770}]) [2025-08-14 09:05:59.743753] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=20] log compressor cycles once. (ret=0, cost_time=994382, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25878753280, fast_delete_log_mode=false) [2025-08-14 09:05:59.745281] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=149] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:59.746270] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][YB42AC4D0007-00063C495F8A1970-0-0] [lt=1][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a7918, timer:0x7f99329e19d0, task:0x7f98f55f1250, task_type:"N9oceanbase5share24ObVectorIndexHistoryTaskE", scheduled_time:1755162339753898, pushed_time:1755162339753989, delay:10000000}, delay_in_thread_pool=19002933, threashold=500000) [2025-08-14 09:05:58.995640] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:05:58.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:05:59.747032] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=750253][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:05:58.995|207=751.39ms|210=0us|222=2us|total=751.41ms) [2025-08-14 09:05:59.747057] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4389] timer task use too much time(end=1755162359747057, start=1755162358995638, delta=751419) [2025-08-14 09:05:59.747079] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=502444, expected_run_ts_us_=1755162359244633, ObClockGenerator::getRealClock()=1755162359747078, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:05:59.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:05:59.747100] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:05:59.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:05:59.725576] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=765201][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:05:58.959|207=766.36ms|210=0us|222=2us|total=781.04ms) [2025-08-14 09:05:59.747159] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=21549][errcode=-4389] timer task use too much time(end=1755162359747158, start=1755162358959211, delta=787947) [2025-08-14 09:05:59.747602] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=347][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aae10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:05:59.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:05:59.747635] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:05:59.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:05:59.747656] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e67b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:05:59.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:05:59.747668] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:05:59.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:05:59.749465] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=207] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:59.750859] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=221] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:05:59.752366] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=30] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:59.752391] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=18] temporary file module use no blocks [2025-08-14 09:05:59.752401] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:59.752413] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:59.752427] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623261189505, flush_io_finished_ret_:0, flush_io_finished_round_:26382, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:59.754257] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=171] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:59.755251] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1083] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:59.756491] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=284][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:05:59.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:05:59.756729] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=291][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:05:59.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:05:59.756851] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=29405] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:05:59.757268] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=164] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:59.757283] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:59.757700] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919bfe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aa810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:05:59.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:05:59.757725] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:05:59.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:05:59.757738] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:05:59.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:05:59.757750] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:05:59.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:05:59.757789] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:59.758343] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=203] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:59.759404] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:59.760260] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=31004] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:05:59.761617] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8681] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:59.757295] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623261196525, flush_io_finished_ret_:0, flush_io_finished_round_:26401, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:59.762955] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=318][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:05:59.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:05:59.767916] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=4950][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:05:59.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:05:59.757769] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:05:59.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:05:59.768592] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10803][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:05:59.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:05:59.768688] WDIAG [STORAGE.TRANS] get_write_store_ctx (ob_trans_service_v4.cpp:1140) [337][EvtHisUpdTask][T1][YB42AC4D0007-00063C4960299C79-0-0] [lt=6911][errcode=-4038] acquire ls snapshot for mvcc write fail(ret=-4038, ls_id={id:1}) [2025-08-14 09:05:59.768736] WDIAG [STORAGE.TRANS] get_write_store_ctx (ob_trans_service_v4.cpp:1140) [338][Occam][T1][YB42AC4D0007-00063C4960499C96-0-0] [lt=1][errcode=-4038] acquire ls snapshot for mvcc write fail(ret=-4038, ls_id={id:1}) [2025-08-14 09:05:59.770049] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=626] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:05:59.770079] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26] temporary file module use no blocks [2025-08-14 09:05:59.770091] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:05:59.770107] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:05:59.770122] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623261209763, flush_io_finished_ret_:0, flush_io_finished_round_:26421, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:05:59.775518] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=350][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:05:59.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:05:59.779444] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=226] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:59.780608] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=195] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:59.781623] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=187] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:59.782047] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=215] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:59.788715] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72707b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a4010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:05:59.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:05:59.798778] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:05:59.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:05:59.806422] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:05:59.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:05:59.808039] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=294] gc stale ls task succ [2025-08-14 09:05:59.808840] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739acb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196bb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:05:59.798, task_priority:1, with_handle_protected:0}) [2025-08-14 09:05:59.812023] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=214] gc stale ls task succ [2025-08-14 09:05:59.818906] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:05:59.806, task_priority:1, with_handle_protected:0}) [2025-08-14 09:05:59.818935] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:05:59.809, task_priority:1, with_handle_protected:0}) [2025-08-14 09:05:59.824784] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=230] gc stale ls task succ [2025-08-14 09:05:59.828973] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:05:59.815, task_priority:1, with_handle_protected:0}) [2025-08-14 09:05:59.831641] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=634][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:05:59.843666] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=921945] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:05:59.847278] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=52][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:05:59.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:05:59.849099] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:05:59.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:05:59.849127] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:05:59.832, task_priority:1, with_handle_protected:0}) [2025-08-14 09:05:59.856553] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:05:59.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:05:59.859175] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:05:59.841, task_priority:1, with_handle_protected:0}) [2025-08-14 09:05:59.869216] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:05:59.859, task_priority:1, with_handle_protected:0}) [2025-08-14 09:05:59.931701] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:05:59.947311] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:05:59.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:05:59.956617] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:05:59.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:05:59.957418] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:05:59.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:05:59.991520] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:05:59.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:00.014567] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:00.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:00.014602] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:00.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:00.015763] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=779513][errcode=0] EASY SLOW: thread: 7f995427d300, lastrun: 1755162250.789339 cost: 109.226421 loop:26617, slowcnt: 819 [2025-08-14 09:06:00.028269] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:00.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:00.031759] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:00.047346] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:05:59.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:00.048351] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=250] PNIO [ratelimit] time: 1755162360048349, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007617, add_bytes: 0 [2025-08-14 09:06:00.056679] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:05:59.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:00.094991] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:00.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:00.131832] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:00.147415] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:00.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:00.156743] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:00.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:00.157538] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:00.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:00.215606] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:00.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:00.231864] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:00.242873] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:00.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:00.247477] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:00.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:00.252928] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:00.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:00.256805] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:00.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:00.265876] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:00.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:00.265922] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:00.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:00.279603] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:00.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:00.309762] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=50][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:00.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:00.331883] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:00.347539] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:00.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:00.356867] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:00.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:00.357653] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:00.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:00.431946] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:00.456931] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:00.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:00.494287] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:00.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:00.517211] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:00.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:00.517263] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:00.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:00.520870] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:00.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:00.532004] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:00.556993] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:00.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:00.557769] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:00.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:00.597645] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:00.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:00.632076] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:00.657054] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:00.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:00.718685] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:00.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:00.720825] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:00.724328] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73516b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196f510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:00.716, task_priority:1, with_handle_protected:0}) [2025-08-14 09:05:58.755918] EDIAG [DETECT] maintain_deadlock_info_when_end_stmt (ob_trans_deadlock_adapter.cpp:649) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=959639][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1973122, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1973111, ObMalloc#start=1, ObMalloc#end=5, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9826e08 0x98267df 0x982650b 0x98262e0 0x1b6cf0d1 0x9460903 0x93d78df 0x9394489 0x9725f91 0x124dfd77 0x96a5a4d 0x97aa16f 0x124eac91 0x97a9944 0x261395f9 0xfa0e430 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:58.755918] INFO [DETECT] maintain_deadlock_info_when_end_stmt (ob_trans_deadlock_adapter.cpp:649) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=959639] maintain deadlock info(step=2, ret=0, ret="OB_SUCCESS", exec_ctx.get_errcode()=-4038, exec_ctx.get_errcode()="OB_NOT_MASTER", session={this:0x7f985b41a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f98c907e020}, desc=NULL, is_rollback=true, conflict_txs=[]) [2025-08-14 09:06:00.733271] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=326] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:00.734394] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:00.720, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:00.734434] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:00.727, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:00.734468] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:00.727, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:00.734486] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72851b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:00.721, task_priority:1, with_handle_protected:0}) [658]OB_MALLOC COST TOO MUCH TIME, cost_time=976183, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=25, ALLOC_CHUNK_END=976119], seq:[0, 1, 2, 3, 4, 5, 6] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=976015, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=19, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=975974, ALLOC_CHUNK_END=3], seq:[0, 1, 2, 3, 4, 5] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=976048, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=976024, WASH_BLOCK_END=24], seq:[0, 1, 2] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=981880, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=6479, ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=975363, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=981942, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=981914, WASH_BLOCK_END=27], seq:[0, 1, 2] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=975328, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=975290, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=975348, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=975339, WASH_BLOCK_END=8], seq:[0, 1, 2] [733]OB_MALLOC COST TOO MUCH TIME, cost_time=974138, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=974098, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [733]OB_MALLOC COST TOO MUCH TIME, cost_time=974168, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=974157, WASH_BLOCK_END=11], seq:[0, 1, 2] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=974039, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=974001, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=974059, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=974051, WASH_BLOCK_END=8], seq:[0, 1, 2] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=974152, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=974120, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=974180, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=974163, WASH_BLOCK_END=17], seq:[0, 1, 2] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=972353, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=972318, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=972371, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=972362, WASH_BLOCK_END=9], seq:[0, 1, 2] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=970016, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=148, ALLOC_CHUNK_END=969828, ALLOC_CHUNK_END=12], seq:[0, 1, 2, 3, 4, 5, 6, 7] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=970037, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=970026, WASH_BLOCK_END=10], seq:[0, 1, 2] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=969877, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=969840, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=969898, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=969888, WASH_BLOCK_END=10], seq:[0, 1, 2] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=968675, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=968634, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [2025-08-14 09:05:59.743514] EDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] LOGGER COST TOO MUCH TIME, cost: 1975903, owner: ObLog, click_count: 8, time dist:[FORMAT_END=998808, ObMalloc#start=1, ObMalloc#end=976057, ObMalloc#start=31, ObMalloc#end=38, ObMalloc#start=326, ObMalloc#end=39, FREE_END=603], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x13b06ffb 0x958a39e 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [474]OB_MALLOC COST TOO MUCH TIME, cost_time=968701, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=968691, WASH_BLOCK_END=10], seq:[0, 1, 2] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=966692, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=966651, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=966711, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=966703, WASH_BLOCK_END=8], seq:[0, 1, 2] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=962607, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=962574, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=962636, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=962619, WASH_BLOCK_END=17], seq:[0, 1, 2] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=964194, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=964160, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=964216, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=964205, WASH_BLOCK_END=11], seq:[0, 1, 2] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=961651, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=961617, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=962920, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=962888, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=961672, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=961662, WASH_BLOCK_END=9], seq:[0, 1, 2] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=962944, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=962931, WASH_BLOCK_END=13], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=961700, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=961668, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=961721, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=961709, WASH_BLOCK_END=12], seq:[0, 1, 2] [717]OB_MALLOC COST TOO MUCH TIME, cost_time=951113, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=951076, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=958085, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=958044, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [717]OB_MALLOC COST TOO MUCH TIME, cost_time=951135, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=951126, WASH_BLOCK_END=9], seq:[0, 1, 2] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=958117, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=958106, WASH_BLOCK_END=11], seq:[0, 1, 2] [861]OB_MALLOC COST TOO MUCH TIME, cost_time=947225, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=947194, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=946240, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=946209, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=959706, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=959666, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=946262, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=946251, WASH_BLOCK_END=10], seq:[0, 1, 2] [861]OB_MALLOC COST TOO MUCH TIME, cost_time=947249, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=947240, WASH_BLOCK_END=9], seq:[0, 1, 2] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=959877, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=959865, WASH_BLOCK_END=12], seq:[0, 1, 2] [383]OB_MALLOC COST TOO MUCH TIME, cost_time=945632, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=945603, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [383]OB_MALLOC COST TOO MUCH TIME, cost_time=981266, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=42, WASH_BLOCK_END=35582, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=945642], seq:[0, 1, 2, 3, 4] [322]OB_MALLOC COST TOO MUCH TIME, cost_time=945395, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=945362, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [322]OB_MALLOC COST TOO MUCH TIME, cost_time=945415, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=945406, WASH_BLOCK_END=9], seq:[0, 1, 2] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=940133, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=940103, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=940157, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=940147, WASH_BLOCK_END=10], seq:[0, 1, 2] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=939502, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=939474, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=939519, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=939511, WASH_BLOCK_END=8], seq:[0, 1, 2] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=940207, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=940181, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=940227, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=940218, WASH_BLOCK_END=9], seq:[0, 1, 2] [411]OB_MALLOC COST TOO MUCH TIME, cost_time=937412, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=937383, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [411]OB_MALLOC COST TOO MUCH TIME, cost_time=937430, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=937422, WASH_BLOCK_END=8], seq:[0, 1, 2] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=909893, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=909860, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=909912, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=909903, WASH_BLOCK_END=9], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=877193, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=877154, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=877209, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=877202, WASH_BLOCK_END=7], seq:[0, 1, 2] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=897228, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=897180, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=897256, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=897237, WASH_BLOCK_END=18], seq:[0, 1, 2] [389]OB_MALLOC COST TOO MUCH TIME, cost_time=963844, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=963810, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [389]OB_MALLOC COST TOO MUCH TIME, cost_time=963865, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=963858, WASH_BLOCK_END=7], seq:[0, 1, 2] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=706209, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=706167, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=706228, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=706218, WASH_BLOCK_END=10], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=960979, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=960922, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=961000, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=960989, WASH_BLOCK_END=11], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=965017, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=964972, ALLOC_CHUNK_END=12], seq:[0, 1, 2, 3, 4, 5, 6, 7] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=965045, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=965027, WASH_BLOCK_END=17], seq:[0, 1, 2] [864]OB_MALLOC COST TOO MUCH TIME, cost_time=961882, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=961841, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [864]OB_MALLOC COST TOO MUCH TIME, cost_time=961901, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=961892, WASH_BLOCK_END=9], seq:[0, 1, 2] [2025-08-14 09:05:59.823302] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=872331][errcode=-4018] LOGGER COST TOO MUCH TIME, cost: 1898225, owner: ObLog, click_count: 8, time dist:[FORMAT_END=949537, ObMalloc#start=1, ObMalloc#end=947272, ObMalloc#start=18, ObMalloc#end=36, ObMalloc#start=1095, ObMalloc#end=30, FREE_END=236], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3b4b 0x113d34aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:59.823101] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=878013][errcode=-4018] LOGGER COST TOO MUCH TIME, cost: 1900382, owner: ObLog, click_count: 8, time dist:[FORMAT_END=921551, ObMalloc#start=1, ObMalloc#end=974175, ObMalloc#start=19, ObMalloc#end=48, ObMalloc#start=4116, ObMalloc#end=56, FREE_END=416], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3b4b 0x113d34aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [660]OB_MALLOC COST TOO MUCH TIME, cost_time=952651, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=952612, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=952675, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=952667, WASH_BLOCK_END=8], seq:[0, 1, 2] [387]OB_MALLOC COST TOO MUCH TIME, cost_time=951631, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=951598, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [387]OB_MALLOC COST TOO MUCH TIME, cost_time=984494, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=39, WASH_BLOCK_END=32805, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=951650], seq:[0, 1, 2, 3, 4] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=951166, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=951139, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=951182, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=951175, WASH_BLOCK_END=7], seq:[0, 1, 2] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=944556, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=944528, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=944579, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=944566, WASH_BLOCK_END=13], seq:[0, 1, 2] [2025-08-14 09:06:00.740065] EDIAG [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [726][T1001_ObXAHbWor][T1001][Y0-0000000000000000-0-0] [lt=15232][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1002103, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1001921, ObMalloc#start=1, ObMalloc#end=50, ObMalloc#start=32, ObMalloc#end=47, FREE_END=52], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b64d8ba 0x1b64d154 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:59.808401] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=21432][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1941392, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1941080, ObMalloc#start=1, ObMalloc#end=39, ObMalloc#start=19, ObMalloc#end=38, FREE_END=215], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:00.746086] EDIAG [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:233) [827][T1002_RFLWorker][T1002][YB42AC4D0007-00063C495D099C16-0-0] [lt=10444][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1005155, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1004067, ObMalloc#start=1, ObMalloc#end=38, ObMalloc#start=23, ObMalloc#end=38, FREE_END=988], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x11359281 0x98b739c 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:00.743480] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:288) [732][T1001_ArcFetche][T1001][YB42AC4D0007-00063C495CA99C16-0-0] [lt=26][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1007800, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1006553, ObMalloc#start=1, ObMalloc#end=43, ObMalloc#start=22, ObMalloc#end=38, FREE_END=1143], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113b2889 0x979b621 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [658]OB_MALLOC COST TOO MUCH TIME, cost_time=1029175, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=976209], seq:[0, 1] [2025-08-14 09:06:00.447601] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:00.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:05:59.810103] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=21675][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1963844, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1962971, ObMalloc#start=1, ObMalloc#end=56, ObMalloc#start=30, ObMalloc#end=41, FREE_END=745], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:59.813130] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=21481][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1936733, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1936563, ObMalloc#start=1, ObMalloc#end=41, ObMalloc#start=18, ObMalloc#end=31, FREE_END=79], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [736]OB_MALLOC COST TOO MUCH TIME, cost_time=1962868, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1962865], seq:[0] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=1998227, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=39, WASH_BLOCK_END=35294, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1962888], seq:[0, 1, 2, 3, 4] [2025-08-14 09:06:00.738799] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=188][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1998262, owner: ObLog, click_count: 5, time dist:[FORMAT_END=15, ObMalloc#start=1, ObMalloc#end=1998238, ALLOC_END=1, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:00.061032] EDIAG [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=179][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1660666, owner: ObLog, click_count: 6, time dist:[FORMAT_END=681310, ObMalloc#start=1, ObMalloc#end=31807, ObMalloc#start=44, ObMalloc#end=946272, FREE_END=1232], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1f9d7fcf 0x99b8be2 0x1f981833 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [858]OB_MALLOC COST TOO MUCH TIME, cost_time=931230, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=41, WASH_BLOCK_END=931149, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=39], seq:[0, 1, 2, 3, 4] [2025-08-14 09:06:00.738799] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=188] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:02.743254] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2163] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:00.808102] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=22][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1933865, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1933523, ObMalloc#start=1, ObMalloc#end=73, ObMalloc#start=25, ObMalloc#end=51, FREE_END=192], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:02.744897] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=199] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:02.744977] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=36] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:02.745025] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:02.745048] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:02.745061] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [770]OB_MALLOC COST TOO MUCH TIME, cost_time=943894, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=42, WASH_BLOCK_END=943852], seq:[0, 1, 2] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=996891, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=33, WASH_BLOCK_END=996858], seq:[0, 1, 2] [466]OB_MALLOC COST TOO MUCH TIME, cost_time=932495, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=50, WASH_BLOCK_END=932445], seq:[0, 1, 2] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=997066, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=30, WASH_BLOCK_END=997036], seq:[0, 1, 2] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=997130, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=44, WASH_BLOCK_END=997085], seq:[0, 1, 2] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=996980, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=39, WASH_BLOCK_END=996941], seq:[0, 1, 2] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=996780, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=996740], seq:[0, 1, 2] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=953204, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=52, WASH_BLOCK_END=953152], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=950363, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=49, WASH_BLOCK_END=950314], seq:[0, 1, 2] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=936595, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=41, WASH_BLOCK_END=936554], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=852322, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=58, WASH_BLOCK_END=852264], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=763220, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=61, WASH_BLOCK_END=763159], seq:[0, 1, 2] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=1022485, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=36495], seq:[0, 1] [2025-08-14 09:06:00.736255] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=192][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3023426, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=4, FORMAT_END=2000880, ObMalloc#start=0, ObMalloc#end=1022530, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:00.736255] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=192] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:03.764284] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3023426][errcode=0] epoll handler too much time :3028037 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:03.764314] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [574]OB_MALLOC COST TOO MUCH TIME, cost_time=995015, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=995008], seq:[0] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=1041738, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=33, WASH_BLOCK_END=46670, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=995030], seq:[0, 1, 2, 3, 4] [2025-08-14 09:06:02.728947] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=973512][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1041763, owner: ObLog, click_count: 5, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=1041747, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:02.728947] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=973512] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:03.756060] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [874][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E899C16-0-0] [lt=2846978] ObArchiveSender is running(thread_index=2) [2025-08-14 09:06:00.732139] EDIAG [RS] get_valid_sts_after (ob_tenant_info_loader.cpp:454) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3036020, owner: ObLog, click_count: 5, time dist:[FORMAT_END=3036013, ObMalloc#start=0, ObMalloc#end=2, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1001d942 0x996d843 0x996d15a 0x996cfd8 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:02.762089] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2446][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1009521, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:02.762089] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2446] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:03.773159] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1009521][errcode=0] epoll handler too much time :1011081 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:03.773178] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:02.026332] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=708836][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1746526, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1746516, ObMalloc#start=0, ObMalloc#end=3, ALLOC_END=1, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [770]OB_MALLOC COST TOO MUCH TIME, cost_time=996052, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=996013, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=996096, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=996063, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=996125, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=996109, WASH_BLOCK_END=16], seq:[0, 1, 2] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=995497, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=995461, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=995520, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=995507, WASH_BLOCK_END=13], seq:[0, 1, 2] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=995519, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=32, ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=995449, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=1991555, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=34, WASH_BLOCK_END=995991, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=995528], seq:[0, 1, 2, 3, 4] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=995536, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=19, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=995487, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=995557, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=995548, WASH_BLOCK_END=9], seq:[0, 1, 2] [466]OB_MALLOC COST TOO MUCH TIME, cost_time=995528, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=995494, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [466]OB_MALLOC COST TOO MUCH TIME, cost_time=995553, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=995541, WASH_BLOCK_END=12], seq:[0, 1, 2] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=995528, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=995495, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=995547, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=995538, WASH_BLOCK_END=9], seq:[0, 1, 2] [386]OB_MALLOC COST TOO MUCH TIME, cost_time=995518, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=995485, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [386]OB_MALLOC COST TOO MUCH TIME, cost_time=995538, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=995529, WASH_BLOCK_END=9], seq:[0, 1, 2] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=995507, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=995465, ALLOC_CHUNK_END=12], seq:[0, 1, 2, 3, 4, 5, 6, 7] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=995526, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=995516, WASH_BLOCK_END=10], seq:[0, 1, 2] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=995469, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=995436, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=995501, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=995492, WASH_BLOCK_END=9], seq:[0, 1, 2] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=995456, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=995423, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=995476, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=995466, WASH_BLOCK_END=9], seq:[0, 1, 2] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=995469, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=995429, ALLOC_CHUNK_END=11], seq:[0, 1, 2, 3, 4, 5, 6, 7] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=995488, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=995479, WASH_BLOCK_END=9], seq:[0, 1, 2] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=995468, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=995431, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=995489, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=995480, WASH_BLOCK_END=9], seq:[0, 1, 2] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=995447, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=995405, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=995465, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=995457, WASH_BLOCK_END=8], seq:[0, 1, 2] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=995418, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=995380, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=995434, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=995426, WASH_BLOCK_END=8], seq:[0, 1, 2] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=995402, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=995360, ALLOC_CHUNK_END=11], seq:[0, 1, 2, 3, 4, 5, 6, 7] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=995419, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=995411, WASH_BLOCK_END=8], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=995378, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=995343, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=995396, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=995388, WASH_BLOCK_END=8], seq:[0, 1, 2] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=995357, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=995315, ALLOC_CHUNK_END=12], seq:[0, 1, 2, 3, 4, 5, 6, 7] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=995376, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=995367, WASH_BLOCK_END=9], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=995338, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=995304, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=995355, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=995347, WASH_BLOCK_END=8], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=995322, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=15, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=995259, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=995342, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=995334, WASH_BLOCK_END=8], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=995291, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=995255, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=995308, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=995300, WASH_BLOCK_END=8], seq:[0, 1, 2] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=995279, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=995239, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=995296, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=995288, WASH_BLOCK_END=8], seq:[0, 1, 2] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=995267, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=995231, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=995284, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=995276, WASH_BLOCK_END=8], seq:[0, 1, 2] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=995263, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=995213, ALLOC_CHUNK_END=18], seq:[0, 1, 2, 3, 4, 5, 6, 7] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=995280, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=995272, WASH_BLOCK_END=8], seq:[0, 1, 2] [552]OB_MALLOC COST TOO MUCH TIME, cost_time=995236, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=995198, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [552]OB_MALLOC COST TOO MUCH TIME, cost_time=995254, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=995246, WASH_BLOCK_END=8], seq:[0, 1, 2] [2025-08-14 09:06:02.026332] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=708836][errcode=0] EASY SLOW: thread: 7f9954779300, lastrun: 1755162361.780483 cost: 0.245847 loop:26712, slowcnt: 2 [393]OB_MALLOC COST TOO MUCH TIME, cost_time=992663, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=992629, ALLOC_CHUNK_END=3], seq:[0, 1, 2, 3, 4, 5, 6, 7] [2025-08-14 09:06:02.733406] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=209][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1042056, owner: ObLog, click_count: 6, time dist:[FORMAT_END=11, ObMalloc#start=2, ObMalloc#end=46601, ObMalloc#start=21, ObMalloc#end=995290, FREE_END=131], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [393]OB_MALLOC COST TOO MUCH TIME, cost_time=992736, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=992701, WASH_BLOCK_END=35], seq:[0, 1, 2] [2025-08-14 09:06:01.824896] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=899114][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1949258, owner: ObLog, click_count: 6, time dist:[FORMAT_END=11, ObMalloc#start=0, ObMalloc#end=953213, ObMalloc#start=19, ObMalloc#end=995440, FREE_END=575], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [389]OB_MALLOC COST TOO MUCH TIME, cost_time=992705, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=992660, ALLOC_CHUNK_END=3], seq:[0, 1, 2, 3, 4, 5, 6, 7] [389]OB_MALLOC COST TOO MUCH TIME, cost_time=992731, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=992715, WASH_BLOCK_END=16], seq:[0, 1, 2] [2025-08-14 09:06:02.727134] EDIAG eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=298][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1048503, owner: ObLog, click_count: 6, time dist:[FORMAT_END=14, ObMalloc#start=1, ObMalloc#end=52745, ObMalloc#start=19, ObMalloc#end=995314, FREE_END=410], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x27a8a8c8 0x97b7e1a 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [391]OB_MALLOC COST TOO MUCH TIME, cost_time=992808, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=992770, ALLOC_CHUNK_END=3], seq:[0, 1, 2, 3, 4, 5, 6, 7] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=992829, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=992819, WASH_BLOCK_END=10], seq:[0, 1, 2] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=992322, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=992282, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=992352, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=992344, WASH_BLOCK_END=8], seq:[0, 1, 2] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=992317, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=992282, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=992335, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=992327, WASH_BLOCK_END=8], seq:[0, 1, 2] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=963806, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=963768, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=963825, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=963815, WASH_BLOCK_END=10], seq:[0, 1, 2] [2025-08-14 09:06:02.757258] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1191][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1018734, owner: ObLog, click_count: 6, time dist:[FORMAT_END=7, ObMalloc#start=0, ObMalloc#end=25587, ObMalloc#start=19, ObMalloc#end=992737, FREE_END=384], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:01.926192] EDIAG try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=879815][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1848617, owner: ObLog, click_count: 6, time dist:[FORMAT_END=18, ObMalloc#start=1, ObMalloc#end=852331, ObMalloc#start=18, ObMalloc#end=995361, FREE_END=888], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x26499505 0x97074a3 0x26498fa9 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:02.757753] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1640][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1018397, owner: ObLog, click_count: 6, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=25159, ObMalloc#start=19, ObMalloc#end=992834, FREE_END=373], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:01.827851] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=473][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1946946, owner: ObLog, click_count: 7, time dist:[FORMAT_END=32, ObMalloc#start=1, ObMalloc#end=950372, ObMalloc#start=19, ObMalloc#end=995402, ALLOC_END=1117, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:02.756532] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1036][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1019738, owner: ObLog, click_count: 6, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=26242, ObMalloc#start=28, ObMalloc#end=992745, FREE_END=713], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:01.772378] EDIAG [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=306][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2002423, owner: ObLog, click_count: 6, time dist:[FORMAT_END=960704, ObMalloc#start=1, ObMalloc#end=45608, ObMalloc#start=30, ObMalloc#end=995262, FREE_END=818], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1d17bb52 0x1d17ada0 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:02.720797] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=854][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1055468, owner: ObLog, click_count: 6, time dist:[FORMAT_END=16, ObMalloc#start=2, ObMalloc#end=58183, ObMalloc#start=21, ObMalloc#end=995544, FREE_END=1702], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:01.781107] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1220][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1993855, owner: ObLog, click_count: 6, time dist:[FORMAT_END=7, ObMalloc#start=1, ObMalloc#end=996790, ObMalloc#start=20, ObMalloc#end=995495, FREE_END=1542], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [416]OB_MALLOC COST TOO MUCH TIME, cost_time=931627, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=931592, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=931647, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=931639, WASH_BLOCK_END=8], seq:[0, 1, 2] [2025-08-14 09:06:01.759032] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6219][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2016390, owner: ObLog, click_count: 10, time dist:[ALLOC_BLOCK#start=9, ALLOC_BLOCK#end=89, ALLOC_BLOCK#start=11113, ALLOC_BLOCK#end=57, FORMAT_END=1998311, ObMalloc#start=0, ObMalloc#end=6588, ObMalloc#start=20, ObMalloc#end=44, FREE_END=159], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:01.780440] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=740][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1995025, owner: ObLog, click_count: 6, time dist:[FORMAT_END=7, ObMalloc#start=0, ObMalloc#end=997074, ObMalloc#start=20, ObMalloc#end=995554, FREE_END=2369], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:02.745269] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=346][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1031710, owner: ObLog, click_count: 8, time dist:[ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=2, FORMAT_END=30337, ObMalloc#start=1, ObMalloc#end=7928, ObMalloc#start=20, ObMalloc#end=992345, FREE_END=1074], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:01.752833] EDIAG [RS.COMPACTION] try_idle (ob_major_merge_scheduler.cpp:226) [1275][T1001_MergeSche][T1001][YB42AC4D0007-00063C4963E99CBE-0-0] [lt=31][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2022692, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1989174, ObMalloc#start=0, ObMalloc#end=32998, ObMalloc#start=18, ObMalloc#end=30, FREE_END=472], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1125d502 0x1125cf81 0x1125a4c3 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:02.755995] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3610][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1021138, owner: ObLog, click_count: 6, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=23583, ObMalloc#start=20, ObMalloc#end=995425, FREE_END=2098], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:02.750819] EDIAG [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=592][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1026422, owner: ObLog, click_count: 6, time dist:[FORMAT_END=10, ObMalloc#start=0, ObMalloc#end=28604, ObMalloc#start=29, ObMalloc#end=995472, FREE_END=2307], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a1f867 0x27a135c3 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:02.729014] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [477][T1_StandbySchem][T1][YB42AC4D0007-00063C495B6A03C8-0-0] [lt=986885][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1048305, owner: ObLog, click_count: 6, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=50931, ObMalloc#start=21, ObMalloc#end=995304, FREE_END=2039], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:01.780494] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1467][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1995433, owner: ObLog, click_count: 6, time dist:[FORMAT_END=10, ObMalloc#start=0, ObMalloc#end=997139, ObMalloc#start=19, ObMalloc#end=995532, FREE_END=2733], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:03.777062] WDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1042056][errcode=0] epoll handler too much time :1043668 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:03.777085] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:01.827851] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=473][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:01.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:03.777769] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1946946][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:01.827|207=1.95s|210=0us|222=3us|total=1.95s) [2025-08-14 09:06:03.777801] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=-4389] timer task use too much time(end=1755162363777800, start=1755162361827849, delta=1949951) [2025-08-14 09:06:02.845266] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:03.778700] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1021138][errcode=0] epoll handler too much time :1022716 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:03.778776] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=67] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:02.723274] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [776][T1002_StandbySc][T1002][YB42AC4D0007-00063C495DFA03BE-0-0] [lt=963024][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1054355, owner: ObLog, click_count: 6, time dist:[FORMAT_END=19, ObMalloc#start=2, ObMalloc#end=55830, ObMalloc#start=45, ObMalloc#end=995508, FREE_END=2951], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:02.725248] EDIAG do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=965670][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1052622, owner: ObLog, click_count: 6, time dist:[FORMAT_END=15, ObMalloc#start=1, ObMalloc#end=53537, ObMalloc#start=20, ObMalloc#end=995570, FREE_END=3479], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x202c530b 0x202c6888 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:01.767507] EDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=366][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 2009044, owner: ObLog, click_count: 8, time dist:[FORMAT_END=1984308, ObMalloc#start=0, ObMalloc#end=24415, ObMalloc#start=20, ObMalloc#end=50, ObMalloc#start=91, ObMalloc#end=36, FREE_END=124], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x9757ced 0x97569df 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:01.780789] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8278][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1995794, owner: ObLog, click_count: 8, time dist:[FORMAT_END=21, ObMalloc#start=1, ObMalloc#end=996989, ObMalloc#start=18, ObMalloc#end=995482, ObMalloc#start=1613, ObMalloc#end=38, FREE_END=1632], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:02.015482] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=3392][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1762073, owner: ObLog, click_count: 8, time dist:[FORMAT_END=28, ObMalloc#start=1, ObMalloc#end=763229, ObMalloc#start=19, ObMalloc#end=995349, ObMalloc#start=2323, ObMalloc#end=50, FREE_END=1074], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:01.729542] EDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=794][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 2047846, , BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1b53d414 0x97ad9aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:02.740310] EDIAG [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=323][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1041626, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1012906, ObMalloc#start=1, ObMalloc#end=28609, ObMalloc#start=18, ObMalloc#end=37, FREE_END=55], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x18bdb084 0x18bda6b1 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:01.780281] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=327][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2953639, owner: ObLog, click_count: 7, time dist:[FORMAT_END=22, ObMalloc#start=1, ObMalloc#end=996908, ObMalloc#start=22, ObMalloc#end=995526, ALLOC_END=961156, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:01.841708] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=367][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fc8b0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcb90, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:1, time_interval:3.00s, expected_run_time:09:06:01.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:01.780281] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=327][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:01.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:04.738432] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=2953639][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:01.780|207=2.96s|210=1us|222=2us|total=2.96s) [2025-08-14 09:06:04.738457] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-4389] timer task use too much time(end=1755162364738456, start=1755162361780279, delta=2958177) [2025-08-14 09:06:04.738471] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=2946121, expected_run_ts_us_=1755162361792349, ObClockGenerator::getRealClock()=1755162364738471, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:01.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:01.781257] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1083][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1995841, owner: ObLog, click_count: 6, time dist:[FORMAT_END=7, ObMalloc#start=0, ObMalloc#end=1991561, ObMalloc#start=20, ObMalloc#end=55, FREE_END=4198], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:03.978436] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:03.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:04.733295] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:04.738952] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=759362][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:03.978|207=760.51ms|210=1us|222=1us|total=760.53ms) [2025-08-14 09:06:04.741362] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=2404][errcode=-4389] timer task use too much time(end=1755162364741361, start=1755162363978434, delta=762927) [2025-08-14 09:06:04.738492] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:01.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:04.739980] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=956507][errcode=0] epoll handler too much time :957959 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [770]OB_MALLOC COST TOO MUCH TIME, cost_time=968461, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=968459], seq:[0] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=1966429, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=996063, WASH_BLOCK_END=50, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=968477], seq:[0, 1, 2, 3, 4] [2025-08-14 09:06:01.832592] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=632][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:01.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:01.832592] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=632][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2910427, owner: ObLog, click_count: 7, time dist:[FORMAT_END=23, ObMalloc#start=1, ObMalloc#end=943915, ObMalloc#start=35, ObMalloc#end=1966446, ALLOC_END=1, APPEND_END=6], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:00.447601] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1324772, owner: ObLog, click_count: 7, time dist:[FORMAT_END=295570, ObMalloc#start=1, ObMalloc#end=1029194, ALLOC_END=2, APPEND_END=5, ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=55], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:00.752616] EDIAG [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3993303, , BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1d189030 0x1d18285e 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:04.742873] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=349] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:04.752971] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [411][sql_nio4][T0][Y0-0000000000000000-0-0] [lt=938523] kill and revert session(conn.sessid_=3221590336, proxy_sessid=0, ret=0) [2025-08-14 09:06:04.747449] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=2910427][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:01.832|207=2.91s|210=1us|222=2us|total=2.91s) [2025-08-14 09:06:04.755549] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=8086][errcode=-4389] timer task use too much time(end=1755162364755548, start=1755162361832590, delta=2922958) [2025-08-14 09:06:04.747350] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=807] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:04.755274] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=716] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:04.738571] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1995841][errcode=0] epoll handler too much time :2957320 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:04.755704] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17095] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:04.755719] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:01.841708] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=367][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2892193, owner: ObLog, click_count: 9, time dist:[FORMAT_END=30, ObMalloc#start=1, ObMalloc#end=936607, ObMalloc#start=19, ObMalloc#end=995383, ALLOC_END=960146, APPEND_END=7, ALLOC_BLOCK#start=7, ALLOC_BLOCK#end=63], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:03.776257] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=2713] [COMPACTION DAG STATUS] (start_time=1755162350757569, end_time=1755162363765399, dag_status={}) [2025-08-14 09:06:04.755805] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=2892193][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:01.841|207=2.91s|210=0us|222=2us|total=2.91s) [2025-08-14 09:06:04.755824] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4389] timer task use too much time(end=1755162364755824, start=1755162361841706, delta=2914118) [2025-08-14 09:06:04.747335] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=335][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd110, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd3f0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:10.00s, expected_run_time:09:06:04.726, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:04.752403] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=851][errcode=-4389] timer task use too much time(end=1755162364752403, start=1755162360447599, delta=4304804) [2025-08-14 09:06:04.758303] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=6007] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:04.758331] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623262192219, flush_io_finished_ret_:0, flush_io_finished_round_:26383, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:03:17.742295] EDIAG [COORDINATOR] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [471][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1977180, owner: ObLog, click_count: 5, time dist:[FORMAT_END=38, ObMalloc#start=2, ObMalloc#end=1977139, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ea7c4 0x9818f9c 0x1231e0e6 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:59.794722] EDIAG [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=203][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4963302, owner: ObLog, click_count: 6, time dist:[FORMAT_END=4963113, ObMalloc#start=1, ObMalloc#end=58, ObMalloc#start=22, ObMalloc#end=29, FREE_END=79], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x989c577 0x989bc96 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:03.781212] EDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16856][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1575217, , BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:02.744934] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=1933865][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1032860, , BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [389]OB_MALLOC COST TOO MUCH TIME, cost_time=969977, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=40, WASH_BLOCK_END=46, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=8475], seq:[0, 1, 2, 3, 4] [2025-08-14 09:06:04.757979] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=127] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:05.729469] WDIAG ev_invoke_pending (ev.c:2136) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=970020][errcode=0] epoll handler too much time :971497 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:04.777189] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=3278] gc stale ls task succ [400]OB_MALLOC COST TOO MUCH TIME, cost_time=971134, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=971133], seq:[0] [400]OB_MALLOC COST TOO MUCH TIME, cost_time=979494, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=41, WASH_BLOCK_END=2041, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=971148], seq:[0, 1, 2, 3, 4] [2025-08-14 09:06:04.069368] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=828274][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1669626, , BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:04.069368] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=828274][errcode=0] EASY SLOW: thread: 7f99540fb300, lastrun: 1755162363.778349 cost: 0.291017 loop:26718, slowcnt: 2 [2025-08-14 09:06:04.819135] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=1936733] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:06:04.810623] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=1941392] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:06:05.458150] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=306][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:05.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:05.741892] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=256][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:05.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:05.741906] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=283322][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:05.458|207=283.75ms|210=1us|222=1us|total=283.78ms) [2025-08-14 09:06:05.741930] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4389] timer task use too much time(end=1755162365741929, start=1755162365458148, delta=283781) [2025-08-14 09:06:05.742279] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=1100] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623266217619, flush_io_finished_ret_:0, flush_io_finished_round_:26424, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:05.742770] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=23] log compressor cycles once. (ret=0, cost_time=990478, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25878794240, fast_delete_log_mode=false) [2025-08-14 09:06:02.745187] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=1104][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1032513, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:05.742847] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=1032513][errcode=0] epoll handler too much time :2997665 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:02.729192] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [663][T1001_StandbySc][T1001][YB42AC4D0007-00063C495C7A03AE-0-0] [lt=951961][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1048714, , BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:05.742873] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:05.742893] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:05.728346] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=978033] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:05.744155] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=15783] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:05.744173] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=18] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:05.744255] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7705] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:03.778419] EDIAG [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=931697][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1973952, , BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1f9ebc06 0x97c9052 0x1f9819be 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:05:59.768749] EDIAG [STORAGE] get_write_store_ctx_guard_ (ob_access_service.cpp:528) [338][Occam][T1][YB42AC4D0007-00063C4960499C96-0-0] [lt=12][errcode=-4038] LOGGER COST TOO MUCH TIME, cost: 5980741, , BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1c3e9da0 0x946f0ab 0x95f6639 0x95f6278 0x93452fe 0x95f5fd5 0x95f5cd2 0x9344f39 0x9343c25 0x9615b91 0x9613406 0x9613912 0x92df373 0x932ff51 0x932e8be 0x96ac524 0x96a672d 0x96a5033 0x97aa16f 0x124eac91 0x97a9944 0x261395f9 0x1f960c89 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:20.763705] EDIAG [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=966818][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1008491, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1008490, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x2006a4c4 0x997aa00 0x2006ed17 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:04.740230] EDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=485053][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1017595, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ed35c 0x97aead0 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:01.756659] EDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=776][errcode=-4016] LOGGER COST TOO MUCH TIME, cost: 2020414, , BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x13b06ffb 0x958a39e 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:05.748564] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=306][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:05.744183] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:05.757183] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8602][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918114b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:05.757217] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728aa90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:05.757231] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:05.757257] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:05.757271] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:05:59.768749] WDIAG [STORAGE] get_write_store_ctx_guard_ (ob_access_service.cpp:528) [338][Occam][T1][YB42AC4D0007-00063C4960499C96-0-0] [lt=12][errcode=-4038] can not get write store ctx(ret=-4038, ls_id={id:1}, snapshot={this:0x7f995ab50cb8, valid:true, source:5, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:21195874}, scn:{branch:0, seq:2}}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, tx_desc={this:0x7f98b2da56e0, tx_id:{txid:21195874}, state:1, addr:"172.77.0.7:2882", tenant_id:1, session_id:1, assoc_session_id:1, client_sid:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1755162356734727, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999983, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1753069619, cluster_version:17180067074, seq_base:1755162225746389, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:2}) [2025-08-14 09:06:05.758833] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:05.759095] INFO [COMMON] loop_running_dag_net_list (ob_tenant_dag_scheduler.cpp:3845) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=1008491] loop running dag net list(ret=0, running_dag_net_list_size=0, slow_dag_net_cnt=0, finished_dag_net_list_size=0) [2025-08-14 09:06:04.782583] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=191] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:05.759306] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=975247][errcode=0] epoll handler too much time :976728 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:04.740230] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=485053][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:04.254|207=485.79ms|210=0us|222=2us|total=1.50s) [2025-08-14 09:06:05.759385] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1017595][errcode=-4389] timer task use too much time(end=1755162365759384, start=1755162364254437, delta=1504947) [2025-08-14 09:06:05.759402] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1275869, expected_run_ts_us_=1755162364483531, ObClockGenerator::getRealClock()=1755162365759401, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:04.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:05.759421] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:04.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:05.759439] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=1014805, expected_run_ts_us_=1755162364744633, ObClockGenerator::getRealClock()=1755162365759439, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:04.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:05.759451] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:04.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:05.759816] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:05.759840] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:05.759854] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:05:59.768713] EDIAG [STORAGE] get_write_store_ctx_guard_ (ob_access_service.cpp:528) [337][EvtHisUpdTask][T1][YB42AC4D0007-00063C4960299C79-0-0] [lt=20][errcode=-4038] LOGGER COST TOO MUCH TIME, cost: 5982591, owner: ObLog, click_count: 8, time dist:[FORMAT_END=5979605, ObMalloc#start=1, ObMalloc#end=74, ObMalloc#start=29, ObMalloc#end=68, ObMalloc#start=2336, ObMalloc#end=56, FREE_END=421], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1c3e9da0 0x946f0ab 0x95f6639 0x95f6278 0x93452fe 0x95f5fd5 0x95f5cd2 0x9344f39 0x9343c25 0x9615b91 0x9613406 0x9613912 0x92df373 0x932ff51 0x932e8be 0x96ac524 0x96a672d 0x96a5033 0x97aa16f 0x124eac91 0x97a9944 0x261395f9 0x1f900517 0x1f8fe2bf 0x9a5f486 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:05.760000] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=158] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:05.760018] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:05.759525] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=2020414][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:05.760255] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=724][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:05.760273] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:05.760286] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:05.760300] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:05.760313] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:05.760465] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:05.760479] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:04.752106] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [717][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10597] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:04.782226] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1038] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:05.761135] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=977430][errcode=0] epoll handler too much time :978916 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:04.736561] EDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=956759][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1023115, , BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:04.736561] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=956759][errcode=0] epoll handler too much time :958211 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:05.761261] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1023115] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:05.761274] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:01.752837] EDIAG [RS.COMPACTION] try_idle (ob_major_merge_scheduler.cpp:226) [1057][T1_MergeSchedul][T1][YB42AC4D0007-00063C4961599CC3-0-0] [lt=27][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2022514, , BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1125d502 0x1125cf81 0x1125a4c3 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:05.763998] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=255][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:05.764269] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1159] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:05.766799] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=277][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:05.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:05.766836] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:05.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:17.765594] EDIAG [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4976664, owner: ObLog, click_count: 7, time dist:[FORMAT_END=13, ObMalloc#start=1, ObMalloc#end=2003616, ObMalloc#start=1997866, ObMalloc#end=975164, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9bcc09 0xf9bcb24 0x96b246f 0x96b1c88 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [404]OB_MALLOC COST TOO MUCH TIME, cost_time=956181, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=27, WASH_BLOCK_END=38, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=9859], seq:[0, 1, 2, 3, 4] [2025-08-14 09:06:05.084393] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=737615][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1635183, owner: ObLog, click_count: 9, time dist:[ALLOC_BLOCK#start=7, ALLOC_BLOCK#end=71, ALLOC_BLOCK#start=665672, ALLOC_BLOCK#end=80, FORMAT_END=13150, ObMalloc#start=1, ObMalloc#end=956195, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:05.084393] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=737615][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162362.745179 cost: 2.339212 loop:26690, slowcnt: 15 [686]OB_MALLOC COST TOO MUCH TIME, cost_time=947635, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=947616], seq:[0, 1, 2] [686]OB_MALLOC COST TOO MUCH TIME, cost_time=947655, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=947644], seq:[0, 1] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=947668, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=947646], seq:[0, 1, 2, 3, 4] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=950832, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=45, WASH_BLOCK_END=3109, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=947676], seq:[0, 1, 2, 3, 4] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=948157, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=948132, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=948191, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=948167, WASH_BLOCK_END=24], seq:[0, 1, 2] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=948213, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=948188, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=948229, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=948221, WASH_BLOCK_END=8], seq:[0, 1, 2] [864]OB_MALLOC COST TOO MUCH TIME, cost_time=941473, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=941431, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [864]OB_MALLOC COST TOO MUCH TIME, cost_time=941490, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=941482, WASH_BLOCK_END=8], seq:[0, 1, 2] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=940274, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=16, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=940230, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=940291, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=940282, WASH_BLOCK_END=9], seq:[0, 1, 2] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=939223, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=939189, ALLOC_CHUNK_END=3], seq:[0, 1, 2, 3, 4, 5, 6, 7] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=939243, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=939235, WASH_BLOCK_END=8], seq:[0, 1, 2] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=939226, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=939193, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=939242, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=939234, WASH_BLOCK_END=8], seq:[0, 1, 2] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=938551, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=938520, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=938578, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=938560, WASH_BLOCK_END=18], seq:[0, 1, 2] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=938088, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=938057, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=938106, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=938096, WASH_BLOCK_END=10], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=930839, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=930798, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=930859, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=930850, WASH_BLOCK_END=9], seq:[0, 1, 2] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=909743, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=909697, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=909763, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=909754, WASH_BLOCK_END=9], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=880199, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=880162, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=880286, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=880247, WASH_BLOCK_END=39], seq:[0, 1, 2] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=880324, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=880284, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=880352, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=880341, WASH_BLOCK_END=11], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=866241, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=866205, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=866267, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=866253, WASH_BLOCK_END=14], seq:[0, 1, 2] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=858617, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=858581, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=858637, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=858628, WASH_BLOCK_END=9], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=842302, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=842268, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=842322, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=842312, WASH_BLOCK_END=10], seq:[0, 1, 2] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=732187, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=732152, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=732207, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=732198, WASH_BLOCK_END=9], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=704318, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=704281, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=704354, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=704340, WASH_BLOCK_END=14], seq:[0, 1, 2] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=637488, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=637444, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=637513, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=637501, WASH_BLOCK_END=12], seq:[0, 1, 2] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=357739, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=17, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=357693, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=357766, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=357754, WASH_BLOCK_END=12], seq:[0, 1, 2] [2025-08-14 09:06:05.759108] INFO [COMMON] loop_finished_dag_net_list (ob_tenant_dag_scheduler.cpp:3868) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=12] loop finsihed dag net list(ret=0, finished_dag_net_list_size=0) [2025-08-14 09:06:05.769926] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fc830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722258] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=950888][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:05.769|207=952.33ms|210=0us|222=2us|total=952.35ms) [2025-08-14 09:06:06.722282] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4389] timer task use too much time(end=1755162366722282, start=1755162365769924, delta=952358) [2025-08-14 09:06:06.722295] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=966105, expected_run_ts_us_=1755162365756188, ObClockGenerator::getRealClock()=1755162366722294, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722313] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722326] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=966006, expected_run_ts_us_=1755162365756319, ObClockGenerator::getRealClock()=1755162366722326, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a54b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722340] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a54b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722352] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=961804, expected_run_ts_us_=1755162365760547, ObClockGenerator::getRealClock()=1755162366722352, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72edc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722364] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72edc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722375] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=957157, expected_run_ts_us_=1755162365765218, ObClockGenerator::getRealClock()=1755162366722375, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72edb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722386] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72edb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722400] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=956277, expected_run_ts_us_=1755162365766122, ObClockGenerator::getRealClock()=1755162366722400, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722411] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722423] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=953789, expected_run_ts_us_=1755162365768633, ObClockGenerator::getRealClock()=1755162366722423, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722434] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722445] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=961447, expected_run_ts_us_=1755162365760998, ObClockGenerator::getRealClock()=1755162366722445, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722459] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722471] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=951636, expected_run_ts_us_=1755162365770834, ObClockGenerator::getRealClock()=1755162366722471, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722482] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722493] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=943154, expected_run_ts_us_=1755162365779339, ObClockGenerator::getRealClock()=1755162366722493, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728ac10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722504] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728ac10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722519] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=919613, expected_run_ts_us_=1755162365802905, ObClockGenerator::getRealClock()=1755162366722518, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.802, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722530] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.802, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722541] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=921066, expected_run_ts_us_=1755162365801474, ObClockGenerator::getRealClock()=1755162366722541, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722552] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722564] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=918253, expected_run_ts_us_=1755162365804310, ObClockGenerator::getRealClock()=1755162366722564, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728f9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722577] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728f9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722588] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=904048, expected_run_ts_us_=1755162365818540, ObClockGenerator::getRealClock()=1755162366722588, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eae10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722599] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eae10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722611] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=899704, expected_run_ts_us_=1755162365822907, ObClockGenerator::getRealClock()=1755162366722611, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722622] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722636] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=857278, expected_run_ts_us_=1755162365865358, ObClockGenerator::getRealClock()=1755162366722636, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722647] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722659] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=855885, expected_run_ts_us_=1755162365866773, ObClockGenerator::getRealClock()=1755162366722658, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.866, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722670] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:05.866, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722687] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=393588, expected_run_ts_us_=1755162366329099, ObClockGenerator::getRealClock()=1755162366722687, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:06.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722700] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:06.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.722843] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=190] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:05.779935] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=181] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:06:05.782392] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:06.723101] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=939287][errcode=0] epoll handler too much time :940719 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:05.783228] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=155] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:06.723256] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=938607][errcode=0] epoll handler too much time :940035 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:05.842025] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:05.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:06.723777] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=880415][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:05.842|207=881.75ms|210=0us|222=2us|total=881.78ms) [2025-08-14 09:06:06.723804] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=-4389] timer task use too much time(end=1755162366723803, start=1755162365842022, delta=881781) [2025-08-14 09:06:06.604265] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fe030, caller:ob_table_lock_service.cpp:start:276, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE6$_1273, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe310, time_wheel:0x7f99535f3990, thread_pool:0x7f99535f3170, is_running:1, time_interval:10.00s, expected_run_time:09:06:06.426, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:06.723887] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=119437][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:06.604|207=119.62ms|210=0us|222=1us|total=119.64ms) [2025-08-14 09:06:06.723903] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] timer task use too much time(end=1755162366723903, start=1755162366604262, delta=119641) [2025-08-14 09:06:05.864059] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:06.724031] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=858661][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:05.757185] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12981] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:06.724184] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=965536] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:06.724196] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:06.724206] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:06.724215] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:05.990845] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:05.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:06.724227] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:06:06.018779] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:06.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:06.724247] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=18] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:06:06.724248] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=732278][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:05.990|207=733.39ms|210=13us|222=2us|total=733.42ms) [2025-08-14 09:06:06.724251] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=704403][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:06.187|207=705.47ms|210=0us|222=2us|total=705.49ms) [2025-08-14 09:06:06.724268] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task use too much time(end=1755162366724268, start=1755162365990842, delta=733426) [2025-08-14 09:06:06.724271] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task use too much time(end=1755162366724271, start=1755162366018776, delta=705495) [2025-08-14 09:06:06.724283] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=479649, expected_run_ts_us_=1755162366244633, ObClockGenerator::getRealClock()=1755162366724283, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:06.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:06.724282] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=715147, expected_run_ts_us_=1755162366009134, ObClockGenerator::getRealClock()=1755162366724282, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:06.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:06.724317] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:06.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:06.724333] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=635016, expected_run_ts_us_=1755162366089316, ObClockGenerator::getRealClock()=1755162366724332, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:06.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:06.724345] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:06.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:06.724362] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=520064, expected_run_ts_us_=1755162366204297, ObClockGenerator::getRealClock()=1755162366724361, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:06.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:06.724375] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:06.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:05:59.746663] EDIAG [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=360][errcode=0] LOGGER COST TOO MUCH TIME, cost: 6003426, owner: ObLog, click_count: 8, time dist:[FORMAT_END=6003179, ObMalloc#start=0, ObMalloc#end=65, ObMalloc#start=22, ObMalloc#end=50, FREE_END=110, ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=82], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xff22be4 0x97a9cd7 0xff1a920 0x982e1ac 0xfefdb03 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:05.758670] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:06.724510] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=964381][errcode=0] epoll handler too much time :965847 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:06.724307] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:06.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:06.085960] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=1746526][errcode=0] EASY SLOW: thread: 7f9954779300, lastrun: 1755162365.783221 cost: 0.302738 loop:26716, slowcnt: 3 [2025-08-14 09:06:06.739793] WDIAG [STORAGE] reset (ob_access_service.cpp:42) [338][Occam][T1][YB42AC4D0007-00063C4960499C96-0-0] [lt=5980741][errcode=-4389] guard used too much time(guard_used_us=5962840, ls_id={id:1}, lbt()="0x26a1806d 0x1c3e21a6 0x940a506 0x946f0bf 0x95f6639 0x95f6278 0x93452fe 0x95f5fd5 0x95f5cd2 0x9344f39 0x9343c25 0x9615b91 0x9613406 0x9613912 0x92df373 0x932ff51 0x932e8be 0x96ac524 0x96a672d 0x96a5033 0x97aa16f 0x124eac91 0x97a9944 0x261395f9 0x1f960c89 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3") [2025-08-14 09:06:04.738883] EDIAG [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1531) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=1973122][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2007631, owner: ObLog, click_count: 8, time dist:[ALLOC_BLOCK#start=2676, ALLOC_BLOCK#end=2, FORMAT_END=2003225, ObMalloc#start=0, ObMalloc#end=57, ObMalloc#start=26, ObMalloc#end=56, FREE_END=1588], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x9ace3c3 0x9acda90 0xfad1d79 0xfad1b46 0x148e2b50 0x93d7bb4 0x9394489 0x9725f91 0x124dfd77 0x96a5a4d 0x97aa16f 0x124eac91 0x97a9944 0x261395f9 0xfa0e430 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:06.741364] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=362][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:06.744500] WDIAG [STORAGE] get_write_store_ctx_guard (ob_access_service.cpp:497) [338][Occam][T1][YB42AC4D0007-00063C4960499C96-0-0] [lt=23][errcode=-4038] fail to get write store ctx gurad(ret=-4038, ls_id={id:1}, tx_desc={this:0x7f98b2da56e0, tx_id:{txid:21195874}, state:1, addr:"172.77.0.7:2882", tenant_id:1, session_id:1, assoc_session_id:1, client_sid:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1755162356734727, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999983, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1753069619, cluster_version:17180067074, seq_base:1755162225746389, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:2}) [2025-08-14 09:06:06.746347] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=960428] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:06.752014] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1085) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=12097] replay status is not follower(min_unreplayed_lsn={lsn:1604088821}, base_lsn={lsn:1409200128}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873479, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:12492179, ret_code_:0}}, next_to_submit_lsn_:{lsn:1604088821}, next_to_submit_scn_:{val:1755100948296325001, v:0}, base_lsn_:{lsn:1409200128}, base_scn_:{val:1753840802540914000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:1604088821}, end_lsn:{lsn:1604088821}, read_buf:{buf_len_:3747840, buf_:0x7f98f6a05000}, block_size:67104768, log_storage_:0x7f990bfebbf0, log_storage_:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:42, read_io_size:147709952, read_disk_cost_ts:1061450, start_lsn:{lsn:1409200128}}}}}}) [2025-08-14 09:06:06.739496] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=851] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1001, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:8571658, max_lsn:7323687369, max_scn:{val:1755161749741193002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:7278610047}, reuse_lsn:{lsn:7323686131}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:7281744329}}, last_submit_lsn:{lsn:7323686131}, last_submit_end_lsn:{lsn:7323687369}, last_submit_log_id:8571658, last_submit_log_pid:49, max_flushed_lsn:{lsn:7323686009}, max_flushed_end_lsn:{lsn:7323686131}, max_flushed_log_pid:49, committed_end_lsn:{lsn:7323686009}, last_slide_log_id:8571656, last_slide_scn:{val:1755161747742105001, v:0}, last_slide_lsn:{lsn:7323685887}, last_slide_end_lsn:{lsn:7323686009}, last_slide_log_pid:49, last_slide_log_accum_checksum:2505458373, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f990112a530}, [StateMgr]={this:0x7f990112e9b0, self:"172.77.0.7:2882", palf_id:1001, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"PENDING", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:49}, leader:"0.0.0.0:0", leader_epoch:-1, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:7323687369}, scan_disk_log_finished:true, last_check_start_id:8571657, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161767744317, palf_role_change_cb:0x7f9901135470, allow_vote:true}, [ConfigMgr]={palf_id:1001, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:49, prev:{version:1, config:{config_version:{proposal_id:48, config_seq:47}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:49, config_seq:48}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_mode_pid:49}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_end_lsn:{lsn:7323685240}, prev_mode_pid:49}, reconfig_barrier:{prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_end_lsn:{lsn:7323685240}, prev_mode_pid:49}, persistent_config_version:{proposal_id:49, config_seq:48}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755161739755312, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755161739755312, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f990112bc70}, [ModeMgr]={palf_id:1001, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, accepted_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1001, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1753069671378640001, v:0}, base_lsn_for_block_gc:{lsn:7247314944}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:49}, log_config_meta:{version:3, proposal_id:49, prev:{version:1, config:{config_version:{proposal_id:48, config_seq:47}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:49, config_seq:48}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_mode_pid:49}, log_snapshot_meta:{version:2, base_lsn:{lsn:7247314944}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1753069671378640000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}}, log_meta_storage:{log_tail:{lsn:765952}, readable_log_tail:{lsn:765952}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7381524480}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/meta", dir_fd:232, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66338816, block_header_serialize_buf_:0x7f9901130bb8, flashback_version:0}, log_storage:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, palf_epoch:2, last_purge_throttling_ts:1755161769777679, this:0x7f990112f0f0}, [Reconfirm]={palf_id:1001, self:"172.77.0.7:2882", state:"WAITING_LOG_FLUSHED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:1755161769777679, this:0x7f990112eb30}) [2025-08-14 09:06:06.752081] INFO [CLOG] operator() (ob_log_replay_service.cpp:1678) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=64] get_replay_process success(id={id:1}, submitted_log_size=194888693, unsubmitted_log_size=0, replayed_log_size=194888693, unreplayed_log_size=0) [2025-08-14 09:06:06.752093] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12565] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [2025-08-14 09:06:06.752101] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=13] get_min_unreplayed_log_info(lsn={lsn:7323685240}, scn={val:1755100946796590002, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873485, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:8946764, ret_code_:0}}, next_to_submit_lsn_:{lsn:7323685240}, next_to_submit_scn_:{val:1755100946796590002, v:0}, base_lsn_:{lsn:7180210176}, base_scn_:{val:1753840803166611000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:7323685240}, end_lsn:{lsn:7323685240}, read_buf:{buf_len_:3747840, buf_:0x7f98f7a05000}, block_size:67104768, log_storage_:0x7f9901131bf0, log_storage_:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:28, read_io_size:98508800, read_disk_cost_ts:645339, start_lsn:{lsn:7180210176}}}}}}) [2025-08-14 09:06:06.752118] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21] current log_task status(palf_id=1001, self="172.77.0.7:2882", log_id=8571657, log_task={header:{begin_lsn:{lsn:7323686009}, end_lsn:{lsn:7323686131}, log_id:8571657, min_scn:{val:1755161749741193000, v:0}, max_scn:{val:1755161749741193000, v:0}, data_len:66, proposal_id:49, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:7323686009}, data_checksum:2222604892, accum_checksum:1531236791, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1755161749745909, freeze_ts:1755161749745914, submit_ts:1755161749745940, flushed_ts:1755161785758706, gen_to_freeze cost time:5, gen_to_submit cost time:31, submit_to_flush cost time:36012766}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [2025-08-14 09:06:06.752806] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1014][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d40b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:06.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.753402] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=592][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:06.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.753427] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:06.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.753440] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:06.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.753452] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:06.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.756376] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=391] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:06.756432] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=236] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:06.763106] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=286][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:06.763429] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=174][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:06.763447] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:06.766446] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=193] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:06.768327] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:06.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:06.768369] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:06.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:03:24.737230] EDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EED7-0-0] [lt=24][errcode=0] LOGGER COST TOO MUCH TIME, cost: 161788037, owner: ObLog, click_count: 7, time dist:[FORMAT_END=62, ObMalloc#start=0, ObMalloc#end=65, ObMalloc#start=161787905, ObMalloc#end=2, ALLOC_END=0, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ed35c 0x9715e63 0x1dd7c9da 0x1dc1f9ca 0x1dd7bdd2 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:06.752143] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1085) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=41] replay status is not follower(min_unreplayed_lsn={lsn:7323685240}, base_lsn={lsn:7180210176}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873485, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:8946764, ret_code_:0}}, next_to_submit_lsn_:{lsn:7323685240}, next_to_submit_scn_:{val:1755100946796590002, v:0}, base_lsn_:{lsn:7180210176}, base_scn_:{val:1753840803166611000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:7323685240}, end_lsn:{lsn:7323685240}, read_buf:{buf_len_:3747840, buf_:0x7f98f7a05000}, block_size:67104768, log_storage_:0x7f9901131bf0, log_storage_:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:28, read_io_size:98508800, read_disk_cost_ts:645339, start_lsn:{lsn:7180210176}}}}}}) [2025-08-14 09:03:24.737230] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EED7-0-0] [lt=24][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:24.736159] EDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEBB-0-0] [lt=27][errcode=0] LOGGER COST TOO MUCH TIME, cost: 161789697, owner: ObLog, click_count: 9, time dist:[FORMAT_END=39, ObMalloc#start=7, ObMalloc#end=59, ObMalloc#start=161789330, ObMalloc#end=71, ObMalloc#start=186, ObMalloc#end=1, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ed35c 0x9715e63 0x1dd7c9da 0x1dc1f9ca 0x1dd7bdd2 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:24.736159] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEBB-0-0] [lt=27][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:06.770413] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1154][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:06.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:06.772106] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=351][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:06.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.757426] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:06.772331] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14882][errcode=0] epoll handler too much time :14915 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:05.777395] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=950934] gc stale ls task succ [2025-08-14 09:06:06.747315] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=5673][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:06.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:06.772411] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=25055][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:06.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:06.728868] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=946213][errcode=0] epoll handler too much time :947653 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:06.777459] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=993462] gc stale ls task succ [2025-08-14 09:06:06.777535] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=4803][errcode=0] get_fetch_log_dst failed or invalid(ret=0, palf_id=1, self="172.77.0.7:2882", fetch_log_dst="172.77.0.7:2882") [2025-08-14 09:06:06.777551] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:06:06.777567] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:8607757, max_lsn:1604090331, max_scn:{val:1755161756730847001, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:1556737504}, reuse_lsn:{lsn:1604090331}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:1562147291}}, last_submit_lsn:{lsn:1604090203}, last_submit_end_lsn:{lsn:1604090331}, last_submit_log_id:8607757, last_submit_log_pid:50, max_flushed_lsn:{lsn:1604090203}, max_flushed_end_lsn:{lsn:1604090331}, max_flushed_log_pid:50, committed_end_lsn:{lsn:1604090090}, last_slide_log_id:8607755, last_slide_scn:{val:1755161749741193000, v:0}, last_slide_lsn:{lsn:1604089968}, last_slide_end_lsn:{lsn:1604090090}, last_slide_log_pid:50, last_slide_log_accum_checksum:1446576759, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f990bfe4530}, [StateMgr]={this:0x7f990bfe89b0, self:"172.77.0.7:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"PENDING", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:51}, leader:"0.0.0.0:0", leader_epoch:-1, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:1604090331}, scan_disk_log_finished:true, last_check_start_id:8607756, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161901828039, palf_role_change_cb:0x7f990bfef470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:50, prev:{version:1, config:{config_version:{proposal_id:49, config_seq:45}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:50, config_seq:46}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_mode_pid:50}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_end_lsn:{lsn:1604088821}, prev_mode_pid:50}, reconfig_barrier:{prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_end_lsn:{lsn:1604088821}, prev_mode_pid:50}, persistent_config_version:{proposal_id:50, config_seq:46}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755161738854289, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755161738854289, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f990bfe5c70}, [ModeMgr]={palf_id:1, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:50, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:1543409664}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:51}, log_config_meta:{version:3, proposal_id:50, prev:{version:1, config:{config_version:{proposal_id:49, config_seq:45}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:50, config_seq:46}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_mode_pid:50}, log_snapshot_meta:{version:2, base_lsn:{lsn:1543409664}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:712704}, readable_log_tail:{lsn:712704}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:1610514432}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/meta", dir_fd:225, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66392064, block_header_serialize_buf_:0x7f990bfeabb8, flashback_version:0}, log_storage:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1755161901828095, this:0x7f990bfe90f0}, [Reconfirm]={palf_id:1, self:"172.77.0.7:2882", state:"RECONFIRM_MODE_META", new_proposal_id:51, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"172.77.0.7:2882", majority_max_accept_pid:50, majority_max_lsn:{lsn:1604090331}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:1755161901828373, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f990bfe8b30}) [2025-08-14 09:06:06.777739] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=167] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161974728707, last_flushed_end_lsn:{lsn:1604090331}}]) [2025-08-14 09:06:06.777754] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12] current log_task status(palf_id=1, self="172.77.0.7:2882", log_id=8607756, log_task={header:{begin_lsn:{lsn:1604090090}, end_lsn:{lsn:1604090203}, log_id:8607756, min_scn:{val:1755161749741193001, v:0}, max_scn:{val:1755161749741193001, v:0}, data_len:57, proposal_id:50, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:1604090090}, data_checksum:2031283194, accum_checksum:759186326, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1755161750716165, freeze_ts:1755161750716173, submit_ts:1755161750716183, flushed_ts:1755161853765770, gen_to_freeze cost time:8, gen_to_submit cost time:18, submit_to_flush cost time:103049587}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161974728707, last_flushed_end_lsn:{lsn:1604090331}}]) [2025-08-14 09:06:06.777779] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:06:06.772290] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=2007631][errcode=-4038] result set close failed(ret=-4038) [2025-08-14 09:06:06.777840] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:199) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=5532][errcode=-4038] result set close failed(ret=-4038) [2025-08-14 09:06:06.777851] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:951) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=8][errcode=-4012] failed to close result(close_ret=-4038, ret=-4012) [2025-08-14 09:06:06.769512] INFO [CLOG] operator() (ob_log_replay_service.cpp:1678) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=17341] get_replay_process success(id={id:1001}, submitted_log_size=143475064, unsubmitted_log_size=0, replayed_log_size=143475064, unreplayed_log_size=0) [2025-08-14 09:06:06.780531] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:174) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=10974] dump tenant replay process(tenant_id=1002, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=322, replayed_log_size(MB)=322, last_submitted_log_size(MB)=322, last_replayed_log_size(MB)=322, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-08-14 09:06:06.780553] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a49d8, timer:0x7f9922d47110, task:0x7f9922eb44d0, task_type:"N9oceanbase10logservice17ReplayProcessStatE", scheduled_time:1755162340759219, pushed_time:1755162340764019, delay:10000000}, delay_in_thread_pool=18982623, threashold=500000) [2025-08-14 09:06:06.780959] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=43939] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:06.772316] WDIAG [COORDINATOR] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [471][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=1977180][errcode=0] cost too much time:ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_failure(), (this=|threshold=1.00s|start at 09:06:06.781|total=172.04s) [2025-08-14 09:06:06.781276] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [471][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8941][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:0, time_interval:100.00ms, expected_run_time:09:06:06.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:06.781333] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [471][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=55][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:06:06.781|total=172.04s) [2025-08-14 09:06:06.781807] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1025] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:06.782177] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919acf10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:06.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.782203] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:06.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.782253] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:06.782345] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:06.792218] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919abe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:06.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.792259] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:06.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.792278] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199fe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919acd90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:06.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.792290] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e80b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:06.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.792302] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a31b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ede10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:06.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.794187] INFO handle (memory_dump.cpp:671) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4976664] statistics: tenant_cnt: 6, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 168 271 0 1 2 110 1 0 1 3 7 0 0 1 4 1 1 0 1 5 85 6 0 1 6 13 0 0 1 7 22 2 0 1 8 53 1 0 1 12 17 3 0 1 13 10 1 0 1 14 1 1 0 1 15 1 1 0 1 16 8 7 0 1 18 4 3 0 1 20 2 1 0 1 22 1 1 0 1 24 1 0 0 1 25 6 0 0 500 0 560 275 0 500 7 192 5 0 500 8 41 1 0 500 9 2 1 0 500 10 2 1 0 500 16 58 4 0 500 17 60 7 0 500 22 38 64 0 500 23 27 10 0 508 0 2 7 0 508 8 5 1 0 509 0 2 7 0 509 8 5 1 0 1001 0 147 252 0 1001 2 26 1 0 1001 3 5 0 0 1001 4 1 1 0 1001 5 85 6 0 1001 6 11 0 0 1001 7 30 2 0 1001 8 36 1 0 1001 12 17 3 0 1001 13 3 1 0 1001 14 1 1 0 1001 16 7 3 0 1001 18 4 3 0 1001 20 4 1 0 1001 25 8 0 0 1002 0 263 279 0 1002 2 198 1 0 1002 3 4 0 0 1002 4 1 1 0 1002 5 101 6 0 1002 6 14 0 0 1002 7 6 2 0 1002 8 73 1 0 1002 12 18 3 0 1002 13 7 1 0 1002 14 1 2 0 1002 16 10 7 0 1002 18 9 3 0 1002 20 2 1 0 1002 21 1 0 0 1002 24 1 0 0 1002 25 7 0 0 cost_time: 169050577 [2025-08-14 09:06:06.794225] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=19] [MEMORY] tenant: 1, limit: 2147483648 hold: 1013174272 cache_hold: 56623104 cache_used: 56623104 cache_item_count: 27 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 444993536 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 230686720 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 1163264 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 483328 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 51617792 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 1458176 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 14204928 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 111149056 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35651584 limit= 429496720 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 8003584 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= PS_CACHE_CTX_ID hold_bytes= 65536 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 14827520 limit= 9223372036854775807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 3481600 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3964928 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 212992 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 884736 limit= 9223372036854775807 [2025-08-14 09:06:06.798556] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=936306] PNIO [ratelimit] time: 1755162366798554, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007439, add_bytes: 0 [2025-08-14 09:06:06.800590] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=811][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:06.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:06.812336] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:06.806, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.812378] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:06.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.812391] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:06.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.812408] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=915113] gc stale ls task succ [2025-08-14 09:06:06.812415] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:06.800, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.812429] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728e330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7290690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:06.805, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.812444] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:06.803, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.824092] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:06:06.824120] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:06.825310] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=1898225][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:06:06.826378] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1083][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:06.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:06.826515] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=1900382][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:06:06.832469] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:06.824, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.842531] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:06.832, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.872717] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ade10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:06.864, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.872755] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:06.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.872770] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919adc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:06.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.872788] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919adb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:06.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.882780] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72865b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:06.878, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.902887] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728de30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7290b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:06.899, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.912957] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:06.902, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:06.924137] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:06.924503] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=842578] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:06.926411] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:06.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:06.993742] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:06.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.012510] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:07.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.012553] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:07.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.021831] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:07.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.024150] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:07.026453] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:06.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.092992] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:07.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.124070] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.111, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:07.124173] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:07.126517] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:07.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.130974] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=643782][errcode=0] EASY SLOW: thread: 7f995427d300, lastrun: 1755162250.789339 cost: 116.341631 loop:26617, slowcnt: 859 [2025-08-14 09:06:07.213692] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:07.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.224223] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:07.226585] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:07.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.245075] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:07.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.255120] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:07.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.263963] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:07.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.263993] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:07.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.273204] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:07.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.303373] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:07.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.324256] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:07.326648] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:07.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.424318] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:07.426727] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:07.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.496447] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:07.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.515391] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:07.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.515437] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:07.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.524345] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:07.524637] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:07.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.526749] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:07.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.595869] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:07.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.624405] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:07.626811] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:07.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.691653] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=339][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571630c0, caller:mds_tenant_service.cpp:mtl_start:117, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_145, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571633a0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.691688] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:1, time_interval:3.00s, expected_run_time:09:06:07.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.716507] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:07.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.724788] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=941522] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:06:07.724975] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:06.823902] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:06.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.263628] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:07.726346] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=898][errcode=0] cost too much time:ob_occam_timer.h:int oceanbase::common::occam::ObOccamTimerTask::commit_task_to_thread_pool_(), (this=|threshold=100.00ms|start at 09:06:06.826|289=899.82ms|total=899.84ms) [2025-08-14 09:06:07.740049] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=13676][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:06.826|187=1us|203=0us|207=913.52ms|210=0us|222=2us|total=913.55ms) [2025-08-14 09:06:07.740075] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4389] timer task use too much time(end=1755162367740074, start=1755162366826524, delta=913550) [2025-08-14 09:06:07.740072] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=20] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 444993536 used= 389779488 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 70 washed_blocks= 287 washed_size= 70221824 [MEMORY] request_cached_chunk_cnt= 6 [MEMORY] hold= 389779488 used= 379526659 count= 32900 avg_used= 11535 mod=SUMMARY [2025-08-14 09:06:07.740252] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=149] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 230686720 used= 228884480 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 block_cnt= 110 chunk_cnt= 110 mod=Memstore [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:06:07.740285] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=26] [MEMORY] tenant_id= 1 ctx_id= EXECUTE_CTX_ID hold= 1163264 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 15 washed_size= 13516800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:07.740304] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:06:07.740330] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=19] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 51617792 used= 47452416 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 84 washed_blocks= 334 washed_size= 126640128 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 47452416 used= 43070492 count= 3992 avg_used= 10789 mod=SUMMARY [2025-08-14 09:06:07.740347] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1 ctx_id= WORK_AREA hold= 1458176 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 13 washed_blocks= 66 washed_size= 25804800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:07.740365] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 14204928 used= 10970224 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 277 washed_size= 31932416 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10970224 used= 6078924 count= 18573 avg_used= 327 mod=SUMMARY [2025-08-14 09:06:07.740393] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 111149056 used= 108896256 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 block_cnt= 211 chunk_cnt= 53 mod=CoStack [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 mod=SUMMARY [2025-08-14 09:06:07.740415] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 56623104 used= 56623104 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 56623104 used= 0 count= 27 avg_used= 0 mod=SUMMARY [2025-08-14 09:06:07.740433] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 34365376 limit= 429496720 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 34365376 used= 31086608 count= 475 avg_used= 65445 mod=SUMMARY [2025-08-14 09:06:07.740450] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 8003584 used= 4947392 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 10 washed_blocks= 23 washed_size= 12967936 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4947392 used= 4793344 count= 604 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:07.731436] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EED7-0-0] [lt=161788037] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-08-14 09:06:07.740468] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:07.740486] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 65536 used= 31024 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 2031616 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 31024 used= 27058 count= 20 avg_used= 1352 mod=SUMMARY [2025-08-14 09:06:07.740480] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [651][T1001_Occam][T1001][YB42AC4D0007-00063C495CE9EED7-0-0] [lt=9014][errcode=0] cost too much time:mds_tenant_service.cpp:static int oceanbase::storage::mds::ObTenantMdsService::for_each_ls_in_tenant(const ObFunction &), (this=|threshold=3.00s|start at 09:03:20.771|207=0us|211=4us|218=2us|211=166.96s|total=166.97s) [2025-08-14 09:06:07.740504] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1 ctx_id= RPC_CTX_ID hold= 14827520 used= 6135808 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 6135808 used= 4051328 count= 253 avg_used= 16013 mod=SUMMARY [2025-08-14 09:06:07.740520] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 3481600 used= 2391232 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 10 washed_size= 4907008 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2391232 used= 2367040 count= 36 avg_used= 65751 mod=SUMMARY [2025-08-14 09:06:07.740548] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 3964928 used= 3080192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 3 washed_size= 229376 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3080192 used= 2983936 count= 376 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:07.740567] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1 ctx_id= SCHEMA_SERVICE hold= 212992 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1884160 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:07.740586] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:07.740609] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=17] [MEMORY] tenant_id= 1 ctx_id= MERGE_NORMAL_CTX_ID hold= 884736 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 6 washed_blocks= 9 washed_size= 11698176 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:07.740640] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant: 500, limit: 9223372036854775807 hold: 692576256 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 343265280 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 35291136 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 85983232 limit= 9223372036854775807 [MEMORY] ctx_id= LIBEASY hold_bytes= 3145728 limit= 9223372036854775807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 6291456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 3104768 limit= 9223372036854775807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 37584896 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 78925824 limit= 9223372036854775807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 98983936 limit= 9223372036854775807 [2025-08-14 09:06:07.740674] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 343265280 used= 301729776 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 493 washed_blocks= 2219 washed_size= 910721024 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 301729776 used= 294291829 count= 12726 avg_used= 23125 mod=SUMMARY [2025-08-14 09:06:07.740704] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 500 ctx_id= GLIBC hold= 35291136 used= 21975792 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 186 washed_blocks= 807 washed_size= 367362048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 21975792 used= 10371253 count= 131704 avg_used= 78 mod=SUMMARY [2025-08-14 09:06:07.740722] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 500 ctx_id= CO_STACK hold= 85983232 used= 83091456 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 83091456 used= 82936896 count= 161 avg_used= 515136 block_cnt= 161 chunk_cnt= 41 mod=CoStack [MEMORY] hold= 83091456 used= 82936896 count= 161 avg_used= 515136 mod=SUMMARY [2025-08-14 09:06:07.740739] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 500 ctx_id= LIBEASY hold= 3145728 used= 3070816 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1048576 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3070816 used= 2859784 count= 223 avg_used= 12824 mod=SUMMARY [2025-08-14 09:06:07.740755] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 500 ctx_id= LOGGER_CTX_ID hold= 8388608 used= 4161536 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4161536 used= 4159488 count= 2 avg_used= 2079744 mod=SUMMARY [2025-08-14 09:06:07.740781] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 500 ctx_id= RPC_CTX_ID hold= 3104768 used= 1507328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 58 washed_blocks= 71 washed_size= 118530048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1507328 used= 995584 count= 62 avg_used= 16057 mod=SUMMARY [2025-08-14 09:06:07.740798] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 500 ctx_id= PKT_NIO hold= 37584896 used= 36013568 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 50 washed_blocks= 71 washed_size= 99016704 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 36013568 used= 35017984 count= 125 avg_used= 280143 mod=SUMMARY [2025-08-14 09:06:07.740824] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 500 ctx_id= SCHEMA_SERVICE hold= 78925824 used= 68945536 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 22 washed_size= 5767168 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 68945536 used= 67682783 count= 4861 avg_used= 13923 mod=SUMMARY [2025-08-14 09:06:07.740841] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 500 ctx_id= UNEXPECTED_IN_500 hold= 98983936 used= 97219920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 97219920 used= 86819136 count= 3016 avg_used= 28786 mod=SUMMARY [2025-08-14 09:06:07.740859] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant: 508, limit: 1073741824 hold: 15613952 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 5128192 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10485760 limit= 9223372036854775807 [2025-08-14 09:06:07.740879] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=16] [MEMORY] tenant_id= 508 ctx_id= DEFAULT_CTX_ID hold= 5128192 used= 3765328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3765328 used= 3723912 count= 84 avg_used= 44332 mod=SUMMARY [2025-08-14 09:06:07.740899] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 508 ctx_id= CO_STACK hold= 10485760 used= 10321920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10321920 used= 10302720 count= 20 avg_used= 515136 mod=SUMMARY [2025-08-14 09:06:07.740927] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=328] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:07.740944] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant: 509, limit: 1073741824 hold: 15613952 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 5128192 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10485760 limit= 9223372036854775807 [2025-08-14 09:06:07.740961] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 509 ctx_id= DEFAULT_CTX_ID hold= 5128192 used= 3765328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3765328 used= 3723912 count= 84 avg_used= 44332 mod=SUMMARY [2025-08-14 09:06:07.740992] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=14] [MEMORY] tenant_id= 509 ctx_id= CO_STACK hold= 10485760 used= 10321920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10321920 used= 10302720 count= 20 avg_used= 515136 mod=SUMMARY [2025-08-14 09:06:07.741040] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant: 1001, limit: 1073741824 hold: 608747520 cache_hold: 44040192 cache_used: 44040192 cache_item_count: 21 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 302309376 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 54525952 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 3080192 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 483328 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 45514752 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 5521408 limit= 157286400 [MEMORY] ctx_id= GLIBC hold_bytes= 15499264 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 73424896 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35651584 limit= 214748360 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 3252224 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 14680064 limit= 1073741800 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 5570560 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3866624 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 1179648 limit= 9223372036854775807 [2025-08-14 09:06:07.741070] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1001 ctx_id= DEFAULT_CTX_ID hold= 302309376 used= 269204240 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 81 washed_blocks= 406 washed_size= 113410048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 269204240 used= 263072286 count= 3791 avg_used= 69393 mod=SUMMARY [2025-08-14 09:06:07.741083] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1001 ctx_id= MEMSTORE_CTX_ID hold= 54525952 used= 54099968 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 54099968 used= 54075216 count= 26 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:06:07.741093] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= EXECUTE_CTX_ID hold= 3080192 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 8 washed_size= 7405568 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:07.741104] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:06:07.741115] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= PLAN_CACHE_CTX_ID hold= 45514752 used= 36544128 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 77 washed_blocks= 201 washed_size= 132743168 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 36544128 used= 33001754 count= 3322 avg_used= 9934 mod=SUMMARY [2025-08-14 09:06:07.741125] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= WORK_AREA hold= 5521408 used= 0 limit= 157286400 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 9 washed_blocks= 40 washed_size= 17547264 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:07.741133] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= GLIBC hold= 15499264 used= 10889728 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 25 washed_blocks= 164 washed_size= 47415296 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10889728 used= 6020141 count= 18313 avg_used= 328 mod=SUMMARY [2025-08-14 09:06:07.741145] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= CO_STACK hold= 73424896 used= 71221248 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 32 washed_blocks= 34 washed_size= 2072576 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 71221248 used= 71088768 count= 138 avg_used= 515136 block_cnt= 138 chunk_cnt= 36 mod=CoStack [MEMORY] hold= 71221248 used= 71088768 count= 138 avg_used= 515136 mod=SUMMARY [2025-08-14 09:06:07.741164] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1001 ctx_id= KVSTORE_CACHE_ID hold= 44040192 used= 44040192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 44040192 used= 0 count= 21 avg_used= 0 mod=SUMMARY [2025-08-14 09:06:07.741178] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1001 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 33906624 limit= 214748360 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33906624 used= 30629648 count= 468 avg_used= 65447 mod=SUMMARY [2025-08-14 09:06:07.741189] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= TX_CALLBACK_CTX_ID hold= 3252224 used= 2342592 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 3 washed_blocks= 8 washed_size= 3039232 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2342592 used= 2269696 count= 286 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:07.741197] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:07.741206] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= RPC_CTX_ID hold= 14680064 used= 4202496 limit= 1073741800 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4202496 used= 2779776 count= 171 avg_used= 16256 mod=SUMMARY [2025-08-14 09:06:07.741223] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1001 ctx_id= TX_DATA_TABLE hold= 5570560 used= 2342784 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 8 washed_size= 2818048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2342784 used= 2319424 count= 30 avg_used= 77314 mod=SUMMARY [2025-08-14 09:06:07.741245] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=17] [MEMORY] tenant_id= 1001 ctx_id= MDS_DATA_ID hold= 3866624 used= 2785280 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 5 washed_size= 4521984 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2785280 used= 2698240 count= 340 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:07.741263] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= MERGE_NORMAL_CTX_ID hold= 1179648 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 8 washed_blocks= 24 washed_size= 15597568 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:07.741287] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant: 1002, limit: 8589934592 hold: 1945841664 cache_hold: 48234496 cache_used: 48234496 cache_item_count: 23 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 1009610752 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 415236096 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 1114112 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 950272 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 211812352 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 1441792 limit= 429496725 [MEMORY] ctx_id= GLIBC hold_bytes= 12582912 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 149913600 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 37748736 limit= 1717986900 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 7766016 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 5783552 limit= 8589934500 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 5718016 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3047424 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 1032192 limit= 9223372036854775807 [2025-08-14 09:06:07.741319] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1002 ctx_id= DEFAULT_CTX_ID hold= 1009610752 used= 918313856 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 14 [MEMORY] hold= 185860096 used= 185534336 count= 114 avg_used= 1627494 block_cnt= 114 chunk_cnt= 104 mod=MysqlRequesReco [MEMORY] hold= 167792640 used= 167772192 count= 1 avg_used= 167772192 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 83927040 used= 83886080 count= 2 avg_used= 41943040 block_cnt= 2 chunk_cnt= 2 mod=LogGroupBuffer [MEMORY] hold= 67190784 used= 67109344 count= 4 avg_used= 16777336 block_cnt= 4 chunk_cnt= 4 mod=MrgMemPoolChk [MEMORY] hold= 918313856 used= 901075310 count= 9500 avg_used= 94850 mod=SUMMARY [2025-08-14 09:06:07.741328] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEBB-0-0] [lt=161789697] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:06:07.741343] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1002 ctx_id= MEMSTORE_CTX_ID hold= 415236096 used= 411992064 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 411992064 used= 411803568 count= 198 avg_used= 2079816 block_cnt= 198 chunk_cnt= 198 mod=Memstore [MEMORY] hold= 411992064 used= 411803568 count= 198 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:06:07.741358] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEBB-0-0] [lt=27][errcode=0] cost too much time:mds_tenant_service.cpp:static int oceanbase::storage::mds::ObTenantMdsService::for_each_ls_in_tenant(const ObFunction &), (this=|threshold=3.00s|start at 09:03:22.601|207=0us|211=2us|218=1us|211=165.13s|218=1us|211=12.04ms|total=165.14s) [2025-08-14 09:06:07.741368] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=23] [MEMORY] tenant_id= 1002 ctx_id= EXECUTE_CTX_ID hold= 1114112 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 7 washed_size= 7274496 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:07.741394] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=22] [MEMORY] tenant_id= 1002 ctx_id= TRANS_CTX_MGR_ID hold= 950272 used= 933888 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1146880 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 933888 used= 921984 count= 2 avg_used= 460992 mod=SUMMARY [2025-08-14 09:06:07.741409] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1002 ctx_id= PLAN_CACHE_CTX_ID hold= 211812352 used= 201995008 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 176930880 used= 160228962 count= 13450 avg_used= 11912 block_cnt= 13450 chunk_cnt= 101 mod=SqlPhyPlan [MEMORY] hold= 201995008 used= 183048143 count= 17906 avg_used= 10222 mod=SUMMARY [2025-08-14 09:06:07.741420] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1002 ctx_id= WORK_AREA hold= 1441792 used= 0 limit= 429496725 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 14 washed_blocks= 54 washed_size= 27918336 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:07.741434] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1002 ctx_id= GLIBC hold= 12582912 used= 11425280 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 11425280 used= 6322556 count= 19216 avg_used= 329 mod=SUMMARY [2025-08-14 09:06:07.741449] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1002 ctx_id= CO_STACK hold= 149913600 used= 147087360 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 68 washed_blocks= 70 washed_size= 3178496 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 147087360 used= 146813760 count= 285 avg_used= 515136 block_cnt= 285 chunk_cnt= 73 mod=CoStack [MEMORY] hold= 147087360 used= 146813760 count= 285 avg_used= 515136 mod=SUMMARY [2025-08-14 09:06:07.741477] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1002 ctx_id= KVSTORE_CACHE_ID hold= 48234496 used= 48234496 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 48234496 used= 0 count= 23 avg_used= 0 mod=SUMMARY [2025-08-14 09:06:07.741491] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1002 ctx_id= META_OBJ_CTX_ID hold= 37748736 used= 35741632 limit= 1717986900 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 35741632 used= 32457488 count= 496 avg_used= 65438 mod=SUMMARY [2025-08-14 09:06:07.741505] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1002 ctx_id= TX_CALLBACK_CTX_ID hold= 7766016 used= 6934400 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 17 washed_size= 6914048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 6934400 used= 6729728 count= 848 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:07.741526] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1002 ctx_id= LOB_CTX_ID hold= 147456 used= 16384 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 16384 used= 15872 count= 2 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:07.741542] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1002 ctx_id= RPC_CTX_ID hold= 5783552 used= 5029888 limit= 8589934500 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 10 washed_blocks= 10 washed_size= 15187968 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 5029888 used= 3315456 count= 210 avg_used= 15787 mod=SUMMARY [2025-08-14 09:06:07.741559] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1002 ctx_id= TX_DATA_TABLE hold= 5718016 used= 4521856 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 9 washed_blocks= 16 washed_size= 13156352 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4521856 used= 4480128 count= 40 avg_used= 112003 mod=SUMMARY [2025-08-14 09:06:07.741575] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1002 ctx_id= MDS_DATA_ID hold= 3047424 used= 1998848 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 4 washed_size= 1146880 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1998848 used= 1936384 count= 244 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:07.741588] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1002 ctx_id= MDS_CTX_ID hold= 147456 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:07.741602] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1002 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:07.741615] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1002 ctx_id= MERGE_NORMAL_CTX_ID hold= 1032192 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 12 washed_size= 13647872 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:07.725299] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=900023][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:06.823|207=901.39ms|210=0us|222=2us|total=917.86ms) [2025-08-14 09:06:07.741773] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=16448][errcode=-4389] timer task use too much time(end=1755162367741772, start=1755162366823900, delta=917872) [2025-08-14 09:06:07.725571] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=461233][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:07.741908] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16309][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:07.741927] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:07.742064] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:07.742279] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=202][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:07.742291] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:07.742306] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:07.742323] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:07.742334] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:07.742349] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:07.742359] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:07.742371] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:07.742416] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4016] thread maybe hung!(point=[thread id=763, timeout ts=09:03:52.601, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:22.601], idx=763, next_allow_print_ts="09:04:22.601") [2025-08-14 09:06:07.742427] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:07.742444] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:07.743415] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7025] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:07.744699] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=352] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:07.745318] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=948] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:07.746437] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=18] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:07.747547] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=660] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:07.742455] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:07.749409] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:07.749451] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:07.749473] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891991430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ce10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:07.755271] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1955] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:07.756286] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:07.742999] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=494] temporary file module use no blocks [2025-08-14 09:06:07.757100] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14076] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:07.757141] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=40] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:07.757156] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623269182169, flush_io_finished_ret_:0, flush_io_finished_round_:26426, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:07.757342] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:07.757893] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=504][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:07.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.726326] WDIAG [SQL.DAS] write_tablet (ob_dml_service.h:406) [338][Occam][T1][YB42AC4D0007-00063C4960499C96-0-0] [lt=77][errcode=-4038] fail to get_write_store_ctx_guard(ret=-4038, ls_id_={id:1}) [2025-08-14 09:06:07.759123] WDIAG [SQL.DAS] insert_rows (ob_das_insert_op.cpp:136) [338][Occam][T1][YB42AC4D0007-00063C4960499C96-0-0] [lt=32740][errcode=-4038] insert rows to access service failed(ret=-4038) [2025-08-14 09:06:07.760889] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=276][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73940b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:07.760924] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2598] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:07.760977] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:07.760711] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEBB-0-0] [lt=21][errcode=0] cost too much time:mds_tenant_service.cpp:void oceanbase::storage::mds::ObTenantMdsTimer::dump_special_mds_table_status_task(), (this=|threshold=1.00s|start at 09:03:22.601|total=165.16s) [2025-08-14 09:06:07.761001] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEBB-0-0] [lt=287][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571630c0, caller:mds_tenant_service.cpp:mtl_start:117, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_145, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571633a0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:0, time_interval:15.00s, expected_run_time:09:06:22.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.761012] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:07.761021] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEBB-0-0] [lt=18][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:06:07.761|total=165.16s) [2025-08-14 09:06:07.761027] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:07.761045] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:07.761190] WDIAG [SQL.DAS] insert_rows (ob_das_insert_op.cpp:136) [337][EvtHisUpdTask][T1][YB42AC4D0007-00063C4960299C79-0-0] [lt=34821][errcode=-4038] insert rows to access service failed(ret=-4038) [2025-08-14 09:06:07.762262] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=386] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:07.762389] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=214] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:07.762422] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=290] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:07.762455] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=20] temporary file module use no blocks [2025-08-14 09:06:07.762468] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:07.762486] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=17] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:07.762503] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=17] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623269201546, flush_io_finished_ret_:0, flush_io_finished_round_:26386, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:03:19.722496] EDIAG [SQL] open_plan (ob_result_set.cpp:104) [813][T1002_DupTbLeas][T1001][YB42AC4D0007-00063C496039A468-0-0] [lt=1982870][errcode=-4012] LOGGER COST TOO MUCH TIME, cost: 6029159, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=7, ALLOC_BLOCK#end=98, FORMAT_END=6029054, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x14943b08 0x9337072 0x932e8ab 0x96ac524 0x96a672d 0x96a5033 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x1ee93940 0x1ee93063 0x1d53bd1b 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:07.761040] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEBB-0-0] [lt=12][errcode=-4016] task executed too delay(delay_time=165159882, *p_task_={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:0, time_interval:3.00s, expected_run_time:09:06:10.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.765811] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=29] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:07.747629] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=5162][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:07.768564] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20898][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:07.768585] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:07.768752] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:07.768772] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:07.769582] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eeb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:07.769617] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73936b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eec90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:07.769643] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:07.769749] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:07.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.769775] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:07.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.777560] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=16] gc stale ls task succ [2025-08-14 09:06:07.777919] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:07.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.779644] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:07.779683] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72765b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:07.779698] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:07.779711] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:07.779725] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:07.781975] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:07.782366] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:07.782792] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:07.783293] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:07.789704] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aab10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.774, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:07.789736] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739ef90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:07.789747] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a45b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:07.789761] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eee10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:07.789786] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73931b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:07.799765] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:07.799797] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:07.799809] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7273c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:07.799819] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d27b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:07.799843] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d94b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:07.799856] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.780, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:07.799869] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739bbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.780, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:07.799882] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fa10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:07.799894] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f727dc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f729ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:07.799907] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f727d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f729af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:07.799920] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:07.806196] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=30] PNIO [ratelimit] time: 1755162367806193, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007639, add_bytes: 0 [2025-08-14 09:06:07.808125] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=52][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:07.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.809850] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891814fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.793, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:07.809882] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:07.812472] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=14] gc stale ls task succ [2025-08-14 09:06:07.819895] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7facb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:07.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:07.825072] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=35] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:06:07.825089] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:07.828727] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1094][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:07.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.840153] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:07.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.841851] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:07.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.857894] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=290][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fc8b0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcb90, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:1, time_interval:3.00s, expected_run_time:09:06:07.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:07.861252] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:07.861311] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:07.861350] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:07.861387] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:07.861422] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:07.861433] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:07.925093] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [276]OB_MALLOC COST TOO MUCH TIME, cost_time=761483, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 12, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=761430, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=726266, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=50, WASH_BLOCK_END=726216], seq:[0, 1, 2] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=727836, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=49, WASH_BLOCK_END=727787], seq:[0, 1, 2] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=458855, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=53, WASH_BLOCK_END=458802], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=716518, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=49, WASH_BLOCK_END=716469], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=787231, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=46, WASH_BLOCK_END=787185], seq:[0, 1, 2] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=698236, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=57, WASH_BLOCK_END=698179], seq:[0, 1, 2] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=595537, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=68, WASH_BLOCK_END=595469], seq:[0, 1, 2] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=702708, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=55, WASH_BLOCK_END=702653], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=812273, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=65, WASH_BLOCK_END=798749, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=13458], seq:[0, 1, 2, 3, 4] [2025-08-14 09:06:07.925210] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=19] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:08.769001] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=1058] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623270196950, flush_io_finished_ret_:0, flush_io_finished_round_:26427, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:08.769678] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][YB42AC4D0007-00063C495F8A1971-0-0] [lt=15][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a7ad8, timer:0x7f9935f59f10, task:0x7f9935f6c710, task_type:"N9oceanbase5table22OBTTLTimerPeriodicTaskE", scheduled_time:1755162341719199, pushed_time:1755162341719247, delay:10000000}, delay_in_thread_pool=25061335, threashold=500000) [2025-08-14 09:06:08.769736] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][YB42AC4D0007-00063C495F8A1972-0-0] [lt=24][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a79f8, timer:0x7f99535c95e0, task:0x7f9935f6c340, task_type:"N9oceanbase5table22OBTTLTimerPeriodicTaskE", scheduled_time:1755162341719219, pushed_time:1755162341719252, delay:10000000}, delay_in_thread_pool=27050472, threashold=500000) [658]OB_MALLOC COST TOO MUCH TIME, cost_time=826647, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=46, WASH_BLOCK_END=785176, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=41419], seq:[0, 1, 2, 3, 4] [2025-08-14 09:06:07.941916] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:07.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:08.769853] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=826683][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:07.941|207=827.94ms|210=0us|222=1us|total=827.95ms) [2025-08-14 09:06:08.769873] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task use too much time(end=1755162368769873, start=1755162367941914, delta=827959) [2025-08-14 09:06:08.729043] WDIAG [SQL.DAS] open_op (ob_das_insert_op.cpp:111) [338][Occam][T1][YB42AC4D0007-00063C4960499C96-0-0] [lt=10][errcode=-4038] fail to do insert(ret=-4038) [2025-08-14 09:06:08.770748] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18569][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:08.770771] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:08.770783] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:08.770799] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:08.770809] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:08.770821] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:08.767920] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=1041626] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:06:08.777623] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=21] gc stale ls task succ [660]OB_MALLOC COST TOO MUCH TIME, cost_time=911937, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=49, WASH_BLOCK_END=911888], seq:[0, 1, 2] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=850669, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=35, WASH_BLOCK_END=850634], seq:[0, 1, 2] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=775783, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=42, WASH_BLOCK_END=775741], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=875160, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=50, WASH_BLOCK_END=875109], seq:[0, 1, 2] [806]OB_MALLOC COST TOO MUCH TIME, cost_time=953973, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=953971], seq:[0] [806]OB_MALLOC COST TOO MUCH TIME, cost_time=954101, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=68, WASH_BLOCK_END=38, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=953989], seq:[0, 1, 2, 3, 4] [2025-08-14 09:03:00.720074] EDIAG [STORAGE] init (ob_tablet_medium_info_reader.cpp:50) [806][T1002_TimerWK2_SSTableGC][T1002][YB42AC4D0007-00063C495F3A18A1-0-0] [lt=75899879][errcode=-4012] LOGGER COST TOO MUCH TIME, cost: 188720014, owner: ObLog, click_count: 7, time dist:[FORMAT_END=29970553, ObMalloc#start=2, ObMalloc#end=72, ObMalloc#start=157795268, ObMalloc#end=954111, ALLOC_END=2, APPEND_END=6], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1bbc1724 0x975abe5 0x975a314 0x975972e 0x9758a1f 0x1aee8055 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:03:00.720074] WDIAG [STORAGE] init (ob_tablet_medium_info_reader.cpp:50) [806][T1002_TimerWK2_SSTableGC][T1002][YB42AC4D0007-00063C495F3A18A1-0-0] [lt=75899879][errcode=-4012] fail to do build query range iter(ret=-4012, ls_id={id:1}, tablet_id={id:101047}, scan_param={tablet_id:{id:101047}, ls_id:{id:1}, column_ids:[16, 17, 18, 19], index_id:88888, key_ranges:[{"range":"table_id:88888,group_idx:0,index_ordered_idx:0,({"TINYINT":3},MIN;{"TINYINT":3},MAX)"}], ss_key_ranges:[], range_array_pos:[], timeout:1755161750753477, scan_flag:{scan_order:0, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_bare_row_scan:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:1, is_select_follower:0, enable_lob_prefetch:0, is_bare_row_scan:0, mr_mv_scan:0, is_plain_insert_gts_opt:0, reserved:0}, sql_mode:134217760, reserved_cell_count:4, schema_version:9527, query_begin_schema_version:-1, limit_offset:{offset_:0, limit_:-1}, for_update:false, wait:-1, frozen_version:9223372036854775807, is_get:false, output_exprs:NULL, op_filters:NULL, trans_desc:NULL, snapshot:{this:0x7f99013cf150, valid:true, source:2, core:{version:{val:4611686018427387903, v:0}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, table_param:{table_id:88888, output_projector:[0, 1, 2, 3], aggregate_projector:[], group_by_projector:[], output_sel_mask:[true, true, true, true], pad_col_projector:[], read_param_version:1, main_read_info:{schema_column_count:4, schema_rowkey_cnt:2, rowkey_cnt:4, trans_col_index:-1, mview_old_new_col_index:-1, group_idx_col_index:-1, seq_read_column_count:2, max_col_index:5, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 4, 5], data:0x7f9898766740, count:4, allocator:0x7f99013cf458, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 3], data:0x7f9898766760, count:4, allocator:0x7f99013cf458, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=16 {type:"TINYINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"CHAR", collation:"binary", coercibility:"INVALID"} order=0, column_id=18 {type:"CHAR", collation:"binary", coercibility:"INVALID"} order=0, column_id=19 {type:"CHAR", collation:"binary", coercibility:"INVALID"} order=0], data:0x7f9898766700, count:4, allocator:0x7f99013cf458, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:4, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}], data:0x7f98987667b0, count:4, allocator:0x7f99013cf458, capacity:4, init_cnt:4}, has_all_column_group:true, need_truncate_filter:false}, use_lob_locator:false, rowid_version:0, rowid_projector:[], enable_lob_locator_v2:false, is_fts_index:false, parser_name:"", parser_properties:"", is_vec_index:false, is_column_replica_table:false, is_normal_cgs_at_the_end:false, is_mlog_table:false}, sample_info:{method:0, percent:"1.000000000000000000e+02", seed:-1, table_id:0, scope:0, force_block:false}, need_scn:false, need_switch_param:false, is_mds_query:true, fb_read_tx_uncommitted:false, external_file_format:{"TYPE":"INVALID"}, external_file_location:"", tx_seq_base:-1, auto_split_filter_type:18446744073709551615, is_tablet_spliting:false, need_update_tablet_param:false, mds_collector:{exist_uncommitted_node:false, exist_new_committed_node:false}}) [2025-08-14 09:06:08.728225] WDIAG [OCCAM] dump_statistics (ob_vtable_event_recycle_buffer.h:144) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEBC-0-0] [lt=2604][errcode=-4006] not init(ret=-4006) [2025-08-14 09:06:09.726119] INFO [MDS] dump_map_holding_item (mds_tenant_service.cpp:462) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEBC-0-0] [lt=996374] finish scan map holding items(scan_cnt=0) [2025-08-14 09:06:08.783673] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [807][T1002_TimerWK3][T1002][YB42AC4D0007-00063C495F8A1974-0-0] [lt=11][errcode=-4389] timer task too much delay in priority queue(token={this:0x7f991b7a4c78, timer:0x7f99329e1b10, task:0x7f98e4e091f8, task_type:"N9oceanbase5table21ObTTLAllTaskSchedulerE", scheduled_time:1755162348769470, pushed_time:1755162350725350, delay:15000000}, delay_in_priority_queue=1955880, threashold=1000000) [2025-08-14 09:06:09.727423] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][YB42AC4D0007-00063C495F8A1974-0-0] [lt=942323][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a4c78, timer:0x7f99329e1b10, task:0x7f98e4e091f8, task_type:"N9oceanbase5table21ObTTLAllTaskSchedulerE", scheduled_time:1755162348769470, pushed_time:1755162350725350, delay:15000000}, delay_in_thread_pool=18044405, threashold=500000) [2025-08-14 09:06:09.728459] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=981603][errcode=0] epoll handler too much time :983097 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:08.743872] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=964337] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:06:09.734264] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:981) [373][Occam][T1][YB42AC4D0007-00063C4960599C9D-0-0] [lt=10][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1755162162745681), 'ELECTION', 'decentralized to be leader', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'server|lease:["172.77.0.7:2882"|2025-08-14 08:44:58.850]', 'HAPPENED_TIME', '2025-08-14 08:44:55.757588', '', '', '172.77.0.7', 2882)"}, record_ret=-4012, ret=-4012) [374]OB_MALLOC COST TOO MUCH TIME, cost_time=948053, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=45, WASH_BLOCK_END=932740, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=15264], seq:[0, 1, 2, 3, 4] [2025-08-14 09:06:08.785348] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=310][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eca90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:08.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.734885] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=948097][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:08.785|207=949.53ms|210=0us|222=2us|total=949.55ms) [2025-08-14 09:06:09.734907] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4389] timer task use too much time(end=1755162369734907, start=1755162368785346, delta=949561) [2025-08-14 09:06:09.734918] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=964300, expected_run_ts_us_=1755162368770617, ObClockGenerator::getRealClock()=1755162369734918, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e94b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:08.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.734937] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e94b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:08.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.734951] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=964165, expected_run_ts_us_=1755162368770786, ObClockGenerator::getRealClock()=1755162369734951, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:08.770, task_priority:1, with_handle_protected:0}) [419]OB_MALLOC COST TOO MUCH TIME, cost_time=768338, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=34, WASH_BLOCK_END=768303], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=721827, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=43, WASH_BLOCK_END=721784], seq:[0, 1, 2] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=596909, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=45, WASH_BLOCK_END=596864], seq:[0, 1, 2] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=373792, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=60, WASH_BLOCK_END=373732], seq:[0, 1, 2] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=926943, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=42, WASH_BLOCK_END=926899], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=925802, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=42, WASH_BLOCK_END=925760], seq:[0, 1, 2] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=896742, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=48, WASH_BLOCK_END=896694], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=893990, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=49, WASH_BLOCK_END=893941], seq:[0, 1, 2] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=740344, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=53, WASH_BLOCK_END=740291], seq:[0, 1, 2] [2025-08-14 09:06:09.724745] WDIAG [SQL.DAS] start_das_task (ob_das_task.cpp:353) [338][Occam][T1][YB42AC4D0007-00063C4960499C96-0-0] [lt=41602][errcode=-4038] open das task op failed(ret=-4038) [2025-08-14 09:03:35.741771] EDIAG [ARCHIVE] produce_log_fetch_task_ (ob_archive_sequencer.cpp:166) [859][T1002_ArcSeq][T1002][YB42AC4D0007-00063C495E299C16-0-0] [lt=19545][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3995836, owner: ObLog, click_count: 6, time dist:[FORMAT_END=3973968, ObMalloc#start=1, ObMalloc#end=21398, ObMalloc#start=19, ObMalloc#end=54, FREE_END=396], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x113df1a7 0x9ac5c05 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:09.755189] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=2084][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728d430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7290f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:08.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.755418] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=627] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:09.755223] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] task scheduled too delay(scheduled_delay=959459, expected_run_ts_us_=1755162368795764, ObClockGenerator::getRealClock()=1755162369755223, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:08.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.755440] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=215][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:08.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.755454] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=959547, expected_run_ts_us_=1755162368795906, ObClockGenerator::getRealClock()=1755162369755453, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73922b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:08.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.755463] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73922b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:08.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.755479] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] task scheduled too delay(scheduled_delay=959443, expected_run_ts_us_=1755162368796036, ObClockGenerator::getRealClock()=1755162369755479, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:08.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.755491] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:08.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.755503] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=956367, expected_run_ts_us_=1755162368799136, ObClockGenerator::getRealClock()=1755162369755503, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:08.799, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.755515] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:08.799, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.755530] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] task scheduled too delay(scheduled_delay=962595, expected_run_ts_us_=1755162368792934, ObClockGenerator::getRealClock()=1755162369755530, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:08.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.755541] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:08.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.755554] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=946343, expected_run_ts_us_=1755162368809210, ObClockGenerator::getRealClock()=1755162369755553, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:08.809, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.755565] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:08.809, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.755577] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=950716, expected_run_ts_us_=1755162368804860, ObClockGenerator::getRealClock()=1755162369755576, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eaa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:08.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.755589] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eaa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:08.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.755602] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=918062, expected_run_ts_us_=1755162368837539, ObClockGenerator::getRealClock()=1755162369755602, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73945b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:08.837, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.755613] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73945b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:08.837, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.755626] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=843857, expected_run_ts_us_=1755162368911768, ObClockGenerator::getRealClock()=1755162369755626, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:08.911, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.755637] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:08.911, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.755661] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.718, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.755674] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.726, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.755686] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a67b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7397290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.728, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.755698] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919d9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.755709] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7277c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eac90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.757125] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=367] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:09.749053] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [813][T1002_DupTbLeas][T1001][YB42AC4D0007-00063C496039A468-0-0] [lt=8634][errcode=-4012] open result set failed(ret=-4012) [2025-08-14 09:06:09.757465] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:801) [813][T1002_DupTbLeas][T1001][YB42AC4D0007-00063C496039A468-0-0] [lt=8395][errcode=-4012] result set open failed(ret=-4012, executor={ObIExecutor:, sql:"select * from __all_ls_status where tenant_id = 1002 and flag like "%DUPLICATE%" order by ls_id limit 1"}) [2025-08-14 09:06:09.758466] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=182] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:09.758563] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.758587] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:08.720836] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5930][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1039272, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:08.720836] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5930] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:09.761727] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1039272][errcode=0] epoll handler too much time :1040907 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:09.761747] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:09.762137] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=496] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:09.763960] INFO handle (memory_dump.cpp:671) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=253] statistics: tenant_cnt: 6, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 168 270 0 1 2 110 1 0 1 3 7 0 0 1 4 1 1 0 1 5 85 6 0 1 6 13 0 0 1 7 22 2 0 1 8 53 1 0 1 12 17 3 0 1 13 10 1 0 1 14 1 1 0 1 15 1 1 0 1 16 8 7 0 1 18 4 3 0 1 20 2 1 0 1 22 1 1 0 1 24 1 0 0 1 25 6 0 0 500 0 560 275 0 500 7 192 5 0 500 8 41 1 0 500 9 2 1 0 500 10 2 1 0 500 16 58 4 0 500 17 60 7 0 500 22 38 64 0 500 23 27 10 0 508 0 2 7 0 508 8 5 1 0 509 0 2 7 0 509 8 5 1 0 1001 0 147 252 0 1001 2 26 1 0 1001 3 5 0 0 1001 4 1 1 0 1001 5 85 6 0 1001 6 11 0 0 1001 7 30 2 0 1001 8 36 1 0 1001 12 17 3 0 1001 13 3 1 0 1001 14 1 1 0 1001 16 7 3 0 1001 18 4 3 0 1001 20 4 1 0 1001 25 8 0 0 1002 0 263 279 0 1002 2 198 1 0 1002 3 4 0 0 1002 4 1 1 0 1002 5 101 6 0 1002 6 14 0 0 1002 7 6 2 0 1002 8 73 1 0 1002 12 18 3 0 1002 13 7 1 0 1002 14 1 2 0 1002 16 10 7 0 1002 18 9 3 0 1002 20 2 1 0 1002 21 1 0 0 1002 24 1 0 0 1002 25 7 0 0 cost_time: 1018141 [2025-08-14 09:06:09.764011] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=27] [MEMORY] tenant: 1, limit: 2147483648 hold: 1013174272 cache_hold: 56623104 cache_used: 56623104 cache_item_count: 27 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 444993536 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 230686720 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 1163264 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 483328 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 51617792 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 1458176 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 14204928 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 111149056 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35651584 limit= 429496720 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 8003584 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= PS_CACHE_CTX_ID hold_bytes= 65536 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 14827520 limit= 9223372036854775807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 3481600 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3964928 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 212992 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 884736 limit= 9223372036854775807 [2025-08-14 09:06:09.764049] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 444993536 used= 389771296 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 70 washed_blocks= 287 washed_size= 70221824 [MEMORY] request_cached_chunk_cnt= 6 [MEMORY] hold= 389771296 used= 379518723 count= 32899 avg_used= 11535 mod=SUMMARY [2025-08-14 09:06:09.764078] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=18] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 230686720 used= 228884480 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 block_cnt= 110 chunk_cnt= 110 mod=Memstore [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:06:09.764093] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= EXECUTE_CTX_ID hold= 1163264 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 15 washed_size= 13516800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:09.764109] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:06:09.764124] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 51617792 used= 47452416 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 84 washed_blocks= 334 washed_size= 126640128 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 47452416 used= 43070492 count= 3992 avg_used= 10789 mod=SUMMARY [2025-08-14 09:06:09.764139] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= WORK_AREA hold= 1458176 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 13 washed_blocks= 66 washed_size= 25804800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:09.764150] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 14204928 used= 10970224 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 277 washed_size= 31932416 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10970224 used= 6078924 count= 18573 avg_used= 327 mod=SUMMARY [2025-08-14 09:06:09.764163] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 111149056 used= 108896256 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 block_cnt= 211 chunk_cnt= 53 mod=CoStack [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 mod=SUMMARY [2025-08-14 09:06:09.764188] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 56623104 used= 56623104 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 56623104 used= 0 count= 27 avg_used= 0 mod=SUMMARY [2025-08-14 09:06:09.764207] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 34365376 limit= 429496720 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 34365376 used= 31086608 count= 475 avg_used= 65445 mod=SUMMARY [2025-08-14 09:06:09.764220] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 8003584 used= 4947392 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 10 washed_blocks= 23 washed_size= 12967936 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4947392 used= 4793344 count= 604 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:09.764243] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:09.764258] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 65536 used= 31024 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 2031616 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 31024 used= 27058 count= 20 avg_used= 1352 mod=SUMMARY [2025-08-14 09:06:09.764272] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= RPC_CTX_ID hold= 14827520 used= 6135808 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 6135808 used= 4051328 count= 253 avg_used= 16013 mod=SUMMARY [2025-08-14 09:06:09.764289] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 3481600 used= 2391232 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 10 washed_size= 4907008 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2391232 used= 2367040 count= 36 avg_used= 65751 mod=SUMMARY [2025-08-14 09:06:09.764314] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 3964928 used= 3080192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 3 washed_size= 229376 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3080192 used= 2983936 count= 376 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:09.764330] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= SCHEMA_SERVICE hold= 212992 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1884160 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:09.764346] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:09.764359] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= MERGE_NORMAL_CTX_ID hold= 884736 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 6 washed_blocks= 9 washed_size= 11698176 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:09.764375] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant: 500, limit: 9223372036854775807 hold: 694624256 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 343265280 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 37339136 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 85983232 limit= 9223372036854775807 [MEMORY] ctx_id= LIBEASY hold_bytes= 3145728 limit= 9223372036854775807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 6291456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 3104768 limit= 9223372036854775807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 37584896 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 78925824 limit= 9223372036854775807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 98983936 limit= 9223372036854775807 [2025-08-14 09:06:09.764404] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 343265280 used= 301605936 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 493 washed_blocks= 2219 washed_size= 910721024 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 301605936 used= 294175621 count= 12708 avg_used= 23148 mod=SUMMARY [2025-08-14 09:06:09.764430] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= GLIBC hold= 37339136 used= 21975792 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 187 washed_blocks= 810 washed_size= 369508352 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 21975792 used= 10371253 count= 131704 avg_used= 78 mod=SUMMARY [2025-08-14 09:06:09.764446] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 500 ctx_id= CO_STACK hold= 85983232 used= 83091456 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 83091456 used= 82936896 count= 161 avg_used= 515136 block_cnt= 161 chunk_cnt= 41 mod=CoStack [MEMORY] hold= 83091456 used= 82936896 count= 161 avg_used= 515136 mod=SUMMARY [2025-08-14 09:06:09.764459] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= LIBEASY hold= 3145728 used= 3070816 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1048576 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3070816 used= 2859784 count= 223 avg_used= 12824 mod=SUMMARY [2025-08-14 09:06:09.764473] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= LOGGER_CTX_ID hold= 6291456 used= 4161536 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4161536 used= 4159488 count= 2 avg_used= 2079744 mod=SUMMARY [2025-08-14 09:06:09.764496] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 500 ctx_id= RPC_CTX_ID hold= 3104768 used= 1507328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 58 washed_blocks= 71 washed_size= 118530048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1507328 used= 995584 count= 62 avg_used= 16057 mod=SUMMARY [2025-08-14 09:06:09.764509] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= PKT_NIO hold= 37584896 used= 36013568 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 50 washed_blocks= 71 washed_size= 99016704 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 36013568 used= 35017984 count= 125 avg_used= 280143 mod=SUMMARY [2025-08-14 09:06:09.764538] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 500 ctx_id= SCHEMA_SERVICE hold= 78925824 used= 68945536 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 22 washed_size= 5767168 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 68945536 used= 67682783 count= 4861 avg_used= 13923 mod=SUMMARY [2025-08-14 09:06:09.764552] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= UNEXPECTED_IN_500 hold= 98983936 used= 97219920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 97219920 used= 86819136 count= 3016 avg_used= 28786 mod=SUMMARY [2025-08-14 09:06:09.764566] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant: 508, limit: 1073741824 hold: 15613952 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 5128192 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10485760 limit= 9223372036854775807 [2025-08-14 09:06:09.764577] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 508 ctx_id= DEFAULT_CTX_ID hold= 5128192 used= 3765328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3765328 used= 3723912 count= 84 avg_used= 44332 mod=SUMMARY [2025-08-14 09:06:09.764600] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 508 ctx_id= CO_STACK hold= 10485760 used= 10321920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10321920 used= 10302720 count= 20 avg_used= 515136 mod=SUMMARY [2025-08-14 09:06:09.764642] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 509, limit: 1073741824 hold: 15613952 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 5128192 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10485760 limit= 9223372036854775807 [2025-08-14 09:06:09.764653] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 509 ctx_id= DEFAULT_CTX_ID hold= 5128192 used= 3765328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3765328 used= 3723912 count= 84 avg_used= 44332 mod=SUMMARY [2025-08-14 09:06:09.764676] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 509 ctx_id= CO_STACK hold= 10485760 used= 10321920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10321920 used= 10302720 count= 20 avg_used= 515136 mod=SUMMARY [2025-08-14 09:06:09.764727] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant: 1001, limit: 1073741824 hold: 608747520 cache_hold: 44040192 cache_used: 44040192 cache_item_count: 21 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 302309376 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 54525952 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 3080192 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 483328 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 45514752 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 5521408 limit= 157286400 [MEMORY] ctx_id= GLIBC hold_bytes= 15499264 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 73424896 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35651584 limit= 214748360 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 3252224 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 14680064 limit= 1073741800 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 5570560 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3866624 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 1179648 limit= 9223372036854775807 [2025-08-14 09:06:09.764752] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= DEFAULT_CTX_ID hold= 302309376 used= 269196048 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 81 washed_blocks= 406 washed_size= 113410048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 269196048 used= 263064350 count= 3790 avg_used= 69410 mod=SUMMARY [2025-08-14 09:06:09.764768] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1001 ctx_id= MEMSTORE_CTX_ID hold= 54525952 used= 54099968 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 54099968 used= 54075216 count= 26 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:06:09.764778] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= EXECUTE_CTX_ID hold= 3080192 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 8 washed_size= 7405568 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:09.764789] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:06:09.764799] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= PLAN_CACHE_CTX_ID hold= 45514752 used= 36544128 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 77 washed_blocks= 201 washed_size= 132743168 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 36544128 used= 33001754 count= 3322 avg_used= 9934 mod=SUMMARY [2025-08-14 09:06:09.764809] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= WORK_AREA hold= 5521408 used= 0 limit= 157286400 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 9 washed_blocks= 40 washed_size= 17547264 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:09.764819] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= GLIBC hold= 15499264 used= 10889728 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 25 washed_blocks= 164 washed_size= 47415296 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10889728 used= 6020141 count= 18313 avg_used= 328 mod=SUMMARY [2025-08-14 09:06:09.764830] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= CO_STACK hold= 73424896 used= 71221248 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 32 washed_blocks= 34 washed_size= 2072576 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 71221248 used= 71088768 count= 138 avg_used= 515136 block_cnt= 138 chunk_cnt= 36 mod=CoStack [MEMORY] hold= 71221248 used= 71088768 count= 138 avg_used= 515136 mod=SUMMARY [2025-08-14 09:06:09.764843] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1001 ctx_id= KVSTORE_CACHE_ID hold= 44040192 used= 44040192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 44040192 used= 0 count= 21 avg_used= 0 mod=SUMMARY [2025-08-14 09:06:09.764851] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 33906624 limit= 214748360 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33906624 used= 30629648 count= 468 avg_used= 65447 mod=SUMMARY [2025-08-14 09:06:09.764859] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= TX_CALLBACK_CTX_ID hold= 3252224 used= 2342592 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 3 washed_blocks= 8 washed_size= 3039232 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2342592 used= 2269696 count= 286 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:09.764867] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1001 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:09.764877] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1001 ctx_id= RPC_CTX_ID hold= 14680064 used= 4202496 limit= 1073741800 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4202496 used= 2779776 count= 171 avg_used= 16256 mod=SUMMARY [2025-08-14 09:06:09.764887] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1001 ctx_id= TX_DATA_TABLE hold= 5570560 used= 2342784 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 8 washed_size= 2818048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2342784 used= 2319424 count= 30 avg_used= 77314 mod=SUMMARY [2025-08-14 09:06:09.764898] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= MDS_DATA_ID hold= 3866624 used= 2785280 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 5 washed_size= 4521984 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2785280 used= 2698240 count= 340 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:09.764913] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1001 ctx_id= MERGE_NORMAL_CTX_ID hold= 1179648 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 8 washed_blocks= 24 washed_size= 15597568 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:09.764930] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant: 1002, limit: 8589934592 hold: 1945841664 cache_hold: 48234496 cache_used: 48234496 cache_item_count: 23 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 1009610752 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 415236096 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 1114112 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 950272 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 211812352 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 1441792 limit= 429496725 [MEMORY] ctx_id= GLIBC hold_bytes= 12582912 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 149913600 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 37748736 limit= 1717986900 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 7766016 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 5783552 limit= 8589934500 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 5718016 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3047424 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 1032192 limit= 9223372036854775807 [2025-08-14 09:06:09.764963] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1002 ctx_id= DEFAULT_CTX_ID hold= 1009610752 used= 918313856 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 14 [MEMORY] hold= 185860096 used= 185534336 count= 114 avg_used= 1627494 block_cnt= 114 chunk_cnt= 104 mod=MysqlRequesReco [MEMORY] hold= 167792640 used= 167772192 count= 1 avg_used= 167772192 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 83927040 used= 83886080 count= 2 avg_used= 41943040 block_cnt= 2 chunk_cnt= 2 mod=LogGroupBuffer [MEMORY] hold= 67190784 used= 67109344 count= 4 avg_used= 16777336 block_cnt= 4 chunk_cnt= 4 mod=MrgMemPoolChk [MEMORY] hold= 918313856 used= 901075310 count= 9500 avg_used= 94850 mod=SUMMARY [2025-08-14 09:06:09.764988] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=20] [MEMORY] tenant_id= 1002 ctx_id= MEMSTORE_CTX_ID hold= 415236096 used= 411992064 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 411992064 used= 411803568 count= 198 avg_used= 2079816 block_cnt= 198 chunk_cnt= 198 mod=Memstore [MEMORY] hold= 411992064 used= 411803568 count= 198 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:06:09.764998] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= EXECUTE_CTX_ID hold= 1114112 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 7 washed_size= 7274496 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:09.765008] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= TRANS_CTX_MGR_ID hold= 950272 used= 933888 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1146880 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 933888 used= 921984 count= 2 avg_used= 460992 mod=SUMMARY [2025-08-14 09:06:09.765020] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= PLAN_CACHE_CTX_ID hold= 211812352 used= 201995008 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 176930880 used= 160228962 count= 13450 avg_used= 11912 block_cnt= 13450 chunk_cnt= 101 mod=SqlPhyPlan [MEMORY] hold= 201995008 used= 183048143 count= 17906 avg_used= 10222 mod=SUMMARY [2025-08-14 09:06:09.765030] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= WORK_AREA hold= 1441792 used= 0 limit= 429496725 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 14 washed_blocks= 54 washed_size= 27918336 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:09.765040] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= GLIBC hold= 12582912 used= 11425280 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 11425280 used= 6322556 count= 19216 avg_used= 329 mod=SUMMARY [2025-08-14 09:06:09.765051] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= CO_STACK hold= 149913600 used= 147087360 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 68 washed_blocks= 70 washed_size= 3178496 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 147087360 used= 146813760 count= 285 avg_used= 515136 block_cnt= 285 chunk_cnt= 73 mod=CoStack [MEMORY] hold= 147087360 used= 146813760 count= 285 avg_used= 515136 mod=SUMMARY [2025-08-14 09:06:09.765066] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= KVSTORE_CACHE_ID hold= 48234496 used= 48234496 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 48234496 used= 0 count= 23 avg_used= 0 mod=SUMMARY [2025-08-14 09:06:09.765076] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= META_OBJ_CTX_ID hold= 37748736 used= 35741632 limit= 1717986900 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 35741632 used= 32457488 count= 496 avg_used= 65438 mod=SUMMARY [2025-08-14 09:06:09.765086] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= TX_CALLBACK_CTX_ID hold= 7766016 used= 6934400 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 17 washed_size= 6914048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 6934400 used= 6729728 count= 848 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:09.765096] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= LOB_CTX_ID hold= 147456 used= 16384 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 16384 used= 15872 count= 2 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:09.765108] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= RPC_CTX_ID hold= 5783552 used= 5029888 limit= 8589934500 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 10 washed_blocks= 10 washed_size= 15187968 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 5029888 used= 3315456 count= 210 avg_used= 15787 mod=SUMMARY [2025-08-14 09:06:09.765120] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= TX_DATA_TABLE hold= 5718016 used= 4521856 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 9 washed_blocks= 16 washed_size= 13156352 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4521856 used= 4480128 count= 40 avg_used= 112003 mod=SUMMARY [2025-08-14 09:06:09.765132] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= MDS_DATA_ID hold= 3047424 used= 1998848 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 4 washed_size= 1146880 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1998848 used= 1936384 count= 244 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:09.765142] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= MDS_CTX_ID hold= 147456 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:09.765155] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:09.765164] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= MERGE_NORMAL_CTX_ID hold= 1032192 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 12 washed_size= 13647872 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:09.762571] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=990558] blacklist_loop exec finished(cost_time=16, is_enabled=true, send_cnt=0) [2025-08-14 09:06:09.767334] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=263][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:09.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:09.754728] WDIAG [SQL.DAS] do_local_das_task (ob_data_access_service.cpp:375) [338][Occam][T1][YB42AC4D0007-00063C4960499C96-0-0] [lt=29933][errcode=-4038] start local das task failed(ret=-4038) [2025-08-14 09:06:09.767491] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=310][errcode=-4389] timer task too much delay in priority queue(token={this:0x7f991b7a6098, timer:0x7f99200ff180, task:0x7f99200ff208, task_type:"N9oceanbase8observer18ObTableLoadService25ObClientTaskAutoAbortTaskE", scheduled_time:1755162298759444, pushed_time:1755162352720784, delay:1000000}, delay_in_priority_queue=53961340, threashold=1000000) [2025-08-14 09:06:09.767567] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=76][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a6098, timer:0x7f99200ff180, task:0x7f99200ff208, task_type:"N9oceanbase8observer18ObTableLoadService25ObClientTaskAutoAbortTaskE", scheduled_time:1755162298759444, pushed_time:1755162352720784, delay:1000000}, delay_in_thread_pool=17028294, threashold=500000) [2025-08-14 09:06:09.767544] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=200][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:09.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:09.767626] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=80][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:09.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:09.742515] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=970206][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:09.768173] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=25611][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:09.768191] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:09.768204] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:09.768217] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:09.768230] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:09.768320] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=87][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:09.768334] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:09.768494] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:09.768508] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:09.768810] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7277230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.768841] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7272330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a5090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.768854] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919822b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.768874] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.768886] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d71b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.768902] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.768914] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a27b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.768926] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a40b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.768938] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7faf30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.769301] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=78] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:09.769318] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] temporary file module use no blocks [2025-08-14 09:06:09.769328] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:09.769339] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:09.777937] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=15] gc stale ls task succ [2025-08-14 09:06:09.778927] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7352330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196fc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:09.779878] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=454][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:09.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:09.782206] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=255] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:09.782476] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=242] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:09.782947] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=202] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:09.783413] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=198] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:09.788987] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=56][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aca90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.771, task_priority:1, with_handle_protected:0}) [374]OB_MALLOC COST TOO MUCH TIME, cost_time=929845, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=929837], seq:[0] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=929873, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=929867], seq:[0, 1] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=919126, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=919076, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=919177, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=919150, WASH_BLOCK_END=27], seq:[0, 1, 2] [2025-08-14 09:06:09.789023] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f734fc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891918610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.720344] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=929908][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:09.789|207=931.32ms|210=0us|222=3us|total=931.36ms) [2025-08-14 09:06:10.720389] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4389] timer task use too much time(end=1755162370720388, start=1755162369789022, delta=931366) [2025-08-14 09:06:10.720405] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=935460, expected_run_ts_us_=1755162369784943, ObClockGenerator::getRealClock()=1755162370720404, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.720425] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.720440] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=934868, expected_run_ts_us_=1755162369785571, ObClockGenerator::getRealClock()=1755162370720440, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.720456] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.720483] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] task scheduled too delay(scheduled_delay=925736, expected_run_ts_us_=1755162369794746, ObClockGenerator::getRealClock()=1755162370720483, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.794, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.720495] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.794, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.720507] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=915970, expected_run_ts_us_=1755162369804536, ObClockGenerator::getRealClock()=1755162370720507, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.720517] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.720531] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=913472, expected_run_ts_us_=1755162369807059, ObClockGenerator::getRealClock()=1755162370720531, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.720543] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.720554] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=913398, expected_run_ts_us_=1755162369807156, ObClockGenerator::getRealClock()=1755162370720554, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.720579] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.807, task_priority:1, with_handle_protected:0}) [858]OB_MALLOC COST TOO MUCH TIME, cost_time=906935, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=906896, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=906972, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=906948, WASH_BLOCK_END=23], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=898313, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=16, ALLOC_CHUNK_END=898259, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=898339, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=898326, WASH_BLOCK_END=13], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=895206, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=895159, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=895223, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=895215, WASH_BLOCK_END=8], seq:[0, 1, 2] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=895208, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=16, ALLOC_CHUNK_END=895159, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=895230, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=895219, WASH_BLOCK_END=11], seq:[0, 1, 2] [822]OB_MALLOC COST TOO MUCH TIME, cost_time=884661, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=884613, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [822]OB_MALLOC COST TOO MUCH TIME, cost_time=884686, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=884675, WASH_BLOCK_END=11], seq:[0, 1, 2] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=878396, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=15, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=878338, ALLOC_CHUNK_END=13], seq:[0, 1, 2, 3, 4, 5, 6, 7] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=878418, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=878407, WASH_BLOCK_END=11], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=877225, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=877178, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=877246, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=877237, WASH_BLOCK_END=9], seq:[0, 1, 2] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=724339, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=724289, ALLOC_CHUNK_END=11], seq:[0, 1, 2, 3, 4, 5, 6, 7] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=724357, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=724349, WASH_BLOCK_END=8], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=708424, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=708378, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=708445, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=708435, WASH_BLOCK_END=9], seq:[0, 1, 2] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=695606, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=16, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=695556, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=695625, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=695615, WASH_BLOCK_END=10], seq:[0, 1, 2] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=480344, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=15, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=480295, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=480376, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=9, ALLOC_BLOCK#end=480358, WASH_BLOCK_END=9], seq:[0, 1, 2] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=453893, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=15, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=453837, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=453917, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=453908, WASH_BLOCK_END=9], seq:[0, 1, 2] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=356365, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=356320, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=356384, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=356375, WASH_BLOCK_END=8], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=683470, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=683434, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=683489, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=683481, WASH_BLOCK_END=8], seq:[0, 1, 2] [2025-08-14 09:06:10.012505] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:10.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:10.693469] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:1, time_interval:3.00s, expected_run_time:09:06:10.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:09.824755] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=276][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:09.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:10.242101] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=598055][errcode=0] EASY SLOW: thread: 7f995427d300, lastrun: 1755162250.789339 cost: 119.452759 loop:26617, slowcnt: 878 [2025-08-14 09:06:10.720593] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=919438, expected_run_ts_us_=1755162369801154, ObClockGenerator::getRealClock()=1755162370720593, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7397b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.723936] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=710352][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:10.125|207=711.43ms|210=0us|222=2us|total=711.45ms) [2025-08-14 09:06:10.727429] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=3486][errcode=-4389] timer task use too much time(end=1755162370727428, start=1755162370012503, delta=714925) [2025-08-14 09:06:10.727445] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=718309, expected_run_ts_us_=1755162370009134, ObClockGenerator::getRealClock()=1755162370727444, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:10.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:10.727482] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:10.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:10.724010] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=897894][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:09.824|207=899.25ms|210=0us|222=1us|total=899.26ms) [2025-08-14 09:06:09.769350] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623271208848, flush_io_finished_ret_:0, flush_io_finished_round_:26428, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:10.726449] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=5845][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7397b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.727522] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=3506][errcode=-4389] timer task use too much time(end=1755162370727522, start=1755162369824753, delta=902769) [2025-08-14 09:06:10.727505] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=0] task scheduled too delay(scheduled_delay=638189, expected_run_ts_us_=1755162370089316, ObClockGenerator::getRealClock()=1755162370727505, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:10.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:10.727670] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1216][errcode=0] task scheduled too delay(scheduled_delay=925951, expected_run_ts_us_=1755162369801719, ObClockGenerator::getRealClock()=1755162370727670, ret=0, ret="OB_SUCCESS", *this={this:0x7f98918299b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891916210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.730534] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=2857][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918299b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891916210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.730736] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=198][errcode=0] task scheduled too delay(scheduled_delay=904875, expected_run_ts_us_=1755162369825860, ObClockGenerator::getRealClock()=1755162370730736, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.825, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.730739] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=2785][errcode=0] task scheduled too delay(scheduled_delay=526441, expected_run_ts_us_=1755162370204297, ObClockGenerator::getRealClock()=1755162370730739, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:10.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:10.730752] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.825, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.730769] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:10.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:10.730771] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] task scheduled too delay(scheduled_delay=904699, expected_run_ts_us_=1755162369826071, ObClockGenerator::getRealClock()=1755162370730771, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.826, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.730783] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.826, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.730795] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=910021, expected_run_ts_us_=1755162369820773, ObClockGenerator::getRealClock()=1755162370730794, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.820, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.730806] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.820, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.730818] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=902479, expected_run_ts_us_=1755162369828338, ObClockGenerator::getRealClock()=1755162370730817, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72879b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.828, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.730829] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72879b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.828, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.730841] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=884431, expected_run_ts_us_=1755162369846410, ObClockGenerator::getRealClock()=1755162370730841, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.846, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.730852] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:09.846, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.730876] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72760b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.719, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.727818] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=929749] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:06:10.728058] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=452][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:10.718422] INFO [ARCHIVE] produce_log_fetch_task_ (ob_archive_sequencer.cpp:166) [859][T1002_ArcSeq][T1002][YB42AC4D0007-00063C495E299C16-0-0] [lt=3995836] archive round not in doing status, just skip(key={incarnation:-1, dest_id:-1, round:-1}, state={status:"INVALID"}) [2025-08-14 09:06:10.740099] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEBD-0-0] [lt=3368] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:06:10.740125] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEBD-0-0] [lt=21][errcode=-4389] run this task use too much time(*p_task_={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:0, time_interval:3.00s, expected_run_time:09:06:13.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:10.740149] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEBD-0-0] [lt=22][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:06:10.740|total=2.98s) [2025-08-14 09:06:10.741272] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2163] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:10.741863] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.741981] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=107][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.741996] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739be30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.742008] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.744309] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=2871] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:10.744514] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=730] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:10.744894] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=1112] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:10.743879] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1958] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:10.744569] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=631] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:06:10.748312] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=581][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:10.748566] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=249][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:10.748587] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:10.748605] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:10.748619] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:10.748607] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=360][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:10.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:10.748632] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:10.748645] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:10.748662] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:10.748692] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:10.748706] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:10.748718] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:10.749009] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:10.749025] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:10.734340] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:2005) [807][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=17] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1002) [2025-08-14 09:06:10.752156] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7edc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.752289] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=130][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ebe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.752327] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.754480] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=207] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:10.761201] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:10.765402] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=231] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:10.766687] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=260] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:10.766715] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=20] temporary file module use no blocks [2025-08-14 09:06:10.766729] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:10.766744] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:10.766763] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=18] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623272206398, flush_io_finished_ret_:0, flush_io_finished_round_:26389, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:10.769025] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:10.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:10.769058] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:10.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:10.778000] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=24] gc stale ls task succ [2025-08-14 09:06:10.778811] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1489][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:10.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:10.782219] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:10.782479] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:10.782872] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:10.783400] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:10.808959] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:10.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:10.812658] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=908784] gc stale ls task succ [2025-08-14 09:06:10.822808] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=886006] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:06:10.824685] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=1861][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:10.826872] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1144][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:10.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:10.826884] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1047][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:10.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:10.827967] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=374][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:10.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:10.829040] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=899638] PNIO [ratelimit] time: 1755162370829037, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007627, add_bytes: 0 [2025-08-14 09:06:10.866049] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fc8b0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcb90, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:1, time_interval:3.00s, expected_run_time:09:06:10.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:10.922873] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=25] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:06:10.924780] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=35] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:06:10.924792] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:10.926934] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:10.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:10.926959] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=86][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:10.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:10.928032] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:10.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:10.990408] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=409][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:10.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.010413] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:11.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.010458] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:11.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.020250] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:11.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.024804] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:11.026985] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:10.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.027022] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=218][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:10.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.028097] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:10.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.028344] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=199] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:11.028406] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:11.028456] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:11.028508] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:11.028552] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:11.028569] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:11.028591] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:11.090840] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:11.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.124389] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=683690] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:11.124819] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:11.127010] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:11.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.127071] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=206][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:11.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.128160] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:11.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.211473] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:11.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.224879] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:11.227065] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:11.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.227132] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:11.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.228225] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:11.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.241847] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:11.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.251907] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:11.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.261776] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:11.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.261809] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:11.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.271679] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:11.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.301824] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:11.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.324283] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=481205][errcode=0] EASY SLOW: thread: 7f995427d300, lastrun: 1755162250.789339 cost: 120.534941 loop:26617, slowcnt: 884 [2025-08-14 09:06:11.324952] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:11.327126] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:11.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.327197] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:11.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.328301] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:11.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.366574] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=356525] gc stale ls task succ [2025-08-14 09:06:11.424977] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:11.427192] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:11.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.427254] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:11.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.428339] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:11.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.493180] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:11.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.513335] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:11.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.513383] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:11.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.523043] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:11.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.525037] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:11.527252] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:11.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.527303] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:11.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.528370] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:11.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.593771] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:11.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.625096] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:11.627312] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:11.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.627379] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:11.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.628407] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:11.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.714446] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:11.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.249297] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:11.718904] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=468888][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:11.718970] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=63][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:11.718992] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:11.719005] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:11.719020] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:11.719041] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:11.719059] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:11.719070] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:11.719082] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:11.719097] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:11.719108] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:11.719120] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:11.719133] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:11.719144] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:11.719156] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:11.719168] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:11.719190] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:11.719249] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:11.719421] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:11.719476] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:11.721772] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1104] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [374]OB_MALLOC COST TOO MUCH TIME, cost_time=958525, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=958524], seq:[0] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=958638, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=42, WASH_BLOCK_END=45, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=958545], seq:[0, 1, 2, 3, 4] [2025-08-14 09:06:10.762166] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1507) [807][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=14987] start lib cache evict(tenant_id=1002, mem_hold=211812352, mem_limit=429496725, cache_obj_num=1331, cache_node_num=1330) [2025-08-14 09:06:11.722688] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1524) [807][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=959066] end lib cache evict(tenant_id=1002, cache_evict_num=0, mem_hold=211812352, mem_limit=429496725, cache_obj_num=1331, cache_node_num=1330) [2025-08-14 09:06:10.762576] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.750, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.722708] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2792) [807][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=17] schedule next cache evict task(evict_interval=5000000) [2025-08-14 09:06:11.722712] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=958681][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:10.762|207=960.13ms|210=0us|222=2us|total=960.16ms) [2025-08-14 09:06:11.722738] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4389] timer task use too much time(end=1755162371722737, start=1755162370762573, delta=960164) [2025-08-14 09:06:11.722764] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=967529, expected_run_ts_us_=1755162370755221, ObClockGenerator::getRealClock()=1755162371722751, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.722783] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.722796] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=965193, expected_run_ts_us_=1755162370757603, ObClockGenerator::getRealClock()=1755162371722796, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728d1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7291110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.722811] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728d1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7291110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.722833] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] task scheduled too delay(scheduled_delay=953939, expected_run_ts_us_=1755162370768893, ObClockGenerator::getRealClock()=1755162371722833, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aa990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.722846] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aa990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.722859] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=958586, expected_run_ts_us_=1755162370764272, ObClockGenerator::getRealClock()=1755162371722859, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.722871] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.722885] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=958302, expected_run_ts_us_=1755162370764582, ObClockGenerator::getRealClock()=1755162371722885, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919afe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.722907] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919afe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.722920] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=954834, expected_run_ts_us_=1755162370768085, ObClockGenerator::getRealClock()=1755162371722920, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d62b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.722932] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d62b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.722945] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=954782, expected_run_ts_us_=1755162370768162, ObClockGenerator::getRealClock()=1755162371722945, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.722956] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.722975] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] task scheduled too delay(scheduled_delay=950859, expected_run_ts_us_=1755162370772115, ObClockGenerator::getRealClock()=1755162371722974, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.722991] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723009] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] task scheduled too delay(scheduled_delay=947865, expected_run_ts_us_=1755162370775143, ObClockGenerator::getRealClock()=1755162371723009, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723021] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723034] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=952804, expected_run_ts_us_=1755162370770229, ObClockGenerator::getRealClock()=1755162371723034, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723045] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723057] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=951309, expected_run_ts_us_=1755162370771747, ObClockGenerator::getRealClock()=1755162371723057, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7397a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723068] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7397a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723079] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=938356, expected_run_ts_us_=1755162370784723, ObClockGenerator::getRealClock()=1755162371723079, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723090] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723101] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=942039, expected_run_ts_us_=1755162370781061, ObClockGenerator::getRealClock()=1755162371723101, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723112] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723125] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=940466, expected_run_ts_us_=1755162370782658, ObClockGenerator::getRealClock()=1755162371723125, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723137] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723150] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=926326, expected_run_ts_us_=1755162370796823, ObClockGenerator::getRealClock()=1755162371723149, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723162] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723175] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=920957, expected_run_ts_us_=1755162370802217, ObClockGenerator::getRealClock()=1755162371723175, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.802, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723187] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.802, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723209] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] task scheduled too delay(scheduled_delay=908878, expected_run_ts_us_=1755162370814330, ObClockGenerator::getRealClock()=1755162371723209, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.814, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723221] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.814, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723249] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] task scheduled too delay(scheduled_delay=905428, expected_run_ts_us_=1755162370817820, ObClockGenerator::getRealClock()=1755162371723248, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.817, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723260] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.817, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723272] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=904298, expected_run_ts_us_=1755162370818973, ObClockGenerator::getRealClock()=1755162371723272, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723283] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723294] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=904175, expected_run_ts_us_=1755162370819118, ObClockGenerator::getRealClock()=1755162371723294, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.819, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723305] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.819, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723316] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=909823, expected_run_ts_us_=1755162370813492, ObClockGenerator::getRealClock()=1755162371723316, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.813, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723338] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.813, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723351] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=894515, expected_run_ts_us_=1755162370828835, ObClockGenerator::getRealClock()=1755162371723351, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919831b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.828, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723363] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919831b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.828, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723376] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=888337, expected_run_ts_us_=1755162370835039, ObClockGenerator::getRealClock()=1755162371723376, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.835, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723400] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.835, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723413] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=875726, expected_run_ts_us_=1755162370847687, ObClockGenerator::getRealClock()=1755162371723413, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.847, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723426] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.847, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723438] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=878691, expected_run_ts_us_=1755162370844747, ObClockGenerator::getRealClock()=1755162371723438, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.844, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723450] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.844, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723464] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=878490, expected_run_ts_us_=1755162370844974, ObClockGenerator::getRealClock()=1755162371723464, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.844, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723474] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:10.844, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.723496] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ebbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:11.615, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:10.750109] INFO [STORAGE.TRANS] statistics (ob_location_adapter.cpp:70) [15497][T1_TimerWK16_FreInfoReload][T1][YB42AC4D0007-00063C4982499C16-0-0] [lt=11] location adapter statistics(renew_access=313, total_access=5447, error_count=0, renew_rate=5.745227635e-02) [2025-08-14 09:06:11.723590] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [15497][T1_TimerWK16_FreInfoReload][T1][YB42AC4D0007-00063C4982499C16-0-0] [lt=972006] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:06:11.723608] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [15497][T1_TimerWK16_FreInfoReload][T1][YB42AC4D0007-00063C4982499C16-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:06:11.723626] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [15497][T1_TimerWK16_FreInfoReload][T1][YB42AC4D0007-00063C4982499C16-0-0] [lt=18][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:06:11.723646] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [15497][T1_TimerWK16_FreInfoReload][T1][YB42AC4D0007-00063C4982499C16-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162231249835]}) [2025-08-14 09:06:11.725454] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:11.726401] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] [COMPACTION DAG STATUS] (start_time=1755162363765399, end_time=1755162371726400, dag_status={}) [2025-08-14 09:06:11.727376] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:11.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.727452] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:11.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.727671] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=956768] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:11.727710] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=33] temporary file module use no blocks [2025-08-14 09:06:11.727717] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:11.727733] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:11.727750] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=16] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623273167336, flush_io_finished_ret_:0, flush_io_finished_round_:26429, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:11.728464] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:11.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.732575] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=671] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:11.733035] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919abd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:11.721, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.733071] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919abb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:11.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.733085] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:11.726, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.733098] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fdc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:11.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.733110] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:11.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.744845] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:11.743109] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728be10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:11.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.741288] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:11.753225] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7585][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72847b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:11.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.754461] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=256] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:11.744271] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4629] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:11.750621] INFO [STORAGE.TRANS] get_gts (ob_gts_source.cpp:257) [15497][T1_TimerWK16_FreInfoReload][T1][YB42AC4D0007-00063C4982499C16-0-0] [lt=24] push gts task success(*task={:""}) [2025-08-14 09:06:11.751260] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7018] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:11.748102] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:11.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.759811] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=11687][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:11.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.755599] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11309][errcode=0] epoll handler too much time :11335 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:11.747489] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=35] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:11.760202] WDIAG ev_invoke_pending (ev.c:2136) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=12693][errcode=0] epoll handler too much time :12716 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:11.748641] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=3972] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:06:10.790369] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [412][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=29048] [sql nio session](*s={this:0x7f98f733a030, session_id:3221590255, trace_id:YB42AC4D0007-00063C49651A3232-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:257}, err:0, last_decode_time:1755161952736711, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1716, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:06:11.744989] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=10] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:11.755546] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=354] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 08:59:51.727949] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:981) [15456][T1_TimerWK14_TntSharedTimer][T1][YB42AC4D0007-00063C4981299C19-0-0] [lt=8][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4"}, record_ret=-4283, ret=-4283) [2025-08-14 09:06:11.763203] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=229] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:11.763417] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:11.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.763453] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891806f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:11.751, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.763468] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:11.750, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.765336] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=256] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:11.765368] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=279] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:11.765512] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=62] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:11.765619] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:11.741046] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2510) [807][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=10] Dumping All Cache Objs(alloc_obj_list.count()=1331, alloc_obj_list=[{obj_id:1, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:49654}, {obj_id:2, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50591}, {obj_id:3, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49570}, {obj_id:4, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:158277}, {obj_id:5, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207030}, {obj_id:6, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:129654}, {obj_id:7, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:170276}, {obj_id:8, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:92905}, {obj_id:9, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55437}, {obj_id:11, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49896}, {obj_id:12, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54788}, {obj_id:13, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60105}, {obj_id:15, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:108509}, {obj_id:16, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54018}, {obj_id:17, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:107151}, {obj_id:18, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127908}, {obj_id:19, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52851}, {obj_id:20, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46488}, {obj_id:21, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62127}, {obj_id:22, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:112117}, {obj_id:23, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71118}, {obj_id:24, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47416}, {obj_id:25, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64955}, {obj_id:26, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:107240}, {obj_id:27, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203554}, {obj_id:28, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:149171}, {obj_id:29, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:99644}, {obj_id:30, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:109417}, {obj_id:31, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:104852}, {obj_id:32, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:81681}, {obj_id:33, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:86868}, {obj_id:34, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:77320}, {obj_id:35, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73401}, {obj_id:36, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67092}, {obj_id:37, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:90423}, {obj_id:38, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73995}, {obj_id:39, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:115258}, {obj_id:40, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78254}, {obj_id:41, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63461}, {obj_id:42, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60610}, {obj_id:43, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59274}, {obj_id:44, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74545}, {obj_id:45, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:68364}, {obj_id:46, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62191}, {obj_id:47, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59473}, {obj_id:48, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62548}, {obj_id:49, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59505}, {obj_id:50, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62549}, {obj_id:51, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59441}, {obj_id:52, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65989}, {obj_id:53, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:61026}, {obj_id:54, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57028}, {obj_id:55, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85137}, {obj_id:56, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56916}, {obj_id:57, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:61962}, {obj_id:58, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57316}, {obj_id:59, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65864}, {obj_id:60, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56996}, {obj_id:61, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58212}, {obj_id:62, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65512}, {obj_id:63, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60453}, {obj_id:64, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56884}, {obj_id:65, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59085}, {obj_id:66, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235524}, {obj_id:67, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:133191}, {obj_id:68, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:122973}, {obj_id:69, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222820}, {obj_id:70, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:119011}, {obj_id:71, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62425}, {obj_id:72, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63273}, {obj_id:73, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173058}, {obj_id:74, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93811}, {obj_id:75, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:90631}, {obj_id:76, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:81787}, {obj_id:77, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71611}, {obj_id:78, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65300}, {obj_id:79, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51380}, {obj_id:80, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:81, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48067}, {obj_id:82, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:83, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52125}, {obj_id:84, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:85, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120160}, {obj_id:86, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:87, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97242}, {obj_id:88, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:90, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:91, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:46504}, {obj_id:92, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:68970}, {obj_id:93, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59166}, {obj_id:94, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60338}, {obj_id:95, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51478}, {obj_id:96, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:109830}, {obj_id:97, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48968}, {obj_id:98, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:112041}, {obj_id:99, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46440}, {obj_id:100, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69133}, {obj_id:101, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:102, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54372}, {obj_id:103, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:104, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48280}, {obj_id:105, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:116242}, {obj_id:106, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53185}, {obj_id:107, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:48908}, {obj_id:108, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51274}, {obj_id:109, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:108920}, {obj_id:110, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57540}, {obj_id:111, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54911}, {obj_id:112, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66049}, {obj_id:113, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:114, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:107403}, {obj_id:115, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59412}, {obj_id:116, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59924}, {obj_id:117, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64784}, {obj_id:118, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:119, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:120, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:121, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:30464}, {obj_id:122, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:123, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:124, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64774}, {obj_id:125, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63004}, {obj_id:126, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:127, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:30482}, {obj_id:133, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59119}, {obj_id:134, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:109167}, {obj_id:140, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57392}, {obj_id:142, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72205}, {obj_id:143, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60318}, {obj_id:144, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59063}, {obj_id:145, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52621}, {obj_id:146, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60640}, {obj_id:147, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:110708}, {obj_id:148, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:79964}, {obj_id:149, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105305}, {obj_id:150, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64637}, {obj_id:152, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:107581}, {obj_id:153, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56979}, {obj_id:154, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:68485}, {obj_id:155, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51889}, {obj_id:157, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50408}, {obj_id:158, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47936}, {obj_id:160, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:112534}, {obj_id:161, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69948}, {obj_id:162, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60075}, {obj_id:163, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50629}, {obj_id:164, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50612}, {obj_id:165, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:33277}, {obj_id:167, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78418}, {obj_id:168, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50072}, {obj_id:169, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51380}, {obj_id:170, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:172, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55901}, {obj_id:173, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57684}, {obj_id:175, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55665}, {obj_id:177, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49689}, {obj_id:178, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62390}, {obj_id:179, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:111173}, {obj_id:181, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67757}, {obj_id:187, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:30668}, {obj_id:188, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67536}, {obj_id:189, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:113436}, {obj_id:191, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67820}, {obj_id:192, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52042}, {obj_id:193, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46840}, {obj_id:194, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46856}, {obj_id:195, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193886}, {obj_id:197, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:128794}, {obj_id:198, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69301}, {obj_id:199, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173059}, {obj_id:201, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:145729}, {obj_id:202, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:203, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:98424}, {obj_id:205, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:137715}, {obj_id:206, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59503}, {obj_id:207, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192000}, {obj_id:210, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47408}, {obj_id:211, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59513}, {obj_id:213, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59188}, {obj_id:215, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46848}, {obj_id:218, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54467}, {obj_id:219, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62656}, {obj_id:222, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57417}, {obj_id:223, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57076}, {obj_id:225, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:80665}, {obj_id:226, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:68110}, {obj_id:227, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59478}, {obj_id:228, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54200}, {obj_id:229, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71579}, {obj_id:230, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:79425}, {obj_id:231, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53027}, {obj_id:232, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50327}, {obj_id:233, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52701}, {obj_id:234, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53495}, {obj_id:235, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54912}, {obj_id:236, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:115782}, {obj_id:237, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49369}, {obj_id:239, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71297}, {obj_id:240, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:45664}, {obj_id:241, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:242, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:29563}, {obj_id:243, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:29547}, {obj_id:244, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72439}, {obj_id:245, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:82003}, {obj_id:246, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60693}, {obj_id:247, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:248, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:249, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:68970}, {obj_id:253, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:13745}, {obj_id:254, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53850}, {obj_id:255, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1244037}, {obj_id:256, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53353}, {obj_id:257, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:979432}, {obj_id:258, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:259, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:441763}, {obj_id:260, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:131652}, {obj_id:261, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102208}, {obj_id:262, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174688}, {obj_id:263, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:68064}, {obj_id:264, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:124055}, {obj_id:265, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51856}, {obj_id:266, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55892}, {obj_id:267, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53476}, {obj_id:268, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248928}, {obj_id:270, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:111648}, {obj_id:272, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55724}, {obj_id:273, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:139752}, {obj_id:274, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65613}, {obj_id:275, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74271}, {obj_id:276, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189587}, {obj_id:277, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46437}, {obj_id:278, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47893}, {obj_id:279, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:150376}, {obj_id:280, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49545}, {obj_id:281, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175345}, {obj_id:282, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:131048}, {obj_id:283, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72474}, {obj_id:284, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:285, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:286, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240919}, {obj_id:288, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55756}, {obj_id:291, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:81957}, {obj_id:292, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268057}, {obj_id:297, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85376}, {obj_id:298, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85602}, {obj_id:299, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:111929}, {obj_id:300, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:286644}, {obj_id:301, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56303}, {obj_id:302, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182524}, {obj_id:303, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:137836}, {obj_id:304, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:216847}, {obj_id:307, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55868}, {obj_id:308, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72832}, {obj_id:309, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102124}, {obj_id:310, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225211}, {obj_id:313, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:75504}, {obj_id:314, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:147275}, {obj_id:315, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204179}, {obj_id:316, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:588871}, {obj_id:319, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69223}, {obj_id:320, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206652}, {obj_id:321, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:324, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74659}, {obj_id:325, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:232949}, {obj_id:326, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53958}, {obj_id:327, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180179}, {obj_id:328, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:135494}, {obj_id:332, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:80174}, {obj_id:333, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:259278}, {obj_id:334, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55154}, {obj_id:335, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181355}, {obj_id:336, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:136712}, {obj_id:342, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67384}, {obj_id:343, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197887}, {obj_id:345, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53618}, {obj_id:346, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179771}, {obj_id:347, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:135134}, {obj_id:348, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:350, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:146442}, {obj_id:364, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:75480}, {obj_id:368, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50876}, {obj_id:369, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74415}, {obj_id:370, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51322}, {obj_id:371, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51084}, {obj_id:372, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51442}, {obj_id:374, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51202}, {obj_id:375, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52790}, {obj_id:376, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52854}, {obj_id:377, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54387}, {obj_id:378, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52854}, {obj_id:379, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51586}, {obj_id:380, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54872}, {obj_id:381, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51378}, {obj_id:382, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53030}, {obj_id:383, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55599}, {obj_id:384, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51378}, {obj_id:385, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51314}, {obj_id:386, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51250}, {obj_id:387, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53046}, {obj_id:388, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54815}, {obj_id:389, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51218}, {obj_id:390, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51218}, {obj_id:391, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51250}, {obj_id:392, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52870}, {obj_id:393, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51266}, {obj_id:394, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51362}, {obj_id:395, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51250}, {obj_id:396, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59119}, {obj_id:397, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51362}, {obj_id:398, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53062}, {obj_id:399, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51362}, {obj_id:400, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51362}, {obj_id:401, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51250}, {obj_id:402, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51218}, {obj_id:409, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:149775}, {obj_id:410, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66617}, {obj_id:415, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:5265}, {obj_id:416, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:31009}, {obj_id:417, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:418, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48336}, {obj_id:419, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59108}, {obj_id:420, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50938}, {obj_id:421, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48671}, {obj_id:422, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56563}, {obj_id:423, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46872}, {obj_id:424, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:61808}, {obj_id:425, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59824}, {obj_id:427, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:618230}, {obj_id:428, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:153758}, {obj_id:429, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69455}, {obj_id:431, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55724}, {obj_id:433, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55756}, {obj_id:436, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85376}, {obj_id:437, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:108532}, {obj_id:439, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55868}, {obj_id:441, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:75504}, {obj_id:443, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:390100}, {obj_id:444, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:360217}, {obj_id:445, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:457, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55740}, {obj_id:458, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:96462}, {obj_id:459, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212819}, {obj_id:460, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:340001}, {obj_id:461, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:473, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:154884}, {obj_id:476, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:375543}, {obj_id:477, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:116952}, {obj_id:478, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:123944}, {obj_id:479, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105139}, {obj_id:490, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50527}, {obj_id:491, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:151767}, {obj_id:494, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47368}, {obj_id:495, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48232}, {obj_id:496, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62071}, {obj_id:497, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:61352}, {obj_id:500, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52851}, {obj_id:504, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:333278}, {obj_id:505, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:129618}, {obj_id:507, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55724}, {obj_id:509, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55756}, {obj_id:512, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85376}, {obj_id:514, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55868}, {obj_id:516, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:75504}, {obj_id:525, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:549, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53306}, {obj_id:550, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62783}, {obj_id:551, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54109}, {obj_id:552, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:103697}, {obj_id:553, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120708}, {obj_id:554, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67452}, {obj_id:560, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:349340}, {obj_id:561, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127623}, {obj_id:563, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55724}, {obj_id:565, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55756}, {obj_id:568, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85376}, {obj_id:570, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55868}, {obj_id:572, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:75504}, {obj_id:581, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:605, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53171}, {obj_id:606, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55852}, {obj_id:607, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:116904}, {obj_id:608, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:258163}, {obj_id:609, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:132926}, {obj_id:610, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:80606}, {obj_id:611, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127311}, {obj_id:612, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:124808}, {obj_id:613, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54894}, {obj_id:614, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58088}, {obj_id:615, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:88608}, {obj_id:616, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:121604}, {obj_id:617, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63204}, {obj_id:618, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52886}, {obj_id:623, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:465799}, {obj_id:625, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55724}, {obj_id:627, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55756}, {obj_id:630, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85376}, {obj_id:632, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55868}, {obj_id:636, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:75504}, {obj_id:647, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:674, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:281099}, {obj_id:675, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:133643}, {obj_id:677, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55724}, {obj_id:679, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55756}, {obj_id:682, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85376}, {obj_id:684, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55868}, {obj_id:686, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:75504}, {obj_id:692, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196769}, {obj_id:700, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55740}, {obj_id:701, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:728, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:277051}, {obj_id:730, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55724}, {obj_id:732, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55756}, {obj_id:735, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85376}, {obj_id:737, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55868}, {obj_id:739, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:75504}, {obj_id:749, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:773, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:603737}, {obj_id:774, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58158}, {obj_id:775, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:84541}, {obj_id:776, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66315}, {obj_id:777, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200157}, {obj_id:778, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:277379}, {obj_id:780, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279299}, {obj_id:781, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54040}, {obj_id:782, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57275}, {obj_id:783, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71438}, {obj_id:785, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:108323}, {obj_id:786, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:145723}, {obj_id:787, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63771}, {obj_id:789, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:792, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:798, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:473748}, {obj_id:800, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55724}, {obj_id:802, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55756}, {obj_id:805, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85376}, {obj_id:807, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55868}, {obj_id:809, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:75504}, {obj_id:820, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:842, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46504}, {obj_id:843, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53171}, {obj_id:844, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55852}, {obj_id:845, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:116904}, {obj_id:846, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:258163}, {obj_id:847, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:80606}, {obj_id:848, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127311}, {obj_id:849, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54894}, {obj_id:855, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:81073}, {obj_id:856, tenant_id:1002, log_del_time:9223372036854775807, re [2025-08-14 09:06:11.744334] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:11.766855] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=22486][errcode=0] epoll handler too much time :22532 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:11.766875] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:11.766902] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=22] temporary file module use no blocks [2025-08-14 09:06:11.766912] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:11.747803] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=295] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:11.766926] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:11.766942] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623273206561, flush_io_finished_ret_:0, flush_io_finished_round_:26390, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:11.768310] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=274][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6098, timer:0x7f99200ff180, task:0x7f99200ff208, task_type:"N9oceanbase8observer18ObTableLoadService25ObClientTaskAutoAbortTaskE", scheduled_time:1755162370767586, pushed_time:0, delay:1000000}, running_task_token={this:0x7f991b7a6178, timer:0x7f99200ff180, task:0x7f99200ff220, task_type:"N9oceanbase8observer18ObTableLoadService21ObClientTaskPurgeTaskE", scheduled_time:1755162300760824, pushed_time:1755162369767600, delay:1000000}, total_delay=1000723, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:11.768366] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:11.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.768399] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:11.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.773483] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73539b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7396b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:11.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.774836] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:11.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.778061] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:06:11.782217] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:11.782424] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:11.782886] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:11.783384] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:11.803672] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:11.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.805036] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=49][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:11.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.813012] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=23] gc stale ls task succ [2025-08-14 09:06:11.813734] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d22b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eaf90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:11.808, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.823529] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=25] LogLoopThread round_cost_time(us)(round_cost_time=13) [2025-08-14 09:06:11.823795] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:11.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.823823] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:11.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.825514] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:11.827413] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:11.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.827516] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:11.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.828526] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=66][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:11.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.836637] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=21] PNIO [ratelimit] time: 1755162371836635, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007598, add_bytes: 0 [2025-08-14 09:06:11.843887] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:11.831, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.843911] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:11.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.853954] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ac10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:11.849, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.864023] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:11.852, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.874086] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:11.861, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.904278] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919859b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:11.896, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:11.925558] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:11.927447] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:11.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.927579] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:11.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.928587] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:11.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.991156] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:11.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.019853] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:12.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.019889] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:12.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.025593] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:12.026270] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:12.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.027503] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:11.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.027639] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:11.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.028646] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:11.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.029291] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:12.029344] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:12.029395] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:12.029438] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:12.029479] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:12.029492] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:12.029512] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:12.090304] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:12.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.125109] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=26] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:12.125651] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:12.127566] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:12.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.127671] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:12.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.128702] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:12.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.211031] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:12.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.219662] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:12.219711] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:12.219730] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:12.219743] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:12.219756] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:12.219768] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:12.219785] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:12.219806] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:12.219821] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:12.219841] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:12.219861] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:12.219876] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:12.219887] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:12.219939] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:12.219951] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:12.219966] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:12.219993] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:12.220008] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:12.220025] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:12.220187] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:12.220211] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:12.225712] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:12.227634] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:12.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.227734] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:12.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.228763] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:12.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.242694] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:12.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.252728] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:12.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.261304] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:12.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.261333] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:12.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.277714] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:12.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.307889] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:12.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.325771] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:12.326588] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] EASY SLOW: thread: 7f995427d300, lastrun: 1755162250.789339 cost: 121.537246 loop:26617, slowcnt: 894 [2025-08-14 09:06:12.327644] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:12.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.327793] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:12.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.328828] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:12.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.366642] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=19] gc stale ls task succ [2025-08-14 09:06:12.425832] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:12.427708] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:12.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.427871] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:12.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.428888] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:12.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.494082] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:12.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.512904] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:12.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.512942] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:12.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.525878] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:12.527773] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:12.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.527897] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:12.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.528952] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:12.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.529170] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:12.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.593421] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:12.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.625937] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:12.627841] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:12.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.627941] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:12.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.628974] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:12.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.714169] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:12.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.720370] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:12.720412] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:12.720425] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:12.720438] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:12.720451] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:12.720476] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:12.720489] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:12.720513] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:12.720536] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:12.720548] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:12.720564] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:12.720575] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:12.720588] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:12.720609] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:12.720621] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:12.720635] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:12.720648] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:12.720671] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:12.720683] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:12.720838] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:12.720859] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 08:59:51.727949] EDIAG [SERVER] query (ob_inner_sql_connection.cpp:981) [15456][T1_TimerWK14_TntSharedTimer][T1][YB42AC4D0007-00063C4981299C19-0-0] [lt=8][errcode=-4283] LOGGER COST TOO MUCH TIME, cost: 379460383, owner: ObLog, click_count: 5, time dist:[FORMAT_END=379460372, ObMalloc#start=1, ObMalloc#end=4, ALLOC_END=0, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x124e01a8 0x96a5705 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x11d0e838 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:12.721751] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:12.727893] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:12.728564] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:12.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.731290] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:12.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.731443] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:12.724, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:12.736002] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:12.739669] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2800) [807][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=25492] schedule next cache evict task(evict_interval=5000000) [2025-08-14 09:06:12.729038] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:12.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:11.925401] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=19][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:06:11.926667] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=24][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:06:11.823579] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] get_fetch_log_dst failed or invalid(ret=0, palf_id=1, self="172.77.0.7:2882", fetch_log_dst="172.77.0.7:2882") [2025-08-14 09:06:12.739967] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=914999] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:06:12.740020] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] get_fetch_log_dst failed or invalid(ret=0, palf_id=1001, self="172.77.0.7:2882", fetch_log_dst="172.77.0.7:2882") [2025-08-14 09:06:12.740030] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:06:12.742112] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9597] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:12.743798] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4385] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:12.744264] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:12.744844] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7954] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:12.739718] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a4118, timer:0x7f9922d73f40, task:0x7f9922ff1b70, task_type:"N9oceanbase3sql26ObPlanCacheEliminationTaskE", scheduled_time:1755162353767337, pushed_time:1755162353828331, delay:5000000}, delay_in_thread_pool=15939261, threashold=500000) [2025-08-14 09:06:12.748897] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=83][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:12.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.749386] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=10] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:12.750801] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=5848] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:12.752316] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:12.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:12.752369] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=50][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:12.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:12.754260] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:12.755422] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4827] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:12.757297] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:12.759001] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:12.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.759297] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:12.760385] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=15308] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:12.761535] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:12.762411] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:12.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:12.763540] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:12.765300] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:12.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.765332] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:12.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.766092] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:12.767103] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=19105] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:12.760361] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=11599] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:06:12.767054] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=34] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:12.767292] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=229] temporary file module use no blocks [2025-08-14 09:06:12.767310] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=17] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:12.767325] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:12.767341] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623274206745, flush_io_finished_ret_:0, flush_io_finished_round_:26391, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:12.772472] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:12.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:12.772511] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:12.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:12.772543] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d36b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ebb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:12.767, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:12.772558] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:12.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:12.772926] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:12.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.775703] WDIAG [STORAGE.TRANS] wait (ob_ts_mgr.cpp:195) [15497][T1_TimerWK16_FreInfoReload][T1][YB42AC4D0007-00063C4982499C16-0-0] [lt=6241][errcode=-4012] ObTsSyncGetTsCbTask cond wait failed(ret=-4012) [2025-08-14 09:06:12.778132] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=22] gc stale ls task succ [2025-08-14 09:06:12.782245] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=37] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:12.782422] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:12.782518] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:12.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:12.782555] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72739b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a5e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:12.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:12.782574] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72779b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eab10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:12.777, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:12.782592] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728bc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:12.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:12.782938] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:12.783421] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:12.792560] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919902b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:12.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:12.792603] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:12.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:12.792619] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:12.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:12.802628] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7277730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:12.794, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:12.802660] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:12.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:12.803152] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:12.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.813050] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=24] gc stale ls task succ [2025-08-14 09:06:12.822733] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:12.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:12.827939] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:12.828629] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=63][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:12.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.829101] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=10712][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:12.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.831318] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:12.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.832809] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:12.820, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:12.842862] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:12.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:12.844255] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=23] PNIO [ratelimit] time: 1755162372844252, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007617, add_bytes: 0 [2025-08-14 09:06:12.927973] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:06:12.928025] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=51][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:12.928691] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:12.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.929166] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:12.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.931382] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:12.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:12.963605] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196aa90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:12.952, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:12.963640] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:12.953, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:12.983725] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891992fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196de90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:12.970, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:12.990317] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:12.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.016647] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:13.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.016681] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:13.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.024661] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:13.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.028033] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:13.028753] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:12.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.029230] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:12.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.031435] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:12.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.031661] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:13.031716] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:13.031760] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:13.031803] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:13.031853] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:13.031866] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:13.031879] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:13.097058] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:13.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.128131] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:13.128133] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=20] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:13.128815] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:13.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.129275] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:13.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.131506] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:13.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.217685] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:13.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.221045] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:13.221087] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:13.221102] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:13.221117] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:13.221130] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:13.221146] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:13.221168] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:13.221189] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:13.221200] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:13.221220] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:13.221252] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:13.221260] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:13.221270] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:13.221281] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:13.221297] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:13.221321] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:13.221331] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:13.221349] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:13.221367] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:13.221574] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:13.221600] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:13.228193] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:13.228878] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:13.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.229334] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:13.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.231568] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:13.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.241758] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:13.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.251823] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:13.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.267872] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:13.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.267907] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:13.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.276213] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:13.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.306355] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:13.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.327099] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] EASY SLOW: thread: 7f995427d300, lastrun: 1755162250.789339 cost: 122.537757 loop:26617, slowcnt: 904 [2025-08-14 09:06:13.328207] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:13.328941] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:13.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.329396] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:13.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.331592] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:13.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.366692] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:06:13.428273] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:13.429018] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:13.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.429461] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:13.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.431656] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:13.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.493291] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:13.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.519301] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:13.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.519349] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:13.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.527618] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:13.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.528286] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:13.529081] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=50][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:13.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.529526] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:13.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.531718] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:13.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.599783] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:13.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.628345] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:13.629147] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:13.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.629576] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:13.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.631762] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:13.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.698460] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEBF-0-0] [lt=12] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:06:13.710308] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=56][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:13.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.719960] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [807][T1002_TimerWK3_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=5298] ====== check clog disk timer task ====== [2025-08-14 09:06:13.720009] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [807][T1002_TimerWK3_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=35] get_disk_usage(ret=0, capacity(MB):=24883, used(MB):=8642) [2025-08-14 09:06:13.720039] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a5298, timer:0x7f9922ffd5d0, task:0x7f9922ffd618, task_type:"N9oceanbase7storage10checkpoint19ObCheckPointService24ObCheckClogDiskUsageTaskE", scheduled_time:1755162354720050, pushed_time:1755162354823523, delay:2000000}, delay_in_thread_pool=17921512, threashold=500000) [2025-08-14 09:06:13.720093] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4389] timer task too much delay in priority queue(token={this:0x7f991b7a4498, timer:0x7f9966a63f10, task:0x7f9922e05010, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr11TableGCTaskE", scheduled_time:1755162331786875, pushed_time:1755162356733580, delay:20000}, delay_in_priority_queue=24946705, threashold=1000000) [2025-08-14 09:06:13.720106] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a4498, timer:0x7f9966a63f10, task:0x7f9922e05010, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr11TableGCTaskE", scheduled_time:1755162331786875, pushed_time:1755162356733580, delay:20000}, delay_in_thread_pool=16986487, threashold=500000) [2025-08-14 09:06:13.721789] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:13.721837] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:13.721854] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:13.721879] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:13.721893] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:13.721917] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:13.721931] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:13.721950] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:13.721972] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:13.721990] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:13.722015] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:13.722027] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:13.722041] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:13.722069] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:13.722082] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:13.722094] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:13.722107] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:13.722133] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:13.722146] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:13.722333] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:13.722348] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:13.722464] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:13.729481] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:13.729639] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:13.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.730504] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:13.730533] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=23] temporary file module use no blocks [2025-08-14 09:06:13.730543] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:13.730557] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:13.730571] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623275168321, flush_io_finished_ret_:0, flush_io_finished_round_:26431, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:13.730610] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:13.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.731785] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:13.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.734545] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:13.740320] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:13.742657] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:06:13.742779] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=114] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:06:13.742814] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=44] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:13.743272] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [854][T1002_ObXAHbWor][T1002][Y0-0000000000000000-0-0] [lt=973904] XA scheduler heartbeat task statistics(avg_time=1) [2025-08-14 09:06:13.744392] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:13.744422] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:13.745068] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=54][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:13.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.748828] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:13.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:13.748865] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:13.737, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:13.748892] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e71b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:13.739, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:13.748904] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918119b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:13.737, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:13.750470] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=23] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:13.751074] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:13.755109] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:13.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.755752] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:13.757264] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=31] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:13.757910] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:13.758891] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72774b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:13.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:13.758926] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fbe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:13.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:13.759393] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=29] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:13.761428] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:13.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.761470] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:13.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.761619] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=26] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:13.761917] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=14] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:13.762524] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:13.740373] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:941) [15497][T1_TimerWK16_FreInfoReload][T1][YB42AC4D0007-00063C4982499C16-0-0] [lt=29][errcode=-4012] failed to wait ObTsSyncGetTsCbTask(ret=-4012, tenant_id=1, timeout_us=1000000) [2025-08-14 09:06:13.766115] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:13.767209] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=34] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:13.767334] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=6878] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:06:13.767435] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:13.768947] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:13.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:13.768979] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:13.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:13.769008] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:13.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:00:40.757049] EDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1014) [360][UserLocAsyncUp0][T0][YB42AC4D0007-00063C495D499F17-0-0] [lt=1001541][errcode=-4664] LOGGER COST TOO MUCH TIME, cost: 2963677, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2963670, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x20218d15 0x2020e7d9 0x9a194f6 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 08:57:47.734198] EDIAG [SERVER] check_tenant_memory (ob_plugin_vector_index_scheduler.cpp:493) [15479][T1002_TimerWK14_TTLTabletMgr][T1002][YB42AC4D0007-00063C4981699C1C-0-0] [lt=1143][errcode=0] LOGGER COST TOO MUCH TIME, cost: 176743342, owner: ObLog, click_count: 5, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=467, ALLOC_END=176742861, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1fb91602 0x1fb85df3 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:13.770689] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:13.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.778154] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=21] gc stale ls task succ [2025-08-14 09:06:13.778988] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919aef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:13.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:13.779036] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7270a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a4190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:13.762, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:13.779068] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:13.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:13.779088] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:13.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:13.779112] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:13.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:13.782082] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:13.782326] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:13.782962] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=29] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:13.783406] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:13.787819] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [878][T1002_ArcSender][T1002][YB42AC4D0007-00063C495EA99C16-0-0] [lt=24229] ObArchiveSender is running(thread_index=4) [2025-08-14 09:06:13.789066] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f734de30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891919810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:13.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:13.789110] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:13.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:13.789131] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:13.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:13.789146] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d59b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:13.773, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:13.789165] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efa10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:13.777, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:13.791966] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=157] [sql nio session](*s={this:0x7f98f72d0030, session_id:3221589415, trace_id:YB42AC4D0007-00063C4965FA43DC-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:245}, err:0, last_decode_time:1755161751714366, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1662, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:06:13.792013] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=46] [sql nio session](*s={this:0x7f98f72d5030, session_id:3221590200, trace_id:YB42AC4D0007-00063C4965FA43E0-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:248}, err:0, last_decode_time:1755161776777167, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1593, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:06:13.799092] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eba10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:13.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:13.799128] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:13.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:13.800894] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:13.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.809149] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:13.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:13.809186] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:13.790, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:13.813112] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=32] gc stale ls task succ [2025-08-14 09:06:13.829520] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:13.829697] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:13.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.830672] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:13.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.831859] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:13.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.839322] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:13.827, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:13.849387] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739af30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196bc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:13.837, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:13.851873] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=23] PNIO [ratelimit] time: 1755162373851871, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007619, add_bytes: 0 [2025-08-14 09:06:13.859440] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:13.841, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:13.859478] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:13.849, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:13.859509] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:13.849, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:13.867804] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fc8b0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcb90, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:1, time_interval:3.00s, expected_run_time:09:06:13.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.900166] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:13.892, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:13.920290] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:13.917, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:13.929615] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=39] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:06:13.929642] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:13.929755] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=59][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:13.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.930352] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:13.922, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:13.930736] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:13.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.931920] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:13.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:13.996560] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:13.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.012896] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:14.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.012949] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=51][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:14.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.022995] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:14.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.029634] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:14.029815] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:13.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.030765] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:13.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.031992] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:13.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.034704] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:14.034769] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:14.034813] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:14.034863] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:14.034903] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:14.034915] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:14.034928] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:14.093229] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:14.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.129059] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=18] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:14.129670] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:14.129872] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:14.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.130827] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:14.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.132045] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:14.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.213823] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:14.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.222530] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:14.222564] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:14.222582] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:14.222607] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:14.222621] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:14.222640] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:14.222656] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:14.222676] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:14.222695] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:14.222718] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:14.222737] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:14.222752] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:14.222765] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:14.222775] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:14.222797] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:14.222806] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:14.222815] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:14.222831] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:14.222856] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:14.223038] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:14.223065] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:14.229712] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:14.229908] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:14.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.230882] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:14.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.232108] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:14.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.247942] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:14.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.258011] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:14.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.264078] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:14.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.264113] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:14.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.274476] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=50][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:14.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.304640] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:14.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.329700] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] EASY SLOW: thread: 7f995427d300, lastrun: 1755162250.789339 cost: 123.540359 loop:26617, slowcnt: 914 [2025-08-14 09:06:14.329774] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:14.329954] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:14.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.330934] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:14.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.332185] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:14.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.366756] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=19] gc stale ls task succ [2025-08-14 09:06:14.429804] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:14.430038] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:14.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.430997] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:14.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.432249] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:14.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.499496] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:14.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.515327] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:14.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.515363] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:14.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.525859] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:14.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.529856] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:14.530093] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:14.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.531058] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:14.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.532309] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:14.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.595765] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:14.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.629911] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:14.630154] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:14.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.631120] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:14.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.632332] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:14.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.716488] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:14.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.722111] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:14.723251] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:14.723289] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:14.723308] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:14.723324] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:14.723340] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:14.723357] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:14.723370] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:14.723395] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:14.723407] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:14.723420] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:14.723436] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:14.723448] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:14.723469] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:14.723483] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:14.723497] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:14.723510] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:14.723524] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:14.723546] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:14.723560] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:14.723717] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:14.723733] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:14.724436] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1832) [15497][T1_TimerWK16_FreInfoReload][T1][YB42AC4D0007-00063C4982499C16-0-0] [lt=23343][errcode=-4012] get gts fail(ret=-4012, timeout_us=1000000, request_time=[mts=1755162370727750]) [2025-08-14 09:06:14.724477] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1844) [15497][T1_TimerWK16_FreInfoReload][T1][YB42AC4D0007-00063C4982499C16-0-0] [lt=40][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1755162386693454, request_time=[mts=1755162370727750]) [2025-08-14 09:06:14.724495] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:662) [15497][T1_TimerWK16_FreInfoReload][T1][YB42AC4D0007-00063C4982499C16-0-0] [lt=13][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f98b4f97440, tx_id:{txid:0}, state:1, addr:"172.77.0.7:2882", tenant_id:1, session_id:3221590981, assoc_session_id:3221590981, client_sid:3221590981, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1755162369721591, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180067074, seq_base:1755162225746390, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}) [2025-08-14 09:06:13.771617] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:551) [360][UserLocAsyncUp0][T0][YB42AC4D0007-00063C495D499F17-0-0] [lt=2963677][errcode=-4664] fail to renew location(ret=-4664, ret="OB_GET_LOCATION_TIME_OUT", task={cluster_id:1753069619, tenant_id:1002, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1755161935748163}) [2025-08-14 09:06:14.730148] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:14.730194] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:14.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.730978] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:14.731184] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:14.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.732275] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:14.732432] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:14.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.734961] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:14.728, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:14.734999] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f734c030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891807390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:14.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:14.739335] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd110, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd3f0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:10.00s, expected_run_time:09:06:14.726, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.740505] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:14.740967] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a4498, timer:0x7f9966a63f10, task:0x7f9922e05010, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr11TableGCTaskE", scheduled_time:1755162373740116, pushed_time:0, delay:20000}, running_task_token={this:0x7f991b7a4578, timer:0x7f9966a63f10, task:0x7f9922e05028, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr17RefreshConfigTaskE", scheduled_time:1755162347785467, pushed_time:1755162373720128, delay:10000000}, total_delay=1000849, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:14.742815] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:14.742947] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:14.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.745131] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=32] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:14.731015] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=30] temporary file module use no blocks [2025-08-14 09:06:14.746689] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:06:14.746736] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=26] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:06:14.746929] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aae10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:14.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:14.746957] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:14.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:14.746979] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e67b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:14.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:14.746992] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:14.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:14.745666] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:14.746346] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=15304] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:14.747583] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=1228] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:14.747599] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623276170368, flush_io_finished_ret_:0, flush_io_finished_round_:26432, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:14.747671] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:603) [360][UserLocAsyncUp0][T0][YB42AC4D0007-00063C495D499F17-0-0] [lt=953694][errcode=-4664] fail to batch process task(ret=-4664) [2025-08-14 09:06:14.747690] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:502) [360][UserLocAsyncUp0][T0][YB42AC4D0007-00063C495D499F17-0-0] [lt=17][errcode=-4664] fail to batch execute task(ret=-4664, tasks.count()=1) [2025-08-14 09:06:14.748626] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=19] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:14.750834] INFO [SERVER] execute_adapter_maintenance (ob_plugin_vector_index_scheduler.cpp:477) [15479][T1002_TimerWK14_TTLTabletMgr][T1002][YB42AC4D0007-00063C4981699C1C-0-0] [lt=176743342] finish generate tenant tablet tasks(ret=0, ret="OB_SUCCESS", tenant_id=1002) [2025-08-14 09:06:13.755699] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [807][T1002_TimerWK3_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=10] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):258, sequence:1}) [2025-08-14 09:06:14.750874] WDIAG alarm_if_necessary (ob_timer_service.cpp:143) [15479][T1002_TimerWK14][T1002][YB42AC4D0007-00063C4981699C1C-0-0] [lt=24][errcode=-4389] timer task's elapsed_time is greater than period(token={this:0x7f991b7a4b98, timer:0x7f99535c9ed0, task:0x7f99004c9710, task_type:"N9oceanbase5share32ObPluginVectorIndexLoadSchedulerE", scheduled_time:1755161866762602, pushed_time:1755161866768067, delay:10000000}, elapsed_time=507982706, period=10000000) [2025-08-14 09:06:14.751186] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=21] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:14.753108] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:14.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.754757] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:14.755845] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:12.751416] EDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [953][T1002_L0_G0][T1002][YB42AC4D0007-00063C4965DA1F35-0-0] [lt=2966][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1026103, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1022786, ObMalloc#start=1, ObMalloc#end=3308, ALLOC_END=4, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x1b511e3c 0x95df1fc 0x95ded4c 0x95de90a 0x95de38d 0x1b899ce2 0x1b83a0a6 0x1b839e80 0x1b863b6d 0x93a2971 0x934006e 0x933e2ee 0x93363f3 0x932e8ab 0x96ac524 0x96a672d 0x96a5033 0x96a366f 0x96622b5 0x9661c70 0x12586874 0x99d034f 0x1fa53591 0x1fa586a3 0x20b3f048 0x20c1e9f2 0x1f11562c 0x9703542 0x9700852 0x970011d 0x15f6f4c7 0x15ebaffc 0x15eba874 0x9343ecf 0x9341ca3 0x92e8803 0x92e30c4 0x92df309 0x92db9ba 0x92d9f5e 0x9292771 0x9285f13 0x927efad 0x926c154 0x12c5a0c9 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:14.757043] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919bfe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aa810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:14.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:14.757078] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:14.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:14.757093] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:14.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:14.757106] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:14.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:14.757117] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:14.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:14.757890] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:14.758340] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:14.761658] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:14.762032] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:14.762419] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:14.762547] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [15479][T1002_TimerWK14_TabletShell][T1002][Y0-0000000000000000-0-0] [lt=323] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-08-14 09:06:14.762598] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [15479][T1002_TimerWK14_TabletShell][T1002][Y0-0000000000000000-0-0] [lt=44] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=2, times=1) [2025-08-14 09:06:14.762614] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [15479][T1002_TimerWK14][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task too much delay in priority queue(token={this:0x7f991b7a5458, timer:0x7f9922ffd748, task:0x7f9922ffd770, task_type:"N9oceanbase7storage10checkpoint16ObEmptyShellTaskE", scheduled_time:1755162362753035, pushed_time:1755162363880598, delay:5000000}, delay_in_priority_queue=1127563, threashold=1000000) [2025-08-14 09:06:14.762635] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [15479][T1002_TimerWK14][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a5458, timer:0x7f9922ffd748, task:0x7f9922ffd770, task_type:"N9oceanbase7storage10checkpoint16ObEmptyShellTaskE", scheduled_time:1755162362753035, pushed_time:1755162363880598, delay:5000000}, delay_in_thread_pool=10870609, threashold=500000) [2025-08-14 09:06:14.766126] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:14.767537] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=22] temporary file module use no blocks [2025-08-14 09:06:14.767558] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=20] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:14.767573] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:14.767588] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623276207228, flush_io_finished_ret_:0, flush_io_finished_round_:26393, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:14.767675] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=19] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:14.767764] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=16] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:06:14.767929] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:14.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:14.769665] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:14.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.769701] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:14.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.770590] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:14.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.775380] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=8306] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25878814720, fast_delete_log_mode=false) [2025-08-14 09:06:14.778196] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=27] gc stale ls task succ [2025-08-14 09:06:14.782090] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=30] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:14.782323] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:14.782978] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:14.783330] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:14.788036] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72707b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a4010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:14.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:14.798118] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:14.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:14.800773] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:14.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.808219] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739acb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196bb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:14.798, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:14.813177] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=23] gc stale ls task succ [2025-08-14 09:06:14.818305] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:14.806, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:14.818338] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:14.809, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:14.828755] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:14.815, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:14.830209] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:14.830246] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:14.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.831245] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:14.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.832494] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:14.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.848865] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:14.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:14.848898] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:14.832, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:14.858927] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:14.841, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:14.859489] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=23] PNIO [ratelimit] time: 1755162374859486, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007615, add_bytes: 0 [2025-08-14 09:06:14.868974] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:14.859, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:14.930283] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:14.930319] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:14.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.931324] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:14.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.932555] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:14.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:14.995037] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:14.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.011145] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:15.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.011191] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:15.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.022112] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:15.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.030327] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:15.030391] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:14.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.031399] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:14.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.032610] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:14.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.035715] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:15.035775] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:15.035823] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:15.035869] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:15.035909] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:15.035936] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:15.035953] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:15.091546] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:15.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.130390] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:15.130416] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=19] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:15.130453] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:15.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.131441] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:15.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.132669] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:15.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.212160] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:15.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.223913] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:15.223964] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:15.223975] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:15.223992] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:15.224002] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:15.224041] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:15.224060] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:15.224079] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:15.224103] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:15.224120] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:15.224141] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:15.224157] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:15.224182] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:15.224196] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:15.224215] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:15.224245] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:15.224263] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:15.224282] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:15.224308] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:15.224485] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:15.224506] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:15.230446] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:15.230511] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:15.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.231512] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:15.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.232732] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:15.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.246360] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:15.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.256414] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:15.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.262484] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:15.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.262531] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:15.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.273555] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:15.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.303749] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:15.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.330506] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:15.330502] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] EASY SLOW: thread: 7f995427d300, lastrun: 1755162250.789339 cost: 124.541160 loop:26617, slowcnt: 924 [2025-08-14 09:06:15.330572] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:15.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.331597] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:15.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.332797] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:15.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.366809] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=28] gc stale ls task succ [2025-08-14 09:06:15.430546] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:15.430637] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:15.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.431639] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:15.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.432860] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:15.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.497782] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:15.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.513861] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:15.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.513900] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:15.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.526047] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:15.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.530591] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:15.530714] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:15.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.531700] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:15.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.532889] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:15.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.594361] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:15.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.630648] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:15.630764] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:15.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.631761] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:15.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.632949] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:15.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.714318] WDIAG [PALF] runTimerTask (block_gc_timer_task.cpp:94) [807][T1002_TimerWK3_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=993910][errcode=-4389] try_recycle_blocks cost too much time(ret=-4389, cost_time_us=1011583, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.77.0.7:2882", log_dir:"/root/ob/observer/store/clog/tenant_1002", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):258, sequence:1}, log_alloc_mgr_:{flying_log_task:1, flying_meta_task:2}}) [2025-08-14 09:06:15.714377] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:97) [807][T1002_TimerWK3_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=59] BlockGCTimerTask success(ret=0, cost_time_us=1011583, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.77.0.7:2882", log_dir:"/root/ob/observer/store/clog/tenant_1002", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):258, sequence:1}, log_alloc_mgr_:{flying_log_task:1, flying_meta_task:2}}) [2025-08-14 09:06:15.714393] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a4818, timer:0x7f99335d9ee0, task:0x7f991c5e9398, task_type:"N9oceanbase4palf16BlockGCTimerTaskE", scheduled_time:1755162358786915, pushed_time:1755162358786979, delay:30000}, delay_in_thread_pool=14933143, threashold=500000) [2025-08-14 09:06:15.714417] WDIAG alarm_if_necessary (ob_timer_service.cpp:143) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4389] timer task's elapsed_time is greater than period(token={this:0x7f991b7a4818, timer:0x7f99335d9ee0, task:0x7f991c5e9398, task_type:"N9oceanbase4palf16BlockGCTimerTaskE", scheduled_time:1755162358786915, pushed_time:1755162358786979, delay:30000}, elapsed_time=1994271, period=30000) [2025-08-14 09:06:15.714888] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:15.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.721109] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:15.724139] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73516b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196f510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.716, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:15.724689] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:15.724723] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:15.724733] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:15.724748] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:15.724761] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:15.724776] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:15.724789] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:15.724805] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:15.724817] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:15.724843] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:15.724860] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:15.724872] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:15.724885] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:15.724898] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:15.724910] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:15.724921] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:15.724944] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:15.724959] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:15.724972] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:15.725152] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:15.725166] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:02:15.738188] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [657][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:06:15.729|total=242.90s) [2025-08-14 09:06:15.730947] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:15.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.731538] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:15.731888] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:15.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.732367] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:15.738188] EDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [657][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 239629173, owner: ObLog, click_count: 5, time dist:[FORMAT_END=239629163, ObMalloc#start=2, ObMalloc#end=2, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ea7c4 0x9818f9c 0x9835405 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:15.732994] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:15.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.734209] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.720, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:15.734246] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.727, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:15.734261] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.727, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:15.734275] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72851b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.721, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:15.734994] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=7808] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:06:15.740419] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:15.742859] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:15.744157] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:15.751263] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=16] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:15.754734] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:15.741240] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:15.733, task_priority:1, with_handle_protected:1}) [374]OB_MALLOC COST TOO MUCH TIME, cost_time=985438, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=985397, ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=993515, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=985478, WASH_BLOCK_END=189, ALLOC_BLOCK#start=7148, ALLOC_BLOCK#end=578], seq:[0, 1, 2, 3, 4] [552]OB_MALLOC COST TOO MUCH TIME, cost_time=990107, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=990086], seq:[0] [718]OB_MALLOC COST TOO MUCH TIME, cost_time=988886, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=988876, ALLOC_CHUNK_END=4], seq:[0, 1] [718]OB_MALLOC COST TOO MUCH TIME, cost_time=988962, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=988905, WASH_BLOCK_END=57], seq:[0, 1, 2] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=990503, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=1510, ALLOC_CHUNK_END=988975, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=990519, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=990511, WASH_BLOCK_END=8], seq:[0, 1, 2] [822]OB_MALLOC COST TOO MUCH TIME, cost_time=990696, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=1686, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=988980, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3, 4, 5] [822]OB_MALLOC COST TOO MUCH TIME, cost_time=990753, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=990709, WASH_BLOCK_END=43], seq:[0, 1, 2] [800]OB_MALLOC COST TOO MUCH TIME, cost_time=988120, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=988088, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [800]OB_MALLOC COST TOO MUCH TIME, cost_time=988138, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=988129, WASH_BLOCK_END=9], seq:[0, 1, 2] [376]OB_MALLOC COST TOO MUCH TIME, cost_time=987695, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=987666, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [376]OB_MALLOC COST TOO MUCH TIME, cost_time=987713, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=987705, WASH_BLOCK_END=8], seq:[0, 1, 2] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=981706, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=981673, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=981723, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=981715, WASH_BLOCK_END=8], seq:[0, 1, 2] [389]OB_MALLOC COST TOO MUCH TIME, cost_time=980529, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=980493, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [389]OB_MALLOC COST TOO MUCH TIME, cost_time=980550, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=980542, WASH_BLOCK_END=8], seq:[0, 1, 2] [15479]OB_MALLOC COST TOO MUCH TIME, cost_time=978861, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=978824, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [15479]OB_MALLOC COST TOO MUCH TIME, cost_time=978885, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=978872, WASH_BLOCK_END=13], seq:[0, 1, 2] [552]OB_MALLOC COST TOO MUCH TIME, cost_time=991106, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=990115], seq:[0, 1] [2025-08-14 09:06:14.767832] EDIAG [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=4935][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1970233, owner: ObLog, click_count: 5, time dist:[FORMAT_END=981219, ObMalloc#start=1, ObMalloc#end=987719, ALLOC_END=1294, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1fce4e91 0x1fce455d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:15.731592] EDIAG [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1007966, owner: ObLog, click_count: 5, time dist:[FORMAT_END=18446, ObMalloc#start=1, ObMalloc#end=988145, ALLOC_END=1374, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x99721a1 0x997a72e 0x2006ed17 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:15.745258] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1845] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:15.760253] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14971][errcode=0] epoll handler too much time :15004 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:15.760244] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=18957][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:15.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:15.757547] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:15.762414] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:15.763079] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][YB42AC4D0007-00063C495F8A1977-0-0] [lt=22][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a7918, timer:0x7f99329e19d0, task:0x7f98f55f1250, task_type:"N9oceanbase5share24ObVectorIndexHistoryTaskE", scheduled_time:1755162369746634, pushed_time:1755162369749522, delay:10000000}, delay_in_thread_pool=5964925, threashold=500000) [2025-08-14 09:06:15.767736] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=23] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:15.767761] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=18] temporary file module use no blocks [833]OB_MALLOC COST TOO MUCH TIME, cost_time=980056, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=980023], seq:[0, 1, 2, 3, 4, 5, 6] [2025-08-14 09:06:15.767768] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:15.767779] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:15.767796] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=17] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623277207381, flush_io_finished_ret_:0, flush_io_finished_round_:26394, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:15.768406] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=16] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:06:15.783282] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:15.831554] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [833]OB_MALLOC COST TOO MUCH TIME, cost_time=980078, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=980076], seq:[0, 1] [2025-08-14 09:06:15.867150] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=20] PNIO [ratelimit] time: 1755162375867148, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007662, add_bytes: 0 [2025-08-14 09:06:15.931612] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:06:15.931637] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [395]OB_MALLOC COST TOO MUCH TIME, cost_time=979344, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=979305], seq:[0, 1, 2, 3, 4, 5, 6] [2025-08-14 09:06:15.991918] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:15.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:16.031673] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:16.131733] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:16.133520] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=13] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:16.231793] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:16.722880] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=490339][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [395]OB_MALLOC COST TOO MUCH TIME, cost_time=979374, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=979372], seq:[0, 1] [2025-08-14 09:06:16.243545] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:16.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:16.263399] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a7058, timer:0x7f994b1dfb70, task:0x7f98ebbf7c30, task_type:"N9oceanbase8observer26ObTableLoadResourceManager21ObRefreshAndCheckTaskE", scheduled_time:1755161774744853, pushed_time:0, delay:30000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=601518544, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:14.729328] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:943) [15497][T1_TimerWK16_FreInfoReload][T1][YB42AC4D0007-00063C4982499C16-0-0] [lt=82][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f9839617338, id:3221590981, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f98b4f97440}) [2025-08-14 09:06:16.721047] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:16.732263] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:15.744271] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.737, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.739393] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task too much delay in priority queue(token={this:0x7f991b7a6178, timer:0x7f99200ff180, task:0x7f99200ff220, task_type:"N9oceanbase8observer18ObTableLoadService21ObClientTaskPurgeTaskE", scheduled_time:1755162300760824, pushed_time:1755162369767600, delay:1000000}, delay_in_priority_queue=69006776, threashold=1000000) [2025-08-14 09:06:16.739411] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a6178, timer:0x7f99200ff180, task:0x7f99200ff220, task_type:"N9oceanbase8observer18ObTableLoadService21ObClientTaskPurgeTaskE", scheduled_time:1755162300760824, pushed_time:1755162369767600, delay:1000000}, delay_in_thread_pool=5995503, threashold=500000) [2025-08-14 09:06:16.736483] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=478631][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:16.243|207=479.36ms|210=0us|222=2us|total=495.96ms) [2025-08-14 09:06:16.739334] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=993556][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:15.744|207=995.06ms|210=0us|222=1us|total=995.62ms) [2025-08-14 09:06:16.740348] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:15.747721] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:16.740394] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=991167] temporary file module use no blocks [2025-08-14 09:06:16.740402] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:16.740415] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:15.756568] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:16.740427] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623277187395, flush_io_finished_ret_:0, flush_io_finished_round_:26433, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:16.741027] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=982970][errcode=0] epoll handler too much time :984467 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:14.767832] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=4935] blacklist_loop exec finished(cost_time=16, is_enabled=true, send_cnt=0) [2025-08-14 09:06:16.535483] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fe030, caller:ob_table_lock_service.cpp:start:276, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE6$_1273, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe310, time_wheel:0x7f99535f3990, thread_pool:0x7f99535f3170, is_running:1, time_interval:10.00s, expected_run_time:09:06:16.426, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:16.741067] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=205271][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:16.535|207=205.59ms|210=1us|222=1us|total=205.60ms) [2025-08-14 09:06:15.731592] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=27] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:16.741100] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=1007966] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:16.741109] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:15.757794] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:16.741129] WDIAG ev_invoke_pending (ev.c:2136) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=981848][errcode=0] epoll handler too much time :983342 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:15.747374] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:06:16.741158] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=992268] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:06:15.747561] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=18] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:16.741372] WDIAG ev_invoke_pending (ev.c:2136) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=992308][errcode=0] epoll handler too much time :993814 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:15.749102] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=4457] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:06:15.761461] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:16.742337] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=979393][errcode=0] epoll handler too much time :980884 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [574]OB_MALLOC COST TOO MUCH TIME, cost_time=978749, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=978718], seq:[0, 1, 2, 3, 4, 5, 6] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=978761, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=978760], seq:[0, 1] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=978717, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=978689], seq:[0, 1, 2, 3, 4, 5, 6] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=978730, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=978728], seq:[0, 1] [1057]OB_MALLOC COST TOO MUCH TIME, cost_time=978201, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=978174], seq:[0, 1, 2, 3, 4, 5, 6] [1057]OB_MALLOC COST TOO MUCH TIME, cost_time=978215, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=978214], seq:[0, 1] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=975802, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=975758], seq:[0, 1, 2, 3, 4, 5, 6] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=975812, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=975811], seq:[0, 1] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=973698, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=16, ALLOC_CHUNK_END=973651], seq:[0, 1, 2, 3, 4, 5, 6] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=973710, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=973708], seq:[0, 1] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=973558, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=973511], seq:[0, 1, 2, 3, 4, 5, 6] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=973573, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=973571], seq:[0, 1] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=973509, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=973464], seq:[0, 1, 2, 3, 4, 5, 6] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=973530, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=973529], seq:[0, 1] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=970111, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=970067], seq:[0, 1, 2, 3, 4, 5, 6] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=970122, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=970121], seq:[0, 1] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=964103, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=964064], seq:[0, 1, 2, 3, 4, 5, 6] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=964114, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=964113], seq:[0, 1] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=959727, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=16, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=959677], seq:[0, 1, 2, 3, 4, 5, 6] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=959742, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=959740], seq:[0, 1] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=958826, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=958786], seq:[0, 1, 2, 3, 4, 5, 6] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=958836, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=958836], seq:[0, 1] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=956881, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=956848], seq:[0, 1, 2, 3, 4, 5, 6] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=956895, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=956893], seq:[0, 1] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=928988, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=928951], seq:[0, 1, 2, 3, 4, 5, 6] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=928999, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=928998], seq:[0, 1] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=910342, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=910287], seq:[0, 1, 2, 3, 4, 5, 6] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=910351, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=910350], seq:[0, 1] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=909262, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=909221, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=809194, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=809159], seq:[0, 1, 2, 3, 4, 5, 6] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=809213, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=809211], seq:[0, 1] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=517657, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=18, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=517602, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=409613, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=409573], seq:[0, 1, 2, 3, 4, 5, 6] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=409626, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=409623], seq:[0, 1] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=376390, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=376344, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=705532, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=705497, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=379113, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=376402, WASH_BLOCK_END=2711], seq:[0, 1, 2] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=708322, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=705542, WASH_BLOCK_END=2779], seq:[0, 1, 2] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=521398, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=517668, WASH_BLOCK_END=3693, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=37], seq:[0, 1, 2, 3, 4] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=914886, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=909272, WASH_BLOCK_END=2844, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2766], seq:[0, 1, 2, 3, 4] [2025-08-14 09:06:15.762137] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:15.765156] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:15.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:16.742491] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=975855][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:15.765|207=977.34ms|210=0us|222=2us|total=977.36ms) [2025-08-14 09:06:15.767340] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:16.742541] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=973728][errcode=0] epoll handler too much time :975224 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:15.768270] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=24] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:16.743365] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:15.777904] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:15.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:16.743515] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=964149][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:15.777|207=965.61ms|210=0us|222=1us|total=965.63ms) [2025-08-14 09:06:15.782332] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:16.743552] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=959767][errcode=0] epoll handler too much time :961229 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:15.783309] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=30] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:16.743614] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=958854][errcode=0] epoll handler too much time :960312 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:15.785264] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:16.743627] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=956915][errcode=0] epoll handler too much time :958374 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:15.813241] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:06:15.831955] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:15.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:16.743723] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=910387][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:15.831|207=911.77ms|210=1us|222=1us|total=911.78ms) [2025-08-14 09:06:15.933346] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:15.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:16.743837] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=809263][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:15.933|207=810.49ms|210=0us|222=2us|total=810.51ms) [2025-08-14 09:06:16.700428] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEC1-0-0] [lt=30] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:06:16.744361] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:16.745271] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:16.741116] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:16.746296] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5170] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:16.746320] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=24] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:16.746329] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:16.746341] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:16.746349] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:16.746363] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:06:16.746374] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:06:16.333583] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] EASY SLOW: thread: 7f9954779300, lastrun: 1755162375.782322 cost: 0.551259 loop:26726, slowcnt: 5 [2025-08-14 09:06:16.366882] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=22] gc stale ls task succ [2025-08-14 09:06:15.749070] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=4258] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:06:15.778505] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=19] gc stale ls task succ [2025-08-14 09:06:16.037273] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:15.749092] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [953][T1002_L0_G0][T1002][YB42AC4D0007-00063C4965DA1F35-0-0] [lt=1026103][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:06:16.746791] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=708395] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:16.746837] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:16.746877] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:16.746919] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:16.746931] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:16.746944] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:15.750099] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=3647] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:06:16.747458] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=19] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:16.225359] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:16.747620] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=521461][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:16.747648] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:16.747663] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:16.749145] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a41f8, timer:0x7f9922df93b0, task:0x7f9922ffab60, task_type:"N9oceanbase3sql27ObSqlPlanMonitorRecycleTaskE", scheduled_time:1755162370740551, pushed_time:1755162370744345, delay:1000000}, delay_in_thread_pool=5995083, threashold=500000) [2025-08-14 09:06:16.746747] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [953][T1002_L0_G0][T1002][YB42AC4D0007-00063C4965DA1F35-0-0] [lt=996144][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:06:16.749281] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][YB42AC4D0007-00063C495F8A1978-0-0] [lt=19][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a5c38, timer:0x7f9935f93f20, task:0x7f99200fe2d0, task_type:"N9oceanbase8datadict17ObDataDictServiceE", scheduled_time:1755162370749071, pushed_time:1755162370754536, delay:1000000}, delay_in_thread_pool=5994636, threashold=500000) [2025-08-14 09:06:15.833056] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:15.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:16.749369] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=914925][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:15.833|207=916.31ms|210=0us|222=2us|total=916.33ms) [2025-08-14 09:06:16.749387] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4389] timer task use too much time(end=1755162376749387, start=1755162375833055, delta=916332) [2025-08-14 09:06:16.748060] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4389] timer task use too much time(end=1755162376742511, start=1755162375765148, delta=977363) [2025-08-14 09:06:16.749420] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1354][errcode=0] task scheduled too delay(scheduled_delay=990284, expected_run_ts_us_=1755162375759134, ObClockGenerator::getRealClock()=1755162376749419, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:15.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:16.749440] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:15.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:16.749461] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=660144, expected_run_ts_us_=1755162376089316, ObClockGenerator::getRealClock()=1755162376749461, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:16.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:16.749474] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:16.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:16.749487] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=545190, expected_run_ts_us_=1755162376204297, ObClockGenerator::getRealClock()=1755162376749487, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:16.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:16.749501] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:16.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:16.748064] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task use too much time(end=1755162376743535, start=1755162375777901, delta=965634) [2025-08-14 09:06:16.749537] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1470][errcode=0] task scheduled too delay(scheduled_delay=957188, expected_run_ts_us_=1755162375792349, ObClockGenerator::getRealClock()=1755162376749537, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:15.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:16.749550] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:15.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:16.748066] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4389] timer task use too much time(end=1755162376743739, start=1755162375831954, delta=911785) [2025-08-14 09:06:16.748069] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=-4389] timer task use too much time(end=1755162376743863, start=1755162375933344, delta=810519) [2025-08-14 09:06:16.749623] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [953][T1002_L0_G0][T1002][YB42AC4D0007-00063C4965DA1F35-0-0] [lt=2500] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162230264152]}) [2025-08-14 09:06:16.748077] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4389] timer task use too much time(end=1755162376741083, start=1755162376535479, delta=205604) [2025-08-14 09:06:16.748251] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=3021][errcode=-4389] timer task use too much time(end=1755162376739509, start=1755162376243543, delta=495966) [2025-08-14 09:06:16.749916] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1660][errcode=0] task scheduled too delay(scheduled_delay=505282, expected_run_ts_us_=1755162376244633, ObClockGenerator::getRealClock()=1755162376749915, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:16.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:16.749933] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:16.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:16.749954] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:16.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:16.748253] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=554][errcode=-4389] timer task use too much time(end=1755162376739890, start=1755162375744269, delta=995621) [2025-08-14 09:06:16.749982] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1724][errcode=0] task scheduled too delay(scheduled_delay=1018506, expected_run_ts_us_=1755162375731475, ObClockGenerator::getRealClock()=1755162376749981, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.749996] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750010] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=1003063, expected_run_ts_us_=1755162375746946, ObClockGenerator::getRealClock()=1755162376750009, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750032] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750044] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1001319, expected_run_ts_us_=1755162375748725, ObClockGenerator::getRealClock()=1755162376750044, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891807210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750056] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891807210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750068] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1000480, expected_run_ts_us_=1755162375749587, ObClockGenerator::getRealClock()=1755162376750068, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750079] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750092] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=1000519, expected_run_ts_us_=1755162375749573, ObClockGenerator::getRealClock()=1755162376750092, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750105] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750118] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=990474, expected_run_ts_us_=1755162375759643, ObClockGenerator::getRealClock()=1755162376750118, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d67b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750130] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d67b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750144] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=967358, expected_run_ts_us_=1755162375782785, ObClockGenerator::getRealClock()=1755162376750143, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891807090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750155] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891807090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750167] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=962997, expected_run_ts_us_=1755162375787169, ObClockGenerator::getRealClock()=1755162376750167, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739aa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750178] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739aa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750190] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=958584, expected_run_ts_us_=1755162375791606, ObClockGenerator::getRealClock()=1755162376750190, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.791, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750203] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.791, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750215] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=958842, expected_run_ts_us_=1755162375791373, ObClockGenerator::getRealClock()=1755162376750215, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ecf10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.791, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750227] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ecf10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.791, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750250] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] task scheduled too delay(scheduled_delay=958225, expected_run_ts_us_=1755162375792025, ObClockGenerator::getRealClock()=1755162376750250, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750261] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750275] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=950946, expected_run_ts_us_=1755162375799328, ObClockGenerator::getRealClock()=1755162376750275, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.799, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750287] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.799, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750298] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=949054, expected_run_ts_us_=1755162375801244, ObClockGenerator::getRealClock()=1755162376750298, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750310] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750322] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=946556, expected_run_ts_us_=1755162375803766, ObClockGenerator::getRealClock()=1755162376750322, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7351e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196f990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.803, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750335] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7351e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196f990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.803, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750347] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=946280, expected_run_ts_us_=1755162375804067, ObClockGenerator::getRealClock()=1755162376750347, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a22b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750358] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a22b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750370] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=943101, expected_run_ts_us_=1755162375807269, ObClockGenerator::getRealClock()=1755162376750370, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750382] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750395] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=923472, expected_run_ts_us_=1755162375826923, ObClockGenerator::getRealClock()=1755162376750395, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ebe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.826, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750407] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ebe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.826, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750419] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=925661, expected_run_ts_us_=1755162375824757, ObClockGenerator::getRealClock()=1755162376750419, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7350530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ea90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.824, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750430] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7350530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ea90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.824, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750443] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=884449, expected_run_ts_us_=1755162375865994, ObClockGenerator::getRealClock()=1755162376750443, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fbbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750455] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fbbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750468] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=876987, expected_run_ts_us_=1755162375873480, ObClockGenerator::getRealClock()=1755162376750468, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72860b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.873, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750480] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72860b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:15.873, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750498] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=421296, expected_run_ts_us_=1755162376329201, ObClockGenerator::getRealClock()=1755162376750498, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7faa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:16.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750509] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7faa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:16.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750522] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=420781, expected_run_ts_us_=1755162376329741, ObClockGenerator::getRealClock()=1755162376750522, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72842b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:16.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750533] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72842b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:16.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750550] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=136218, expected_run_ts_us_=1755162376614331, ObClockGenerator::getRealClock()=1755162376750549, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eef90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:16.614, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750561] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eef90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:16.614, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750573] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=135754, expected_run_ts_us_=1755162376614819, ObClockGenerator::getRealClock()=1755162376750573, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eaf30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:16.614, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750586] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eaf30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:16.614, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750601] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:16.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750613] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:16.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750624] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919827b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:16.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.750658] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4389] timer task too much delay in priority queue(token={this:0x7f991b7a4578, timer:0x7f9966a63f10, task:0x7f9922e05028, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr17RefreshConfigTaskE", scheduled_time:1755162347785467, pushed_time:1755162373720128, delay:10000000}, delay_in_priority_queue=25934661, threashold=1000000) [2025-08-14 09:06:16.750670] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a4578, timer:0x7f9966a63f10, task:0x7f9922e05028, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr17RefreshConfigTaskE", scheduled_time:1755162347785467, pushed_time:1755162373720128, delay:10000000}, delay_in_thread_pool=3029185, threashold=500000) [2025-08-14 09:06:16.750694] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [807][T1002_TimerWK3_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=9] ====== check clog disk timer task ====== [2025-08-14 09:06:16.750714] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [807][T1002_TimerWK3_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=16] get_disk_usage(ret=0, capacity(MB):=24883, used(MB):=8642) [2025-08-14 09:06:16.750735] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a5298, timer:0x7f9922ffd5d0, task:0x7f9922ffd618, task_type:"N9oceanbase7storage10checkpoint19ObCheckPointService24ObCheckClogDiskUsageTaskE", scheduled_time:1755162375720058, pushed_time:1755162375724523, delay:2000000}, delay_in_thread_pool=1026165, threashold=500000) [2025-08-14 09:06:16.750759] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [807][T1002_TimerWK3_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=12] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):258, sequence:1}) [2025-08-14 09:06:16.750785] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a4818, timer:0x7f99335d9ee0, task:0x7f991c5e9398, task_type:"N9oceanbase4palf16BlockGCTimerTaskE", scheduled_time:1755162375744441, pushed_time:1755162375744501, delay:30000}, delay_in_thread_pool=1006253, threashold=500000) [2025-08-14 09:06:16.750859] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task too much delay in priority queue(token={this:0x7f991b7a5df8, timer:0x7f99200ff180, task:0x7f99200ff1c0, task_type:"N9oceanbase8observer18ObTableLoadService15ObHeartBeatTaskE", scheduled_time:1755162321771928, pushed_time:1755162376739603, delay:10000000}, delay_in_priority_queue=54967675, threashold=1000000) [2025-08-14 09:06:16.750882] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] timer task too much delay in priority queue(token={this:0x7f991b7a4658, timer:0x7f9966a63f10, task:0x7f9922e05038, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr12TabletGCTaskE", scheduled_time:1755162356753558, pushed_time:1755162376750694, delay:20000}, delay_in_priority_queue=19997136, threashold=1000000) [2025-08-14 09:06:16.752485] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=12] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:16.749582] WDIAG [SQL] open_plan (ob_result_set.cpp:104) [830][T1002_GCCollect][T1001][YB42AC4D0007-00063C496069AFFA-0-0] [lt=12][errcode=-4012] query is timeout(ret=-4012, timeout_ts=1755161964778561, start_time=1755161936720938) [2025-08-14 09:06:16.754737] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:16.757208] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:16.758320] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2762] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:16.758394] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:16.748121] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=453][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:16.758452] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10309][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:16.758466] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:16.758481] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:16.758492] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:16.758504] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:16.758526] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:16.758537] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:16.758550] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:16.758562] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:16.758573] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:16.758584] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:16.758596] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:16.758609] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:16.758621] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:16.758779] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:16.758793] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:16.749641] INFO [STORAGE.TRANS] get_gts (ob_gts_source.cpp:257) [953][T1002_L0_G0][T1002][YB42AC4D0007-00063C4965DA1F35-0-0] [lt=15] push gts task success(*task={:""}) [2025-08-14 09:06:16.760834] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:16.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.761309] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:16.761756] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] timer task too much delay in priority queue(token={this:0x7f991b7a5d18, timer:0x7f99200ff180, task:0x7f99200ff1a8, task_type:"N9oceanbase8observer18ObTableLoadService17ObCheckTenantTaskE", scheduled_time:1755162337763401, pushed_time:1755162376750879, delay:1000000}, delay_in_priority_queue=38987478, threashold=1000000) [2025-08-14 09:06:16.761789] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4389] timer task too much delay in priority queue(token={this:0x7f991b7a4498, timer:0x7f9966a63f10, task:0x7f9922e05010, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr11TableGCTaskE", scheduled_time:1755162373740116, pushed_time:1755162376750901, delay:20000}, delay_in_priority_queue=3010785, threashold=1000000) [2025-08-14 09:06:16.762476] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:16.764636] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:16.767892] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:16.767946] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=50] temporary file module use no blocks [2025-08-14 09:06:16.767955] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:16.767966] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:16.767979] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623278207594, flush_io_finished_ret_:0, flush_io_finished_round_:26395, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:16.768503] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=8] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:06:16.769582] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:16.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:16.769615] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:16.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:16.770894] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72742b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:16.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.770923] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919840b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:16.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.770939] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180ef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:16.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.770954] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180ecb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:16.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.770969] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:16.767, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.770986] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:16.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.778525] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=966713] gc stale ls task succ [2025-08-14 09:06:16.779726] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:16.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:16.780957] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919925b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196d890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:16.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.780985] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:16.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.781000] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:16.777, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.781631] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:16.781683] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:16.781696] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:16.790986] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728ca30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7291590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:16.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.801049] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739ee10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:16.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.809887] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:16.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:16.811110] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ebd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:16.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.811136] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739ec90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:16.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.813318] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=929020] gc stale ls task succ [2025-08-14 09:06:16.821131] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72874b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:16.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.822942] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:16.831151] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:16.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.831183] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739dc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:16.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.849477] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:16.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:16.849668] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1519][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:16.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:16.849712] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1532][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:16.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:16.871388] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fc8b0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcb90, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:1, time_interval:3.00s, expected_run_time:09:06:16.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:16.874717] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=14] PNIO [ratelimit] time: 1755162376874715, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007567, add_bytes: 0 [2025-08-14 09:06:16.901499] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a59b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:16.895, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:16.923004] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:16.949490] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:16.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:16.949729] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:16.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:16.949742] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:16.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:16.991302] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:16.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.011006] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:17.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.011048] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:17.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.020972] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:17.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.023100] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=35] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:06:17.023113] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:17.047269] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:17.047325] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:17.047370] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:17.047411] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:17.047448] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:17.047460] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:17.047471] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:17.049555] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:16.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.049766] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:16.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.049798] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:16.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.091460] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:17.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.123127] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:17.134208] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=21] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:17.149620] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:17.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.149826] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:17.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.149866] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:17.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.212125] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:17.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.223190] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:17.242688] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:17.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.249683] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:17.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.249870] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:17.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.249918] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:17.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.252762] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:17.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.258943] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:17.258975] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:17.258990] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:17.259004] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:17.259019] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:17.259044] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:17.259066] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:17.259083] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:17.259094] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:17.259114] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:17.259129] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:17.259141] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:17.259163] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:17.259177] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:17.259188] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:17.259209] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:17.259268] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:17.259283] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:17.259293] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:17.259456] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:17.259475] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:17.262334] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:17.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.262368] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:17.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.272364] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:17.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.302491] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:17.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.323253] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:17.346740] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=412208][errcode=0] EASY SLOW: thread: 7f995427d300, lastrun: 1755162250.789339 cost: 126.557399 loop:26617, slowcnt: 940 [2025-08-14 09:06:17.349704] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:17.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.349909] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:17.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.349988] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:17.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.366947] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=379175] gc stale ls task succ [2025-08-14 09:06:17.423317] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:17.449768] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:17.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.449971] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:17.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.450061] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:17.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.452280] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=776415][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc41f8, timer:0x7f9970f40f80, task:0x564d6d07c3b0, task_type:"N9oceanbase5share22ObDeviceCredentialTaskE", scheduled_time:1755162016246491, pushed_time:0, delay:1200000000}, running_task_token={this:0x7f9970fc5fb8, timer:0x7f9970f40f80, task:0x564d727182f0, task_type:"N9oceanbase5share16ObAshRefreshTaskE", scheduled_time:1755161611717613, pushed_time:1755161614757742, delay:120000000}, total_delay=361205788, threashold=1000000, tenant_id=500) [2025-08-14 09:06:17.494065] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:17.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.513700] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:17.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.513736] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:17.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.523379] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:17.523849] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:17.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.549831] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:17.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.550045] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:17.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.550086] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:17.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.594271] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:17.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.623443] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:17.649878] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:17.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.650112] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:17.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.650121] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:17.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.714944] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:17.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.720716] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3599] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:17.723570] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:16.782895] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:17.725691] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=941371][errcode=0] epoll handler too much time :942806 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:17.726098] WDIAG [SQL] open (ob_result_set.cpp:137) [830][T1002_GCCollect][T1001][YB42AC4D0007-00063C496069AFFA-0-0] [lt=4791][errcode=-4012] execute plan failed(ret=-4012) [2025-08-14 09:06:17.732265] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7014] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:17.736492] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:17.726, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:17.736531] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739eb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:17.721, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:17.738524] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=237] finish to resize log external storage handler(current_ts=1755162376770641, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-08-14 09:06:17.738559] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=33] snapshot_traffic_info once [2025-08-14 09:06:17.740337] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:16.746426] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1001, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:8571658, max_lsn:7323687369, max_scn:{val:1755161749741193002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:7278610047}, reuse_lsn:{lsn:7323686131}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:7281744329}}, last_submit_lsn:{lsn:7323686131}, last_submit_end_lsn:{lsn:7323687369}, last_submit_log_id:8571658, last_submit_log_pid:49, max_flushed_lsn:{lsn:7323686009}, max_flushed_end_lsn:{lsn:7323686131}, max_flushed_log_pid:49, committed_end_lsn:{lsn:7323686009}, last_slide_log_id:8571656, last_slide_scn:{val:1755161747742105001, v:0}, last_slide_lsn:{lsn:7323685887}, last_slide_end_lsn:{lsn:7323686009}, last_slide_log_pid:49, last_slide_log_accum_checksum:2505458373, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f990112a530}, [StateMgr]={this:0x7f990112e9b0, self:"172.77.0.7:2882", palf_id:1001, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"PENDING", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:49}, leader:"0.0.0.0:0", leader_epoch:-1, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:7323687369}, scan_disk_log_finished:true, last_check_start_id:8571657, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161767744317, palf_role_change_cb:0x7f9901135470, allow_vote:true}, [ConfigMgr]={palf_id:1001, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:49, prev:{version:1, config:{config_version:{proposal_id:48, config_seq:47}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:49, config_seq:48}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_mode_pid:49}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_end_lsn:{lsn:7323685240}, prev_mode_pid:49}, reconfig_barrier:{prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_end_lsn:{lsn:7323685240}, prev_mode_pid:49}, persistent_config_version:{proposal_id:49, config_seq:48}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755161739755312, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755161739755312, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f990112bc70}, [ModeMgr]={palf_id:1001, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, accepted_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1001, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1753069671378640001, v:0}, base_lsn_for_block_gc:{lsn:7247314944}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:49}, log_config_meta:{version:3, proposal_id:49, prev:{version:1, config:{config_version:{proposal_id:48, config_seq:47}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:49, config_seq:48}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_mode_pid:49}, log_snapshot_meta:{version:2, base_lsn:{lsn:7247314944}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1753069671378640000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}}, log_meta_storage:{log_tail:{lsn:765952}, readable_log_tail:{lsn:765952}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7381524480}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/meta", dir_fd:232, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66338816, block_header_serialize_buf_:0x7f9901130bb8, flashback_version:0}, log_storage:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, palf_epoch:2, last_purge_throttling_ts:1755161769777679, this:0x7f990112f0f0}, [Reconfirm]={palf_id:1001, self:"172.77.0.7:2882", state:"WAITING_LOG_FLUSHED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:1755161769777679, this:0x7f990112eb30}) [2025-08-14 09:06:17.741065] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=605] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:17.741088] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=18] temporary file module use no blocks [2025-08-14 09:06:17.741098] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:17.741109] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:17.741121] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623279180801, flush_io_finished_ret_:0, flush_io_finished_round_:26434, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:17.742509] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=978784] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:17.743332] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:17.743407] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=973603] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:17.745213] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:17.745520] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:17.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.746043] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:17.746554] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:17.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:17.749507] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=10] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:16.746426] EDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1001611, owner: ObLog, click_count: 5, time dist:[FORMAT_END=993043, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0xfc0e73d 0x993f560 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:17.749944] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:17.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.750183] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:17.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.750195] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task too much delay in priority queue(token={this:0x7f991b7a5fb8, timer:0x7f99200ff180, task:0x7f99200ff1f0, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162353720083, pushed_time:1755162376761819, delay:1000000}, delay_in_priority_queue=23041736, threashold=1000000) [2025-08-14 09:06:17.750223] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:17.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.750283] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a4658, timer:0x7f9966a63f10, task:0x7f9922e05038, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr12TabletGCTaskE", scheduled_time:1755162376770891, pushed_time:1755162376771878, delay:20000}, delay_in_thread_pool=978346, threashold=500000) [2025-08-14 09:06:17.755573] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=154] [COMPACTION DAG STATUS] (start_time=1755162367731949, end_time=1755162377755572, dag_status={}) [2025-08-14 09:06:17.755606] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=29] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:17.755618] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:17.755627] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:17.755636] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:17.755635] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:17.755644] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:17.755670] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=26] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:17.755679] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:17.755688] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:17.755696] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:17.755707] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:06:17.755715] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:06:17.755768] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:17.755878] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:17.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.757256] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:17.758281] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=37] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:17.758328] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:17.758948] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [830][T1002_GCCollect][T1001][YB42AC4D0007-00063C496069AFFA-0-0] [lt=24][errcode=-4012] open result set failed(ret=-4012) [2025-08-14 09:06:17.759669] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:17.759699] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:17.759711] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:17.759723] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:17.759736] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:17.759751] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:17.759769] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:17.759785] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:17.759796] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:17.759807] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:17.759824] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:17.759836] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:17.759849] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:17.759867] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:17.759878] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:17.759890] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:17.759902] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:17.759916] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:17.759929] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:17.760050] WDIAG [STORAGE.TRANS] wait (ob_ts_mgr.cpp:195) [953][T1002_L0_G0][T1002][YB42AC4D0007-00063C4965DA1F35-0-0] [lt=9573][errcode=-4012] ObTsSyncGetTsCbTask cond wait failed(ret=-4012) [2025-08-14 09:06:17.760074] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:941) [953][T1002_L0_G0][T1002][YB42AC4D0007-00063C4965DA1F35-0-0] [lt=20][errcode=-4012] failed to wait ObTsSyncGetTsCbTask(ret=-4012, tenant_id=1002, timeout_us=1000000) [2025-08-14 09:06:17.760087] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1832) [953][T1002_L0_G0][T1002][YB42AC4D0007-00063C4965DA1F35-0-0] [lt=11][errcode=-4012] get gts fail(ret=-4012, timeout_us=1000000, request_time=[mts=1755162132743550]) [2025-08-14 09:06:17.760099] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1844) [953][T1002_L0_G0][T1002][YB42AC4D0007-00063C4965DA1F35-0-0] [lt=10][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1755164762712712, request_time=[mts=1755162132743550]) [2025-08-14 09:06:17.760116] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:17.760131] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:17.760111] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:662) [953][T1002_L0_G0][T1002][YB42AC4D0007-00063C4965DA1F35-0-0] [lt=9][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f98b290fb20, tx_id:{txid:0}, state:1, addr:"172.77.0.7:2882", tenant_id:1002, session_id:3221590962, assoc_session_id:3221590962, client_sid:3221590962, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1755162130764939, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180067074, seq_base:1755162130755646, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}) [2025-08-14 09:06:17.761453] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:17.765700] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:17.766689] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:17.750, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:17.766724] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:17.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:17.766737] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d80b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:17.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:17.766747] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:17.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:17.766763] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891992330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196d710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:17.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:17.766776] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:17.760185] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:943) [953][T1002_L0_G0][T1002][YB42AC4D0007-00063C4965DA1F35-0-0] [lt=71][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f98f5d05a98, id:3221590962, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1002, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f98b290fb20}) [2025-08-14 09:06:17.762514] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:17.755427] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=13] get_min_unreplayed_log_info(lsn={lsn:1604088821}, scn={val:1755100948296325001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873479, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:12492179, ret_code_:0}}, next_to_submit_lsn_:{lsn:1604088821}, next_to_submit_scn_:{val:1755100948296325001, v:0}, base_lsn_:{lsn:1409200128}, base_scn_:{val:1753840802540914000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:1604088821}, end_lsn:{lsn:1604088821}, read_buf:{buf_len_:3747840, buf_:0x7f98f6a05000}, block_size:67104768, log_storage_:0x7f990bfebbf0, log_storage_:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:42, read_io_size:147709952, read_disk_cost_ts:1061450, start_lsn:{lsn:1409200128}}}}}}) [2025-08-14 09:06:17.768349] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1085) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=12899] replay status is not follower(min_unreplayed_lsn={lsn:1604088821}, base_lsn={lsn:1409200128}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873479, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:12492179, ret_code_:0}}, next_to_submit_lsn_:{lsn:1604088821}, next_to_submit_scn_:{val:1755100948296325001, v:0}, base_lsn_:{lsn:1409200128}, base_scn_:{val:1753840802540914000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:1604088821}, end_lsn:{lsn:1604088821}, read_buf:{buf_len_:3747840, buf_:0x7f98f6a05000}, block_size:67104768, log_storage_:0x7f990bfebbf0, log_storage_:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:42, read_io_size:147709952, read_disk_cost_ts:1061450, start_lsn:{lsn:1409200128}}}}}}) [2025-08-14 09:06:17.768407] INFO [CLOG] operator() (ob_log_replay_service.cpp:1678) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=57] get_replay_process success(id={id:1}, submitted_log_size=194888693, unsubmitted_log_size=0, replayed_log_size=194888693, unreplayed_log_size=0) [2025-08-14 09:06:17.768422] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=10] get_min_unreplayed_log_info(lsn={lsn:7323685240}, scn={val:1755100946796590002, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873485, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:8946764, ret_code_:0}}, next_to_submit_lsn_:{lsn:7323685240}, next_to_submit_scn_:{val:1755100946796590002, v:0}, base_lsn_:{lsn:7180210176}, base_scn_:{val:1753840803166611000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:7323685240}, end_lsn:{lsn:7323685240}, read_buf:{buf_len_:3747840, buf_:0x7f98f7a05000}, block_size:67104768, log_storage_:0x7f9901131bf0, log_storage_:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:28, read_io_size:98508800, read_disk_cost_ts:645339, start_lsn:{lsn:7180210176}}}}}}) [2025-08-14 09:06:17.768467] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1085) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=44] replay status is not follower(min_unreplayed_lsn={lsn:7323685240}, base_lsn={lsn:7180210176}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873485, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:8946764, ret_code_:0}}, next_to_submit_lsn_:{lsn:7323685240}, next_to_submit_scn_:{val:1755100946796590002, v:0}, base_lsn_:{lsn:7180210176}, base_scn_:{val:1753840803166611000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:7323685240}, end_lsn:{lsn:7323685240}, read_buf:{buf_len_:3747840, buf_:0x7f98f7a05000}, block_size:67104768, log_storage_:0x7f9901131bf0, log_storage_:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:28, read_io_size:98508800, read_disk_cost_ts:645339, start_lsn:{lsn:7180210176}}}}}}) [2025-08-14 09:06:17.768516] INFO [CLOG] operator() (ob_log_replay_service.cpp:1678) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=48] get_replay_process success(id={id:1001}, submitted_log_size=143475064, unsubmitted_log_size=0, replayed_log_size=143475064, unreplayed_log_size=0) [2025-08-14 09:06:17.768556] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:174) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=8] dump tenant replay process(tenant_id=1002, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=322, replayed_log_size(MB)=322, last_submitted_log_size(MB)=322, last_replayed_log_size(MB)=322, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-08-14 09:06:17.768572] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a49d8, timer:0x7f9922d47110, task:0x7f9922eb44d0, task_type:"N9oceanbase10logservice17ReplayProcessStatE", scheduled_time:1755162376780571, pushed_time:1755162376781948, delay:10000000}, delay_in_thread_pool=968349, threashold=500000) [2025-08-14 09:06:17.768634] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a4818, timer:0x7f99335d9ee0, task:0x7f991c5e9398, task_type:"N9oceanbase4palf16BlockGCTimerTaskE", scheduled_time:1755162376780795, pushed_time:1755162376781951, delay:30000}, delay_in_thread_pool=986650, threashold=500000) [2025-08-14 09:06:17.769312] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:17.768020] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=20] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:17.769358] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=1331] temporary file module use no blocks [2025-08-14 09:06:17.769367] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:17.769377] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=8] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:17.769388] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623279207769, flush_io_finished_ret_:0, flush_io_finished_round_:26396, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:17.769375] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=17] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:06:17.769551] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1001611] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [2025-08-14 09:06:17.769630] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=72] current log_task status(palf_id=1001, self="172.77.0.7:2882", log_id=8571657, log_task={header:{begin_lsn:{lsn:7323686009}, end_lsn:{lsn:7323686131}, log_id:8571657, min_scn:{val:1755161749741193000, v:0}, max_scn:{val:1755161749741193000, v:0}, data_len:66, proposal_id:49, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:7323686009}, data_checksum:2222604892, accum_checksum:1531236791, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1755161749745909, freeze_ts:1755161749745914, submit_ts:1755161749745940, flushed_ts:1755161785758706, gen_to_freeze cost time:5, gen_to_submit cost time:31, submit_to_flush cost time:36012766}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [2025-08-14 09:06:17.775624] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:17.778543] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=45] gc stale ls task succ [2025-08-14 09:06:17.781798] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=58] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:17.782546] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:17.782558] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:17.782727] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:17.786762] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:17.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:17.786798] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72751b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:17.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:17.786809] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d31b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:17.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:17.786826] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73959b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:17.773, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:17.796823] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:17.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:17.796856] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72856b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:17.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:17.796870] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:17.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:17.806882] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919920b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196d590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:17.790, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:17.808047] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:17.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.813886] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=18] gc stale ls task succ [2025-08-14 09:06:17.823601] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:17.850008] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:17.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.850244] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:17.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.850252] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:17.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.857213] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:17.840, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:17.857259] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d54b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ecd90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:17.846, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:17.857276] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ecc10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:17.848, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:17.882336] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=24] PNIO [ratelimit] time: 1755162377882333, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007618, add_bytes: 0 [2025-08-14 09:06:17.923662] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:17.950045] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:17.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.950283] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:17.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.950269] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:17.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:17.992427] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:17.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.014067] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:18.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.014101] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:18.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.023723] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:18.028229] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:18.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.048225] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:18.048305] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=33] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:18.048364] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:18.048409] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:18.048452] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:18.048477] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:18.048494] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:18.050097] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:17.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.050308] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:17.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.050335] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:17.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.094551] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:18.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.123784] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:18.134883] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=15] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:18.150162] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:18.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.150371] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:18.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.150408] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:18.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.215419] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:18.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.223845] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:18.250208] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:18.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.250404] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:18.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.250469] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:18.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.260301] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:18.260344] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:18.260362] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:18.260379] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:18.265743] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:18.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.265786] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:18.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.279666] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:18.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.309852] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:18.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.323908] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:18.347329] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] EASY SLOW: thread: 7f995427d300, lastrun: 1755162250.789339 cost: 127.557988 loop:26617, slowcnt: 950 [2025-08-14 09:06:18.350284] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:18.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.350448] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:18.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.350531] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:18.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.366986] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=28] gc stale ls task succ [2025-08-14 09:06:18.423971] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:18.450313] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:18.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.450511] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:18.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.450594] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:18.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.517096] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:18.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.517134] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:18.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.520810] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:18.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.524040] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:18.550376] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:18.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.550572] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:18.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.550653] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:18.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.597639] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:18.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.624104] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:18.650437] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:18.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.650633] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:18.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.650713] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:18.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.718323] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:18.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.724571] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:18.243854] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:18.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.731790] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=487193][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:18.243|207=487.93ms|210=0us|222=2us|total=487.95ms) [2025-08-14 09:06:18.731810] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4389] timer task use too much time(end=1755162378731809, start=1755162378243852, delta=487957) [2025-08-14 09:06:18.731823] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=487188, expected_run_ts_us_=1755162378244633, ObClockGenerator::getRealClock()=1755162378731822, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:18.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.731837] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:18.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.260396] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:18.731902] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=470787][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:18.731916] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:18.731934] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:18.731957] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:18.731970] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:18.731986] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:18.731997] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:18.732011] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:18.732025] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:18.732047] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:18.732059] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:18.732073] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:18.732088] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:18.732110] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:18.732287] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:18.732302] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:18.732401] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:18.732475] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:18.722, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:18.738427] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc190, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc470, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:10.00s, expected_run_time:09:06:18.720, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.740305] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:18.741231] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=42] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:18.741270] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=31] temporary file module use no blocks [2025-08-14 09:06:18.741284] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:18.741312] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:18.741330] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=17] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623280180932, flush_io_finished_ret_:0, flush_io_finished_round_:26435, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:18.741846] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8030, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8310, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:10.00s, expected_run_time:09:06:18.730, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.741888] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:18.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.742538] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:18.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:18.742573] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891826cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891918190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:18.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:18.742630] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=14] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:18.743318] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:18.743507] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=18] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:18.744255] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:18.748026] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=33] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:18.721568] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:18.748342] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task too much delay in priority queue(token={this:0x7f991b7a5ed8, timer:0x7f99200ff180, task:0x7f99200ff1d8, task_type:"N9oceanbase8observer18ObTableLoadService8ObGCTaskE", scheduled_time:1755162361766822, pushed_time:1755162377752561, delay:30000000}, delay_in_priority_queue=15985739, threashold=1000000) [2025-08-14 09:06:18.748341] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26732][errcode=0] epoll handler too much time :26785 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:18.748402] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a4498, timer:0x7f9966a63f10, task:0x7f9922e05010, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr11TableGCTaskE", scheduled_time:1755162376781799, pushed_time:1755162377752564, delay:20000}, delay_in_thread_pool=995813, threashold=500000) [2025-08-14 09:06:18.749113] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=459169][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6178, timer:0x7f99200ff180, task:0x7f99200ff220, task_type:"N9oceanbase8observer18ObTableLoadService21ObClientTaskPurgeTaskE", scheduled_time:1755162377739421, pushed_time:0, delay:1000000}, running_task_token={this:0x7f991b7a6098, timer:0x7f99200ff180, task:0x7f99200ff208, task_type:"N9oceanbase8observer18ObTableLoadService25ObClientTaskAutoAbortTaskE", scheduled_time:1755162370767586, pushed_time:1755162378749107, delay:1000000}, total_delay=1009690, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:17.866128] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:213) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-08-14 09:06:18.749289] INFO [COMMON] handle (memory_dump.cpp:544) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=881824] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-08-14 09:06:18.749282] WDIAG [COORDINATOR] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [657][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=239629173][errcode=0] cost too much time:ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_(), (this=|threshold=1.00s|start at 09:06:18.749|total=2.00s) [2025-08-14 09:06:18.744876] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:18.750701] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:18.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.750774] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:18.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.751454] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:18.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.751706] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=31] LogLoopThread round_cost_time(us)(round_cost_time=3004103) [2025-08-14 09:06:18.751909] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:18.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.752603] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:18.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:18.752630] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919845b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:18.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:18.754668] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:18.755871] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=20] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:18.756307] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:18.757273] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:18.758348] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:18.743352] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [854][T1002_ObXAHbWor][T1002][Y0-0000000000000000-0-0] [lt=17] XA scheduler heartbeat task statistics(avg_time=2) [2025-08-14 09:06:18.761258] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:18.761694] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:18.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.761732] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:18.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.762333] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5804] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:18.762699] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:18.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:18.762732] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:18.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:18.762746] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:18.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:18.762759] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:18.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:18.762772] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eacb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:18.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:18.765592] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:18.768548] WDIAG [COORDINATOR] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [657][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] cost too much time:ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_failure(), (this=|threshold=1.00s|start at 09:06:18.768|total=2.03s) [2025-08-14 09:06:18.770051] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a5d18, timer:0x7f99200ff180, task:0x7f99200ff1a8, task_type:"N9oceanbase8observer18ObTableLoadService17ObCheckTenantTaskE", scheduled_time:1755162377761774, pushed_time:0, delay:1000000}, running_task_token={this:0x7f991b7a6098, timer:0x7f99200ff180, task:0x7f99200ff208, task_type:"N9oceanbase8observer18ObTableLoadService25ObClientTaskAutoAbortTaskE", scheduled_time:1755162370767586, pushed_time:1755162378749107, delay:1000000}, total_delay=1008276, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:18.770207] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=115] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:06:17.840031] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=811997][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:06:17.839936] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=813175][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:06:18.772754] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:18.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.772825] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:18.767, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:18.768590] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [657][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:0, time_interval:100.00ms, expected_run_time:09:06:18.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.773641] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [657][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=5036][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:06:18.773|total=2.94s) [2025-08-14 09:06:18.778607] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=23] gc stale ls task succ [2025-08-14 09:06:18.782380] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:18.782570] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:18.782585] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:18.782829] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:18.782878] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:18.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:18.782906] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:18.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:18.782930] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:18.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:18.782942] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:18.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:18.792930] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72716b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a4910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:18.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:18.792962] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:18.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:18.792985] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918100b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:18.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:18.792997] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:18.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:18.802951] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:18.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.803006] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:18.793, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:18.803027] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180fe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:18.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:18.813043] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40b7f0, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40bad0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:10.00s, expected_run_time:09:06:18.802, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.813103] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:18.805, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:18.813959] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=21] gc stale ls task succ [2025-08-14 09:06:18.823177] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73927b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:18.812, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:18.823211] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7399090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:18.815, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:18.823224] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:18.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:18.824633] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:18.834020] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73954b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:18.827, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:18.850762] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:18.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.851509] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:18.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.884389] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:18.873, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:18.889959] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=25] PNIO [ratelimit] time: 1755162378889956, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007623, add_bytes: 0 [2025-08-14 09:06:18.924694] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:18.950825] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:18.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.951565] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:18.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:18.994947] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:18.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.013600] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:19.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.013641] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:19.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.024756] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:06:19.024792] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:19.025202] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:19.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.049456] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:19.049518] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:19.049572] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:19.049613] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:19.049659] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:19.049673] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:19.049686] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:19.050887] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:18.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.051621] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:18.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.094107] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:19.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.124818] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:19.135550] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=50] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:19.150949] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:19.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.151683] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:19.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.214777] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:19.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.224884] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:19.232493] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:19.232538] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:19.232553] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:19.232569] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:19.232587] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:19.232614] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:19.232627] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:19.232641] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:19.232654] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:19.232666] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:19.232678] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:19.232705] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:19.232715] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:19.232730] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:19.232743] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:19.232758] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:19.232784] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:19.232800] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:19.232812] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:19.232988] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:19.233006] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:19.246491] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:19.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.251010] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:19.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.251753] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:19.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.256552] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:19.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.256708] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:19.248, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:19.265065] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:19.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.265098] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:19.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.276791] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:19.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.306996] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:19.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.324944] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:19.348448] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] EASY SLOW: thread: 7f995427d300, lastrun: 1755162250.789339 cost: 128.559107 loop:26617, slowcnt: 960 [2025-08-14 09:06:19.351052] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:19.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.351816] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:19.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.367067] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=21] gc stale ls task succ [2025-08-14 09:06:19.425005] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:19.451119] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:19.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.451887] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:19.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.498008] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:19.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.516426] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:19.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.516466] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:19.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.525067] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:19.528271] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:19.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.551184] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:19.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.551950] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:19.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.596889] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:19.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.625130] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:19.651246] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:19.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.651974] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:19.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.718175] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:19.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.721549] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=37] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:19.726423] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:17.771531] EDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:801) [830][T1002_GCCollect][T1001][YB42AC4D0007-00063C496069AFFA-0-0] [lt=23][errcode=-4012] LOGGER COST TOO MUCH TIME, cost: 1952886, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1952874, ObMalloc#start=1, ObMalloc#end=4, ALLOC_END=1, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x124de942 0x96a686c 0x96a5033 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x1ee93940 0x1ee92221 0x1ee928a5 0xfd80a58 0xfd91a9a 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:17.771531] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:801) [830][T1002_GCCollect][T1001][YB42AC4D0007-00063C496069AFFA-0-0] [lt=23][errcode=-4012] result set open failed(ret=-4012, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1002"}) [2025-08-14 09:06:19.732388] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=39] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:19.733187] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:19.733242] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:19.733253] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:19.733263] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:19.733278] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:19.733298] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:19.733322] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:19.733343] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:19.733356] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:19.733368] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:19.733392] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:19.733411] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:19.733451] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:19.733463] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:19.733474] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:19.733485] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:19.733498] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:19.733517] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:19.733534] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:19.733735] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:19.733755] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:19.726393] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:2005) [807][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=11] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1002) [2025-08-14 09:06:19.740352] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:19.741413] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=34] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:19.741443] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=23] temporary file module use no blocks [2025-08-14 09:06:19.741450] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:19.741461] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:19.741478] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=17] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623281181133, flush_io_finished_ret_:0, flush_io_finished_round_:26436, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:19.742734] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=11] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:19.743334] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:19.743586] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=14] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:19.746163] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5243] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:19.748067] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=29] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:19.749353] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:19.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.749449] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7270030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a3b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:19.735, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:19.749471] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ecb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:19.739, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:19.749485] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7353730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7396990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:19.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:19.749498] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:19.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:19.752317] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] update_check_range(min_check_version=2540, max_check_version=2540, global_version=2647) [2025-08-14 09:06:19.753345] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:19.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.753411] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:19.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.754713] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:19.755965] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=21] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:19.756307] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:19.757254] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:19.758335] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:19.744281] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:19.759646] WDIAG ev_invoke_pending (ev.c:2136) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=15340][errcode=0] epoll handler too much time :15372 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:19.759712] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=0] cost too much time:ob_occam_timer.h:int oceanbase::common::occam::ObOccamTimerTask::commit_task_to_thread_pool_(), (this=|threshold=100.00ms|start at 09:06:18.850|289=908.86ms|total=908.88ms) [2025-08-14 09:06:19.759720] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:19.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.759745] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:18.850|187=0us|203=1us|207=908.89ms|210=0us|222=2us|total=908.90ms) [2025-08-14 09:06:19.759766] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4389] timer task use too much time(end=1755162379759766, start=1755162378850851, delta=908915) [2025-08-14 09:06:19.759774] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ea30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:19.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:19.759799] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [657][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=46][errcode=-4016] task executed too delay(delay_time=908946, *p_task_={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:0, time_interval:100.00ms, expected_run_time:09:06:19.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.761258] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:19.762922] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a5fb8, timer:0x7f99200ff180, task:0x7f99200ff1f0, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162378750218, pushed_time:0, delay:1000000}, running_task_token={this:0x7f991b7a6098, timer:0x7f99200ff180, task:0x7f99200ff208, task_type:"N9oceanbase8observer18ObTableLoadService25ObClientTaskAutoAbortTaskE", scheduled_time:1755162370767586, pushed_time:1755162378749107, delay:1000000}, total_delay=1012703, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:19.763077] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEC1-0-0] [lt=43457][errcode=-4016] task executed too delay(delay_time=64613, *p_task_={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:0, time_interval:3.00s, expected_run_time:09:06:22.554, task_priority:1, with_handle_protected:1}) [280]OB_MALLOC COST TOO MUCH TIME, cost_time=1971026, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 10, time dist:[ALLOC_CHUNK_END=17, ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1954340, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9] [280]OB_MALLOC COST TOO MUCH TIME, cost_time=1982047, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1981992, WASH_BLOCK_END=55], seq:[0, 1, 2] [2025-08-14 09:06:19.767428] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:19.769403] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:19.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.769436] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:19.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.771299] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:06:19.772450] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:19.772470] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=15] temporary file module use no blocks [2025-08-14 09:06:19.772477] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=5] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:19.772486] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=8] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:19.772497] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623281210852, flush_io_finished_ret_:0, flush_io_finished_round_:26398, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:19.779497] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=21] gc stale ls task succ [2025-08-14 09:06:19.779602] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919afbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:19.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:19.779624] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919854b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:19.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:19.779635] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:19.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:19.779645] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:19.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:19.779655] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d45b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:19.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:19.779665] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72747b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:19.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:19.779674] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:19.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:19.782596] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:19.783398] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:19.789695] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:19.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:19.789721] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:19.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:19.789738] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:19.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:19.789747] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:19.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:19.799782] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:19.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:19.799808] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:19.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:19.819943] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919836b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:19.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:19.826490] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:19.840023] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:19.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:19.840058] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:19.831, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:19.859821] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:19.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.863028] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a4498, timer:0x7f9966a63f10, task:0x7f9922e05010, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr11TableGCTaskE", scheduled_time:1755162378768413, pushed_time:0, delay:20000}, running_task_token={this:0x7f991b7a4658, timer:0x7f9966a63f10, task:0x7f9922e05038, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr12TabletGCTaskE", scheduled_time:1755162377770297, pushed_time:1755162378749140, delay:20000}, total_delay=1094614, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:19.870191] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:19.862, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:19.926552] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:19.959859] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:19.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:19.990732] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:19.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:20.010767] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:20.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:20.010796] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:20.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:20.059893] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:19.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:20.091224] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:20.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:20.159954] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:20.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:20.211939] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:20.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:20.243560] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:20.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:20.253615] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:20.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:20.260026] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:20.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:20.262214] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:20.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:20.262247] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:20.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:20.349023] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] EASY SLOW: thread: 7f995427d300, lastrun: 1755162250.789339 cost: 129.559682 loop:26617, slowcnt: 970 [2025-08-14 09:06:20.360086] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:20.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:20.367129] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:06:20.460150] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:20.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:20.495039] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:20.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:20.513735] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:20.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:20.513762] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:20.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:20.560207] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:20.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:20.594085] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:20.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:20.660261] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:20.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:20.715361] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:20.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:20.721848] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:20.734406] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:288) [884][T1002_ArcFetche][T1002][YB42AC4D0007-00063C495ED99C16-0-0] [lt=27] ObArchiveFetcher is running(thread_index=1) [2025-08-14 09:06:20.746813] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:20.747618] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=44][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:19.836985] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=996069] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:06:20.757434] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:20.760436] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:20.761274] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:20.762360] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:20.765425] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:20.767446] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [657][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:0, time_interval:100.00ms, expected_run_time:09:06:20.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:20.767472] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [657][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:20.770944] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=743204] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:06:20.770966] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:20.770977] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:20.771404] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=16] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:06:20.772544] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=20] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:20.772605] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=55] temporary file module use no blocks [2025-08-14 09:06:20.772613] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:20.772624] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:20.772635] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623282212286, flush_io_finished_ret_:0, flush_io_finished_round_:26399, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:20.782269] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=11] gc stale ls task succ [2025-08-14 09:06:20.782429] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:20.783319] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:20.871043] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:20.971105] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:21.071168] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:21.171240] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:21.271295] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:21.350505] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:21.367190] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=13] gc stale ls task succ [2025-08-14 09:06:21.371359] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:21.471422] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:21.571486] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:21.671550] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:21.721945] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:21.733128] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEC3-0-0] [lt=24][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:21.741655] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:21.741683] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=19] temporary file module use no blocks [2025-08-14 09:06:21.741693] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:21.741706] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:21.741719] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623283181343, flush_io_finished_ret_:0, flush_io_finished_round_:26438, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:20.727388] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:946) [830][T1002_GCCollect][T1001][YB42AC4D0007-00063C496069AFFA-0-0] [lt=1952886][errcode=-4012] execute failed(ret=-4012, tenant_id=1001, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1002"}, retry_cnt=0, local_sys_schema_version=1753073979411448, local_tenant_schema_version=1753074886385104) [393]OB_MALLOC COST TOO MUCH TIME, cost_time=1970567, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1970526, ALLOC_CHUNK_END=3], seq:[0, 1, 2, 3, 4, 5, 6, 7] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=1983802, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1983732, WASH_BLOCK_END=70], seq:[0, 1, 2] [2025-08-14 09:06:21.749628] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=995507] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [777]OB_MALLOC COST TOO MUCH TIME, cost_time=1959026, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 7, time dist:[ALLOC_CHUNK_END=16, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=1958986], seq:[0, 1, 2, 3, 4, 5, 6] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=1973564, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1973562], seq:[0, 1] [2025-08-14 09:06:19.774428] EDIAG [RS] skip_check_member_list_change_ (ob_ls_recovery_stat_handler.cpp:109) [777][T1002_TeRec][T1002][YB42AC4D0007-00063C495DDA00A6-0-0] [lt=23][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1973589, owner: ObLog, click_count: 7, time dist:[FORMAT_END=10, ObMalloc#start=0, ObMalloc#end=1973572, ALLOC_END=1, APPEND_END=6, ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x1067fb2c 0x1067ecc2 0x1067e03d 0x1002f2a3 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [660]OB_MALLOC COST TOO MUCH TIME, cost_time=1954699, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1954658, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=1970567, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1970470, WASH_BLOCK_END=92, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=4], seq:[0, 1, 2, 3, 4] [2025-08-14 09:06:19.778771] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1970612, owner: ObLog, click_count: 5, time dist:[FORMAT_END=23, ObMalloc#start=1, ObMalloc#end=1970581, ALLOC_END=1, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [399]OB_MALLOC COST TOO MUCH TIME, cost_time=1950897, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1950858, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [399]OB_MALLOC COST TOO MUCH TIME, cost_time=1966888, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1966873, WASH_BLOCK_END=15], seq:[0, 1, 2] [2025-08-14 09:06:19.778771] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:19.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:21.752388] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1970612][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:21.752414] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=-4389] timer task use too much time(end=1755162381752413, start=1755162379778769, delta=1973644) [396]OB_MALLOC COST TOO MUCH TIME, cost_time=1950211, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1950169, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=1973266, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1973237, WASH_BLOCK_END=29], seq:[0, 1, 2] [2025-08-14 09:06:21.759153] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:21.759836] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:21.760268] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:21.748288] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] [COMPACTION DAG STATUS] (start_time=1755162371726400, end_time=1755162381748287, dag_status={}) [303]OB_MALLOC COST TOO MUCH TIME, cost_time=1922463, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1922419, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [303]OB_MALLOC COST TOO MUCH TIME, cost_time=1948032, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1947999, WASH_BLOCK_END=33], seq:[0, 1, 2] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=1919668, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1919625, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=1945271, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1945261, WASH_BLOCK_END=9], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=1880296, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=18, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1880244, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=1905892, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1905884, WASH_BLOCK_END=8], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=1880264, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1880226, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=1905878, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1905870, WASH_BLOCK_END=8], seq:[0, 1, 2] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=1854615, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1854573, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=1880232, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1880225, WASH_BLOCK_END=7], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=1836338, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1836296, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=1861973, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1861965, WASH_BLOCK_END=8], seq:[0, 1, 2] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=1683748, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=16, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1683695, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=1709411, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1709403, WASH_BLOCK_END=8], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=1598120, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=16, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1598063, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3, 4, 5, 6, 7] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=1623801, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1623794, WASH_BLOCK_END=7], seq:[0, 1, 2] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=1500572, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1500527, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=1526272, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1526264, WASH_BLOCK_END=8], seq:[0, 1, 2] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=1002594, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=17, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1002543, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=1029444, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1029402, WASH_BLOCK_END=42], seq:[0, 1, 2] [383]OB_MALLOC COST TOO MUCH TIME, cost_time=994958, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=22, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=994901, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [383]OB_MALLOC COST TOO MUCH TIME, cost_time=1021836, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1021825, WASH_BLOCK_END=11], seq:[0, 1, 2] [376]OB_MALLOC COST TOO MUCH TIME, cost_time=994034, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=993992, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [376]OB_MALLOC COST TOO MUCH TIME, cost_time=1020946, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1020936, WASH_BLOCK_END=10], seq:[0, 1, 2] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=992530, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=992485, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=1019472, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1019461, WASH_BLOCK_END=11], seq:[0, 1, 2] [387]OB_MALLOC COST TOO MUCH TIME, cost_time=991982, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=17, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=991931, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [387]OB_MALLOC COST TOO MUCH TIME, cost_time=1018946, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1018936, WASH_BLOCK_END=10], seq:[0, 1, 2] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=991767, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=991729, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=1018741, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1018730, WASH_BLOCK_END=10], seq:[0, 1, 2] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=991207, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=991166, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=1018210, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1018200, WASH_BLOCK_END=10], seq:[0, 1, 2] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=990864, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=990825, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3, 4, 5, 6, 7] [403]OB_MALLOC COST TOO MUCH TIME, cost_time=1017895, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1017877, WASH_BLOCK_END=18], seq:[0, 1, 2] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=987733, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=81, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=987618, ALLOC_CHUNK_END=3], seq:[0, 1, 2, 3, 4, 5, 6, 7] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=1014791, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1014782, WASH_BLOCK_END=9], seq:[0, 1, 2] [553]OB_MALLOC COST TOO MUCH TIME, cost_time=986018, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=15, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=985964, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [553]OB_MALLOC COST TOO MUCH TIME, cost_time=1013096, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1013086, WASH_BLOCK_END=10], seq:[0, 1, 2] [833]OB_MALLOC COST TOO MUCH TIME, cost_time=985400, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=985353, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [833]OB_MALLOC COST TOO MUCH TIME, cost_time=1012496, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1012486, WASH_BLOCK_END=10], seq:[0, 1, 2] [709]OB_MALLOC COST TOO MUCH TIME, cost_time=983818, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=983780, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [953]OB_MALLOC COST TOO MUCH TIME, cost_time=983559, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=983505, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [953]OB_MALLOC COST TOO MUCH TIME, cost_time=1010990, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1010970, WASH_BLOCK_END=20], seq:[0, 1, 2] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=980320, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=980280, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=1007767, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1007758, WASH_BLOCK_END=9], seq:[0, 1, 2] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=978355, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=978316, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=1005833, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1005824, WASH_BLOCK_END=8], seq:[0, 1, 2] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=976058, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=976018, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=1003515, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1003507, WASH_BLOCK_END=8], seq:[0, 1, 2] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=974447, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=974407, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=1001900, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1001892, WASH_BLOCK_END=8], seq:[0, 1, 2] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=978336, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=978296, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=1005903, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1005895, WASH_BLOCK_END=8], seq:[0, 1, 2] [279]OB_MALLOC COST TOO MUCH TIME, cost_time=969538, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=969498, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [279]OB_MALLOC COST TOO MUCH TIME, cost_time=997070, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=997062, WASH_BLOCK_END=8], seq:[0, 1, 2] [822]OB_MALLOC COST TOO MUCH TIME, cost_time=960127, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=960082, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [822]OB_MALLOC COST TOO MUCH TIME, cost_time=987659, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=987651, WASH_BLOCK_END=8], seq:[0, 1, 2] [2025-08-14 09:06:20.727388] EDIAG [SERVER] query (ob_inner_sql_connection.cpp:946) [830][T1002_GCCollect][T1001][YB42AC4D0007-00063C496069AFFA-0-0] [lt=1952886][errcode=-4012] LOGGER COST TOO MUCH TIME, cost: 1017603, owner: ObLog, click_count: 5, time dist:[FORMAT_END=994122, ObMalloc#start=2, ObMalloc#end=23478, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x124dfcec 0x96a5a2f 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x1ee93940 0x1ee92221 0x1ee928a5 0xfd80a58 0xfd91a9a 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [807]OB_MALLOC COST TOO MUCH TIME, cost_time=964396, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=964357, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [807]OB_MALLOC COST TOO MUCH TIME, cost_time=992269, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=992256, WASH_BLOCK_END=13], seq:[0, 1, 2] [2025-08-14 09:06:21.765269] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [709]OB_MALLOC COST TOO MUCH TIME, cost_time=1011271, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1011238, WASH_BLOCK_END=33], seq:[0, 1, 2] [2025-08-14 09:06:21.770841] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:21.771531] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:06:21.771833] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEC3-0-0] [lt=0] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:06:21.772215] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:21.773339] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=21] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:21.773362] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=19] temporary file module use no blocks [2025-08-14 09:06:21.773368] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=5] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:21.773382] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:21.773404] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=22] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623283212424, flush_io_finished_ret_:0, flush_io_finished_round_:26400, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:21.782333] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=14] gc stale ls task succ [2025-08-14 09:06:21.782407] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:21.783258] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:21.872277] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:21.972287] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:21.992304] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:22.072345] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:06:22.072388] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:22.172404] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:22.272463] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:22.359385] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:22.367252] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=13] gc stale ls task succ [2025-08-14 09:06:22.372502] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:22.472561] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:22.572598] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:22.600109] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=30453][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:22.672656] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:19.783296] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2927933, owner: ObLog, click_count: 9, time dist:[ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=1, FORMAT_END=39, ObMalloc#start=0, ObMalloc#end=1973278, ObMalloc#start=954601, ObMalloc#end=4, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:20.751772] EDIAG [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:233) [709][T1001_RFLWorker][T1001][YB42AC4D0007-00063C495BE99C16-0-0] [lt=2001148][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1960946, owner: ObLog, click_count: 5, time dist:[FORMAT_END=14, ObMalloc#start=2, ObMalloc#end=1013577, ALLOC_END=947349, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x11359281 0x98b739c 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:20.743669] EDIAG do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1969084, owner: ObLog, click_count: 5, time dist:[FORMAT_END=10, ObMalloc#start=0, ObMalloc#end=1018752, ALLOC_END=950319, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x202c530b 0x202c6888 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:20.743417] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1969374, owner: ObLog, click_count: 7, time dist:[FORMAT_END=8, ObMalloc#start=1, ObMalloc#end=1018952, ObMalloc#start=950405, ObMalloc#end=4, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:20.775726] EDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1937123, owner: ObLog, click_count: 7, time dist:[FORMAT_END=16, ObMalloc#start=1, ObMalloc#end=987671, ObMalloc#start=949428, ObMalloc#end=3, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0xfb622cd 0xfb619cb 0xfb892d3 0x994150a 0x993eecd 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:20.771423] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1507) [807][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=13423] start lib cache evict(tenant_id=1002, mem_hold=211812352, mem_limit=429496725, cache_obj_num=1331, cache_node_num=1330) [2025-08-14 09:06:19.783296] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:20.751772] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:233) [709][T1001_RFLWorker][T1001][YB42AC4D0007-00063C495BE99C16-0-0] [lt=2001148] ObRemoteFetchWorker is running(thread_index=0) [2025-08-14 09:06:22.715702] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2927933][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:20.743669] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=18] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:22.715726] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:22.715756] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:20.743417] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:20.775726] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:22.715792] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1969374][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:22.715842] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=41] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:20.771423] EDIAG [SQL.PC] cache_evict (ob_plan_cache.cpp:1507) [807][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=13423][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1941297, owner: ObLog, click_count: 7, time dist:[FORMAT_END=21, ObMalloc#start=1, ObMalloc#end=992276, ObMalloc#start=948990, ObMalloc#end=4, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1535920e 0x153639eb 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:20.759764] EDIAG [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1962898, owner: ObLog, click_count: 7, time dist:[FORMAT_END=6498, ObMalloc#start=2, ObMalloc#end=997077, ObMalloc#start=959317, ObMalloc#end=3, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x260b4cc1 0x260b9960 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:20.136204] EDIAG try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2585537, owner: ObLog, click_count: 5, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=1623809, ALLOC_END=961718, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x26499505 0x97074a3 0x26498fa9 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:20.233931] EDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] LOGGER COST TOO MUCH TIME, cost: 2487928, owner: ObLog, click_count: 7, time dist:[FORMAT_END=17, ObMalloc#start=1, ObMalloc#end=1526279, ObMalloc#start=961626, ObMalloc#end=4, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x13b06ffb 0x958a39e 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:19.830098] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=4905][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2891249, owner: ObLog, click_count: 5, time dist:[FORMAT_END=918047, ObMalloc#start=2, ObMalloc#end=1013102, ALLOC_END=960098, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:19.853429] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2867987, owner: ObLog, click_count: 5, time dist:[FORMAT_END=26, ObMalloc#start=1, ObMalloc#end=1905908, ALLOC_END=962051, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:19.813980] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=19][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2907472, owner: ObLog, click_count: 5, time dist:[FORMAT_END=10, ObMalloc#start=0, ObMalloc#end=1945278, ALLOC_END=962184, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:20.755323] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1967589, owner: ObLog, click_count: 5, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=1007775, ALLOC_END=959803, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:19.811148] EDIAG handle (memory_dump.cpp:671) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2910772, owner: ObLog, click_count: 5, time dist:[FORMAT_END=11, ObMalloc#start=0, ObMalloc#end=1948049, ALLOC_END=962712, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0xf98da43 0x96b2d7e 0x96b1c88 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:22.720337] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:20.233931] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:20.759764] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=22] log compressor cycles once. (ret=0, cost_time=22362, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25861955584, fast_delete_log_mode=false) [2025-08-14 09:06:19.782622] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2939367, owner: ObLog, click_count: 5, time dist:[FORMAT_END=5, ObMalloc#start=0, ObMalloc#end=1966895, ALLOC_END=972467, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:20.757365] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1966225, owner: ObLog, click_count: 7, time dist:[FORMAT_END=6, ObMalloc#start=0, ObMalloc#end=1005911, ObMalloc#start=959405, ObMalloc#end=901, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:20.742839] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=19][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1981752, owner: ObLog, click_count: 7, time dist:[FORMAT_END=14, ObMalloc#start=0, ObMalloc#end=1019480, ObMalloc#start=960338, ObMalloc#end=1919, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:20.136204] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=20] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:19.830098] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=4905] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:06:22.725709] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=2487928][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:22.725724] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:22.725734] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:22.725744] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:19.853429] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:19.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:22.725768] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:22.725780] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:22.725782] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=2867987][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:22.725800] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:22.725807] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4389] timer task use too much time(end=1755162382725806, start=1755162379853427, delta=2872379) [2025-08-14 09:06:22.725809] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:22.725828] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:22.725841] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:19.813980] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=19] gc stale ls task succ [2025-08-14 09:06:22.725849] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:22.725860] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:22.725869] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=2907472] gc stale ls task succ [2025-08-14 09:06:22.725870] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:22.725879] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:20.755323] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=26] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:22.725888] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:22.725898] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:22.725910] WDIAG ev_invoke_pending (ev.c:2136) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1967589][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:22.725919] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:22.725925] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:22.725930] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:22.726117] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:22.726128] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:19.811148] INFO handle (memory_dump.cpp:671) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=34] statistics: tenant_cnt: 6, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 168 270 0 1 2 110 1 0 1 3 7 0 0 1 4 1 1 0 1 5 85 6 0 1 6 13 0 0 1 7 22 2 0 1 8 53 1 0 1 12 17 3 0 1 13 10 1 0 1 14 1 1 0 1 15 1 1 0 1 16 8 7 0 1 18 4 3 0 1 20 2 1 0 1 22 1 1 0 1 24 1 0 0 1 25 6 0 0 500 0 561 275 0 500 7 188 5 0 500 8 41 1 0 500 9 2 1 0 500 10 6 2 0 500 16 58 4 0 500 17 60 7 0 500 22 38 64 0 500 23 27 10 0 508 0 2 7 0 508 8 5 1 0 509 0 2 7 0 509 8 5 1 0 1001 0 147 252 0 1001 2 26 1 0 1001 3 5 0 0 1001 4 1 1 0 1001 5 85 6 0 1001 6 11 0 0 1001 7 30 2 0 1001 8 36 1 0 1001 12 17 3 0 1001 13 3 1 0 1001 14 1 1 0 1001 16 7 3 0 1001 18 4 3 0 1001 20 4 1 0 1001 25 8 0 0 1002 0 263 279 0 1002 2 198 1 0 1002 3 4 0 0 1002 4 1 1 0 1002 5 101 6 0 1002 6 14 0 0 1002 7 6 2 0 1002 8 73 1 0 1002 12 18 3 0 1002 13 7 1 0 1002 14 1 2 0 1002 16 10 7 0 1002 18 9 3 0 1002 20 2 1 0 1002 21 1 0 0 1002 24 1 0 0 1002 25 7 0 0 cost_time: 1036629 [2025-08-14 09:06:22.716102] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1524) [807][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=1941297] end lib cache evict(tenant_id=1002, cache_evict_num=0, mem_hold=211812352, mem_limit=429496725, cache_obj_num=1331, cache_node_num=1330) [2025-08-14 09:06:19.782622] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:22.726377] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=2910772] [MEMORY] tenant: 1, limit: 2147483648 hold: 1013174272 cache_hold: 56623104 cache_used: 56623104 cache_item_count: 27 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 444993536 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 230686720 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 1163264 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 483328 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 51617792 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 1458176 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 14204928 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 111149056 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35651584 limit= 429496720 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 8003584 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= PS_CACHE_CTX_ID hold_bytes= 65536 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 14827520 limit= 9223372036854775807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 3481600 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3964928 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 212992 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 884736 limit= 9223372036854775807 [2025-08-14 09:06:22.726456] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2939367][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:22.726558] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=92] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:20.757365] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:22.726568] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:22.726604] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1966225][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:22.726621] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:20.742839] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:22.741311] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEC3-0-0] [lt=28][errcode=-4389] run this task use too much time(*p_task_={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:0, time_interval:3.00s, expected_run_time:09:06:25.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:22.741345] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEC3-0-0] [lt=32][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:22.741358] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEC3-0-0] [lt=9][errcode=-4016] task executed too delay(delay_time=141257, *p_task_={this:0x7f99571630c0, caller:mds_tenant_service.cpp:mtl_start:117, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_145, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571633a0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:0, time_interval:15.00s, expected_run_time:09:06:37.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:22.727679] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2792) [807][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=10308] schedule next cache evict task(evict_interval=5000000) [2025-08-14 09:06:20.732687] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2012323, owner: ObLog, click_count: 7, time dist:[FORMAT_END=9, ObMalloc#start=2, ObMalloc#end=1029456, ObMalloc#start=960512, ObMalloc#end=22339, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:20.761318] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1983843, owner: ObLog, click_count: 7, time dist:[FORMAT_END=15, ObMalloc#start=1, ObMalloc#end=1001908, ObMalloc#start=959475, ObMalloc#end=22440, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:20.741303] EDIAG [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=1970233][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2003889, owner: ObLog, click_count: 7, time dist:[FORMAT_END=17, ObMalloc#start=1, ObMalloc#end=1020954, ObMalloc#start=960399, ObMalloc#end=22514, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1fce4e91 0x1fce455d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:22.742043] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:20.732687] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:22.748096] WDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2012323][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:22.748119] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:22.748135] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:20.761318] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:22.748184] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1983843][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:22.748201] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:20.741303] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=1970233] blacklist_loop exec finished(cost_time=18, is_enabled=true, send_cnt=0) [2025-08-14 09:06:20.050416] EDIAG [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2695398, owner: ObLog, click_count: 7, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=1709418, ObMalloc#start=961808, ObMalloc#end=24154, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1f9d7fcf 0x99b8be2 0x1f981833 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:20.759674] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [663][T1001_StandbySc][T1001][YB42AC4D0007-00063C495C7A03BC-0-0] [lt=16909][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1987221, owner: ObLog, click_count: 7, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=1003523, ObMalloc#start=959534, ObMalloc#end=24146, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:19.897567] EDIAG eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2848079, owner: ObLog, click_count: 7, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=1861979, ObMalloc#start=961872, ObMalloc#end=24211, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x27a8a8c8 0x97b7e1a 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:19.879211] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2866476, owner: ObLog, click_count: 7, time dist:[FORMAT_END=37, ObMalloc#start=1, ObMalloc#end=1880239, ObMalloc#start=961944, ObMalloc#end=24250, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:20.050416] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:19.897567] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=26] PNIO [ratelimit] time: 1755162379897565, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007609, add_bytes: 0 [2025-08-14 09:06:22.749968] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2695398] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:22.749993] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=2848079] PNIO [ratelimit] time: 1755162382749991, bytes: 0, bw: 0.000000 MB/s, add_ts: 2852426, add_bytes: 0 [2025-08-14 09:06:22.750011] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:19.879211] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fcc90, caller:mds_tenant_service.cpp:mtl_start:117, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_145, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcf70, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:1, time_interval:15.00s, expected_run_time:09:06:19.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:22.750051] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:22.750048] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=2866476][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:22.750077] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=-4389] timer task use too much time(end=1755162382750076, start=1755162379879209, delta=2870867) [2025-08-14 09:06:22.750098] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:22.750112] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:20.757297] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1989852, owner: ObLog, click_count: 7, time dist:[FORMAT_END=22, ObMalloc#start=0, ObMalloc#end=1005839, ObMalloc#start=959750, ObMalloc#end=24236, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:20.747779] EDIAG [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=156][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1999363, owner: ObLog, click_count: 7, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=1014799, ObMalloc#start=960175, ObMalloc#end=24374, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a1f867 0x27a135c3 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:20.744254] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2002958, owner: ObLog, click_count: 7, time dist:[FORMAT_END=6, ObMalloc#start=1, ObMalloc#end=1018217, ObMalloc#start=960513, ObMalloc#end=24206, ALLOC_END=1, APPEND_END=14], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:19.830049] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=991714][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2915859, owner: ObLog, click_count: 7, time dist:[FORMAT_END=918744, ObMalloc#start=1, ObMalloc#end=1012503, ObMalloc#start=960292, ObMalloc#end=24315, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:20.757297] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=29] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:20.747779] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=156] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:22.750197] WDIAG ev_invoke_pending (ev.c:2136) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=1999363][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:20.744626] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [403][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:22.750218] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=9] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:20.744254] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:22.750243] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=8] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:22.750264] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2002958][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:22.750282] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:22.750295] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:19.830049] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=991714] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:06:22.748290] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1937123] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:06:19.853509] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2892597, owner: ObLog, click_count: 7, time dist:[FORMAT_END=14, ObMalloc#start=0, ObMalloc#end=1905884, ObMalloc#start=961994, ObMalloc#end=24367, ALLOC_END=335, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:22.750124] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:19.853509] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:19.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:22.750509] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=2892597][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:22.750531] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4389] timer task use too much time(end=1755162382750531, start=1755162379853508, delta=2897023) [2025-08-14 09:06:22.739676] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=146] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 444993536 used= 389771296 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 70 washed_blocks= 287 washed_size= 70221824 [MEMORY] request_cached_chunk_cnt= 6 [MEMORY] hold= 41963520 used= 41943040 count= 1 avg_used= 41943040 block_cnt= 1 chunk_cnt= 1 mod=LogGroupBuffer [MEMORY] hold= 41963520 used= 41943072 count= 1 avg_used= 41943072 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 29130752 used= 29116416 count= 14 avg_used= 2079744 block_cnt= 14 chunk_cnt= 14 mod=CACHE_MAP_NODE [MEMORY] hold= 25640960 used= 25394048 count= 37 avg_used= 686325 block_cnt= 37 chunk_cnt= 27 mod=MysqlRequesReco [MEMORY] hold= 20615168 used= 20545536 count= 4 avg_used= 5136384 block_cnt= 4 chunk_cnt= 4 mod=ServerObjecPool [MEMORY] hold= 16908288 used= 16777728 count= 16 avg_used= 1048608 block_cnt= 16 chunk_cnt= 16 mod=DiagnosticInfos [MEMORY] hold= 16797696 used= 16777336 count= 1 avg_used= 16777336 block_cnt= 1 chunk_cnt= 1 mod=MrgMemPoolChk [MEMORY] hold= 16083712 used= 15926225 count= 112 avg_used= 142198 block_cnt= 38 chunk_cnt= 9 mod=OmtTenant [MEMORY] hold= 10510336 used= 10491392 count= 1 avg_used= 10491392 block_cnt= 1 chunk_cnt= 1 mod=ASHListBuffer [MEMORY] hold= 9457664 used= 9437784 count= 1 avg_used= 9437784 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 8777728 used= 8760064 count= 2 avg_used= 4380032 block_cnt= 2 chunk_cnt= 2 mod=FreeTbltStream [MEMORY] hold= 8540160 used= 8519680 count= 1 avg_used= 8519680 block_cnt= 1 chunk_cnt= 1 mod=ArcFetchQueue [MEMORY] hold= 8540160 used= 8519680 count= 1 avg_used= 8519680 block_cnt= 1 chunk_cnt= 1 mod=RCSrv [MEMORY] hold= 6856704 used= 6788160 count= 9 avg_used= 754240 block_cnt= 9 chunk_cnt= 9 mod=ResourceGroup [MEMORY] hold= 6438912 used= 5701632 count= 90 avg_used= 63351 block_cnt= 90 chunk_cnt= 30 mod=[T]char [MEMORY] hold= 5505024 used= 4822272 count= 168 avg_used= 28704 block_cnt= 168 chunk_cnt= 31 mod=DiagnosticInfo [MEMORY] hold= 5386240 used= 5368320 count= 1 avg_used= 5368320 block_cnt= 1 chunk_cnt= 1 mod=RespTimeColl [MEMORY] hold= 4943872 used= 4915456 count= 2 avg_used= 2457728 block_cnt= 2 chunk_cnt= 2 mod=HashBuckDmId [MEMORY] hold= 4739072 used= 4718712 count= 1 avg_used= 4718712 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmChe [MEMORY] hold= 4472832 used= 2529072 count= 546 avg_used= 4632 block_cnt= 132 chunk_cnt= 55 mod=WorkerMap [MEMORY] hold= 4235264 used= 4194304 count= 2 avg_used= 2097152 block_cnt= 2 chunk_cnt= 2 mod=SHARE_BLOCK [MEMORY] hold= 3865600 used= 3705600 count= 800 avg_used= 4632 block_cnt= 800 chunk_cnt= 13 mod=CkptDgnMemCU [MEMORY] hold= 3865600 used= 3705600 count= 800 avg_used= 4632 block_cnt= 800 chunk_cnt= 13 mod=CkptDgnMem [MEMORY] hold= 3772416 used= 3751944 count= 1 avg_used= 3751944 block_cnt= 1 chunk_cnt= 1 mod=IteratorStorage [MEMORY] hold= 3723264 used= 3698704 count= 2 avg_used= 1849352 block_cnt= 2 chunk_cnt= 2 mod=LogDIOAligned [MEMORY] hold= 3162112 used= 3145808 count= 2 avg_used= 1572904 block_cnt= 2 chunk_cnt= 2 mod=TenantDLMgr [MEMORY] hold= 3145728 used= 2228224 count= 128 avg_used= 17408 block_cnt= 128 chunk_cnt= 6 mod=SqlDtlQueue [MEMORY] hold= 3047424 used= 3013468 count= 10 avg_used= 301346 block_cnt= 10 chunk_cnt= 2 mod=LDIOSetup [MEMORY] hold= 3031040 used= 3013952 count= 1 avg_used= 3013952 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2850816 used= 2832384 count= 12 avg_used= 236032 block_cnt= 12 chunk_cnt= 10 mod=DiagnosticSum [MEMORY] hold= 2379776 used= 2359608 count= 1 avg_used= 2359608 block_cnt= 1 chunk_cnt= 1 mod=FastFrezCkr [MEMORY] hold= 2379776 used= 2359608 count= 1 avg_used= 2359608 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMonMap [MEMORY] hold= 2379776 used= 2359608 count= 1 avg_used= 2359608 block_cnt= 1 chunk_cnt= 1 mod=MediumTabletMap [MEMORY] hold= 2375680 used= 2359536 count= 2 avg_used= 1179768 block_cnt= 2 chunk_cnt= 2 mod=HashBuckLCSta [MEMORY] hold= 2248704 used= 2228224 count= 1 avg_used= 2228224 block_cnt= 1 chunk_cnt= 1 mod=LogIOCb [MEMORY] hold= 2198720 used= 416700 count= 25277 avg_used= 16 block_cnt= 272 chunk_cnt= 3 mod=Number [MEMORY] hold= 2197056 used= 2174312 count= 77 avg_used= 28237 block_cnt= 77 chunk_cnt= 2 mod=TabletMap [MEMORY] hold= 2064384 used= 1835008 count= 28 avg_used= 65536 block_cnt= 28 chunk_cnt= 11 mod=MajorMergeMgr [MEMORY] hold= 1843200 used= 1770696 count= 9 avg_used= 196744 block_cnt= 9 chunk_cnt= 5 mod=DagMap [MEMORY] hold= 1571328 used= 1566720 count= 24 avg_used= 65280 block_cnt= 24 chunk_cnt= 9 mod=ResultSet [MEMORY] hold= 1294336 used= 1280384 count= 2 avg_used= 640192 block_cnt= 2 chunk_cnt= 1 mod=XATimeWheel [MEMORY] hold= 1294336 used= 1280384 count= 2 avg_used= 640192 block_cnt= 2 chunk_cnt= 2 mod=TransTimeWheel [MEMORY] hold= 1187840 used= 1179768 count= 1 avg_used= 1179768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPlanCac [MEMORY] hold= 1187840 used= 1179768 count= 1 avg_used= 1179768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsCache [MEMORY] hold= 1187840 used= 1179768 count= 1 avg_used= 1179768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsInfo [MEMORY] hold= 1187840 used= 1179768 count= 1 avg_used= 1179768 block_cnt= 1 chunk_cnt= 1 mod=RewriteRuleMap [MEMORY] hold= 1179648 used= 1142528 count= 24 avg_used= 47605 block_cnt= 16 chunk_cnt= 11 mod=CommSysVarFac [MEMORY] hold= 1130496 used= 1126688 count= 6 avg_used= 187781 block_cnt= 6 chunk_cnt= 6 mod=IoControl [MEMORY] hold= 1114112 used= 1079296 count= 131 avg_used= 8238 block_cnt= 80 chunk_cnt= 44 mod=ParamCharsType [MEMORY] hold= 958464 used= 950272 count= 1 avg_used= 950272 block_cnt= 1 chunk_cnt= 1 mod=IOWorkerLQ [MEMORY] hold= 933888 used= 931072 count= 1 avg_used= 931072 block_cnt= 1 chunk_cnt= 1 mod=TransService [MEMORY] hold= 933888 used= 931072 count= 1 avg_used= 931072 block_cnt= 1 chunk_cnt= 1 mod=ArcSenderQueue [MEMORY] hold= 829632 used= 821248 count= 2 avg_used= 410624 block_cnt= 2 chunk_cnt= 2 mod=LSMap [MEMORY] hold= 802816 used= 800000 count= 1 avg_used= 800000 block_cnt= 1 chunk_cnt= 1 mod=SqlFltSpanRec [MEMORY] hold= 798112 used= 789624 count= 3 avg_used= 263208 block_cnt= 2 chunk_cnt= 2 mod=PartitTableTask [MEMORY] hold= 786688 used= 524352 count= 33 avg_used= 15889 block_cnt= 33 chunk_cnt= 4 mod=LogAlloc [MEMORY] hold= 786432 used= 386304 count= 48 avg_used= 8048 block_cnt= 48 chunk_cnt= 20 mod=[T]ObPerfEventR [MEMORY] hold= 679872 used= 658688 count= 83 avg_used= 7936 block_cnt= 10 chunk_cnt= 7 mod=TltMemtablMgr [MEMORY] hold= 671744 used= 663552 count= 1 avg_used= 663552 block_cnt= 1 chunk_cnt= 1 mod=ApplySrv [MEMORY] hold= 666880 used= 404416 count= 89 avg_used= 4544 block_cnt= 66 chunk_cnt= 32 mod=[T]ObTraceEvent [MEMORY] hold= 663552 used= 655360 count= 1 avg_used= 655360 block_cnt= 1 chunk_cnt= 1 mod=FetchLog [MEMORY] hold= 663552 used= 659200 count= 1 avg_used= 659200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 655360 used= 626816 count= 4 avg_used= 156704 block_cnt= 4 chunk_cnt= 4 mod=SQLSessionInfo [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=MultiVersionGC [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=MdsT [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=CoordTF [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=ElectTimer [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=DupTbLease [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=FrzTrigger [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=OBJLockGC [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=CoordTR [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=DetectorTimer [MEMORY] hold= 598016 used= 590232 count= 1 avg_used= 590232 block_cnt= 1 chunk_cnt= 1 mod=DagNetIdMap [MEMORY] hold= 516096 used= 507904 count= 1 avg_used= 507904 block_cnt= 1 chunk_cnt= 1 mod=CommitVersions [MEMORY] hold= 507904 used= 491520 count= 2 avg_used= 245760 block_cnt= 2 chunk_cnt= 1 mod=SqlExecutor [MEMORY] hold= 483328 used= 477216 count= 1 avg_used= 477216 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMon [MEMORY] hold= 475136 used= 458816 count= 2 avg_used= 229408 block_cnt= 2 chunk_cnt= 2 mod=ts_token [MEMORY] hold= 442368 used= 393216 count= 6 avg_used= 65536 block_cnt= 6 chunk_cnt= 6 mod=MinorMergeMgr [MEMORY] hold= 409600 used= 401408 count= 1 avg_used= 401408 block_cnt= 1 chunk_cnt= 1 mod=ReplaySrv [MEMORY] hold= 401408 used= 393504 count= 1 avg_used= 393504 block_cnt= 1 chunk_cnt= 1 mod=ClogGe [MEMORY] hold= 385024 used= 375520 count= 2 avg_used= 187760 block_cnt= 2 chunk_cnt= 2 mod=bf_queue [MEMORY] hold= 376832 used= 223376 count= 23 avg_used= 9712 block_cnt= 23 chunk_cnt= 17 mod=[T]ObDSActionAr [MEMORY] hold= 311104 used= 303904 count= 6 avg_used= 50650 block_cnt= 6 chunk_cnt= 3 mod=PoolFreeList [MEMORY] hold= 303104 used= 294936 count= 1 avg_used= 294936 block_cnt= 1 chunk_cnt= 1 mod=ColUsagHashMap [MEMORY] hold= 303104 used= 294936 count= 1 avg_used= 294936 block_cnt= 1 chunk_cnt= 1 mod=DmlStatHashMap [MEMORY] hold= 303104 used= 294936 count= 1 avg_used= 294936 block_cnt= 1 chunk_cnt= 1 mod=RSCompStMap [MEMORY] hold= 303104 used= 294936 count= 1 avg_used= 294936 block_cnt= 1 chunk_cnt= 1 mod=RSCompactMap [MEMORY] hold= 286720 used= 280192 count= 1 avg_used= 280192 block_cnt= 1 chunk_cnt= 1 mod=MetaMemMgr [MEMORY] hold= 278528 used= 270468 count= 2 avg_used= 135234 block_cnt= 2 chunk_cnt= 2 mod=CsSstableReader [MEMORY] hold= 247120 used= 3318 count= 3061 avg_used= 1 block_cnt= 31 chunk_cnt= 1 mod=CharsetUtil [MEMORY] hold= 212992 used= 211072 count= 1 avg_used= 211072 block_cnt= 1 chunk_cnt= 1 mod=TimerWK [MEMORY] hold= 204800 used= 196616 count= 1 avg_used= 196616 block_cnt= 1 chunk_cnt= 1 mod=TenDLBucket [MEMORY] hold= 204800 used= 196616 count= 1 avg_used= 196616 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 204800 used= 196744 count= 1 avg_used= 196744 block_cnt= 1 chunk_cnt= 1 mod=DagNetMap [MEMORY] hold= 204800 used= 196616 count= 1 avg_used= 196616 block_cnt= 1 chunk_cnt= 1 mod=T3MBucket [MEMORY] hold= 188416 used= 182528 count= 23 avg_used= 7936 block_cnt= 12 chunk_cnt= 8 mod=SqlSession [MEMORY] hold= 155648 used= 150784 count= 19 avg_used= 7936 block_cnt= 19 chunk_cnt= 15 mod=LinearHashMapNo [MEMORY] hold= 147456 used= 139264 count= 1 avg_used= 139264 block_cnt= 1 chunk_cnt= 1 mod=RFLTaskQueue [MEMORY] hold= 131072 used= 130560 count= 2 avg_used= 65280 block_cnt= 2 chunk_cnt= 2 mod=MemTblObj [MEMORY] hold= 122880 used= 110608 count= 2 avg_used= 55304 block_cnt= 2 chunk_cnt= 2 mod=TabletIDSetBkt [MEMORY] hold= 114688 used= 111096 count= 1 avg_used= 111096 block_cnt= 1 chunk_cnt= 1 mod=DbmsSched_Job [MEMORY] hold= 114688 used= 106496 count= 1 avg_used= 106496 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlMgr [MEMORY] hold= 114688 used= 111096 count= 1 avg_used= 111096 block_cnt= 1 chunk_cnt= 1 mod=AsyncFreezeTabl [MEMORY] hold= 114688 used= 111104 count= 14 avg_used= 7936 block_cnt= 13 chunk_cnt= 10 mod=QueryAllocator [MEMORY] hold= 102512 used= 27016 count= 370 avg_used= 73 block_cnt= 74 chunk_cnt= 31 mod=Coro [MEMORY] hold= 98304 used= 95520 count= 1 avg_used= 95520 block_cnt= 1 chunk_cnt= 1 mod=LSSvr [MEMORY] hold= 98304 used= 91968 count= 1 avg_used= 91968 block_cnt= 1 chunk_cnt= 1 mod=tenant_mf_mgr [MEMORY] hold= 88000 used= 85888 count= 11 avg_used= 7808 block_cnt= 11 chunk_cnt= 1 mod=TabletIDSetNode [MEMORY] hold= 81920 used= 73896 count= 1 avg_used= 73896 block_cnt= 1 chunk_cnt= 1 mod=RSCompPairCache [MEMORY] hold= 81824 used= 73472 count= 2 avg_used= 36736 block_cnt= 2 chunk_cnt= 2 mod=ts_queue [MEMORY] hold= 81824 used= 73472 count= 2 avg_used= 36736 block_cnt= 2 chunk_cnt= 2 mod=ts_cancel_set [MEMORY] hold= 81824 used= 73472 count= 2 avg_used= 36736 block_cnt= 2 chunk_cnt= 2 mod=ts_run_set [MEMORY] hold= 81536 used= 78080 count= 10 avg_used= 7808 block_cnt= 10 chunk_cnt= 8 mod=HashNodeLCSta [MEMORY] hold= 73728 used= 72736 count= 1 avg_used= 72736 block_cnt= 1 chunk_cnt= 1 mod=LogSubmitLogQue [MEMORY] hold= 73728 used= 72736 count= 1 avg_used= 72736 block_cnt= 1 chunk_cnt= 1 mod=LogSharedQueueT [MEMORY] hold= 73728 used= 66176 count= 1 avg_used= 66176 block_cnt= 1 chunk_cnt= 1 mod=Rpc [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=TxCtxMemObj [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=FlushLog [MEMORY] hold= 65536 used= 63488 count= 8 avg_used= 7936 block_cnt= 8 chunk_cnt= 6 mod=PlanVaIdx [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=TxDataMemObj [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=DEVICE_MANAGER [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=StorMetaCacheIO [MEMORY] hold= 65536 used= 63488 count= 8 avg_used= 7936 block_cnt= 8 chunk_cnt= 7 mod=SqlSessiVarMap [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=LockMemObj [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=FlushMeta [MEMORY] hold= 57344 used= 51328 count= 1 avg_used= 51328 block_cnt= 1 chunk_cnt= 1 mod=palf_env [MEMORY] hold= 56448 used= 48384 count= 14 avg_used= 3456 block_cnt= 11 chunk_cnt= 9 mod=CACHE_INST [MEMORY] hold= 52672 used= 34944 count= 52 avg_used= 672 block_cnt= 36 chunk_cnt= 17 mod=[T]ObWarningBuf [MEMORY] hold= 49152 used= 47616 count= 2 avg_used= 23808 block_cnt= 2 chunk_cnt= 1 mod=IOUsageInfo [MEMORY] hold= 49152 used= 45184 count= 1 avg_used= 45184 block_cnt= 1 chunk_cnt= 1 mod=IO_MGR [MEMORY] hold= 48960 used= 44840 count= 2 avg_used= 22420 block_cnt= 2 chunk_cnt= 2 mod=DynamicFactor [MEMORY] hold= 48112 used= 5016 count= 209 avg_used= 24 block_cnt= 74 chunk_cnt= 29 mod=[T]MemoryContex [MEMORY] hold= 46400 used= 32160 count= 67 avg_used= 480 block_cnt= 25 chunk_cnt= 8 mod=OMT_Worker [MEMORY] hold= 42096 used= 28792 count= 59 avg_used= 488 block_cnt= 30 chunk_cnt= 12 mod=[T]ObSQLMockedT [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletPtrMap [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CliTaskMap [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=DagWarnHisBkt [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoBkt [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=Autoincrement [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=TLD_TblCtxMap [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CTBriefMap [MEMORY] hold= 40960 used= 39680 count= 5 avg_used= 7936 block_cnt= 5 chunk_cnt= 5 mod=LSIter [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=HTblRkMapBkt [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=HashBucCliInFo [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=TableAQueryBkt [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=HB_SERVICE [MEMORY] hold= 40896 used= 39680 count= 5 avg_used= 7936 block_cnt= 5 chunk_cnt= 5 mod=MdsTableMgr [MEMORY] hold= 40768 used= 39040 count= 5 avg_used= 7808 block_cnt= 5 chunk_cnt= 5 mod=HashNodePlanCac [MEMORY] hold= 40640 used= 39680 count= 5 avg_used= 7936 block_cnt= 5 chunk_cnt= 1 mod=ObDMMDL [MEMORY] hold= 36096 used= 33216 count= 3 avg_used= 11072 block_cnt= 3 chunk_cnt= 3 mod=DI_CONTAINER [MEMORY] hold= 32768 used= 15872 count= 4 avg_used= 3968 block_cnt= 4 chunk_cnt= 4 mod=CkpDgn [MEMORY] hold= 32768 used= 27960 count= 3 avg_used= 9320 block_cnt= 3 chunk_cnt= 3 mod=PlGdbHelper [MEMORY] hold= 32768 used= 24688 count= 2 avg_used= 12344 block_cnt= 2 chunk_cnt= 2 mod=TLD_AssignedMgr [MEMORY] hold= 32704 used= 25191 count= 2 avg_used= 12595 block_cnt= 2 chunk_cnt= 2 mod=CGReadInfoMgr [MEMORY] hold= 24576 used= 20136 count= 2 avg_used= 10068 block_cnt= 2 chunk_cnt= 2 mod=TabletStats [MEMORY] hold= 24576 used= 19776 count= 1 avg_used= 19776 block_cnt= 1 chunk_cnt= 1 mod=LogApplyStatus [MEMORY] hold= 24576 used= 16392 count= 1 avg_used= 16392 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 17024 used= 8448 count= 3 avg_used= 2816 block_cnt= 2 chunk_cnt= 2 mod=BaseLogWriter [MEMORY] hold= 16384 used= 8200 count= 1 avg_used= 8200 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 16384 used= 15584 count= 2 avg_used= 7792 block_cnt= 2 chunk_cnt= 2 mod=CkptDgnCUNode [MEMORY] hold= 16384 used= 14016 count= 1 avg_used= 14016 block_cnt= 1 chunk_cnt= 1 mod=LogReplayStatus [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=MaBlkLoadMap [MEMORY] hold= 16384 used= 8448 count= 1 avg_used= 8448 block_cnt= 1 chunk_cnt= 1 mod=PalfEnv [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=MedCheckSet [MEMORY] hold= 16384 used= 9792 count= 1 avg_used= 9792 block_cnt= 1 chunk_cnt= 1 mod=TSQLSessionMgr [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=backupTaskSched [MEMORY] hold= 16384 used= 10248 count= 1 avg_used= 10248 block_cnt= 1 chunk_cnt= 1 mod=TabletSvrBucket [MEMORY] hold= 16384 used= 13896 count= 1 avg_used= 13896 block_cnt= 1 chunk_cnt= 1 mod=task_cache [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=TLD_ResourceMgr [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletMap [MEMORY] hold= 12864 used= 12288 count= 3 avg_used= 4096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapDi [MEMORY] hold= 12864 used= 12288 count= 3 avg_used= 4096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapCn [MEMORY] hold= 11008 used= 10624 count= 2 avg_used= 5312 block_cnt= 1 chunk_cnt= 1 mod=Scheduler [MEMORY] hold= 9456 used= 9064 count= 2 avg_used= 4532 block_cnt= 2 chunk_cnt= 2 mod=GROUP_INDEX_MAP [MEMORY] hold= 8192 used= 7792 count= 1 avg_used= 7792 block_cnt= 1 chunk_cnt= 1 mod=CkptDgnMemNode [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=RSCompPairs [MEMORY] hold= 8192 used= 7728 count= 1 avg_used= 7728 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsCache [MEMORY] hold= 8192 used= 7792 count= 1 avg_used= 7792 block_cnt= 1 chunk_cnt= 1 mod=ColUsagNode [MEMORY] hold= 8192 used= 7808 count= 1 avg_used= 7808 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsInfo [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=FillCache [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=GetMediumList [MEMORY] hold= 8192 used= 7712 count= 1 avg_used= 7712 block_cnt= 1 chunk_cnt= 1 mod=DmlStatNode [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=HandleSubmit [MEMORY] hold= 8192 used= 7352 count= 1 avg_used= 7352 block_cnt= 1 chunk_cnt= 1 mod=mrg_prog_cker [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=MediumList [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=GetKeptShotInfo [MEMORY] hold= 8192 used= 7632 count= 1 avg_used= 7632 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoNode [MEMORY] hold= 8192 used= 7632 count= 1 avg_used= 7632 block_cnt= 1 chunk_cnt= 1 mod=CheckMapNode [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=LsElectRefInfo [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=ls_id [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=DASIDCache [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=MatchLsnMap [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=HTableLockMap [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=ShareBlksMap [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=LCLSender [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=MdsDebugMap [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=CPUUSage [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=REPLAY_STATUS [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=APPLY_STATUS [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=leakChecker [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=LSInfo [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=PurgeThrottle [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileMap [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=LockWaitMgr [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileBlkMgr [MEMORY] hold= 6880 used= 5600 count= 7 avg_used= 800 block_cnt= 7 chunk_cnt= 6 mod=DICollector [MEMORY] hold= 5888 used= 3392 count= 8 avg_used= 424 block_cnt= 8 chunk_cnt= 7 mod=SequenceIdMap [MEMORY] hold= 5632 used= 4096 count= 8 avg_used= 512 block_cnt= 7 chunk_cnt= 6 mod=SqlSessiQuerSql [MEMORY] hold= 5216 used= 1976 count= 13 avg_used= 152 block_cnt= 2 chunk_cnt= 2 mod=ObTxLSLogCb [MEMORY] hold= 5120 used= 3392 count= 8 avg_used= 424 block_cnt= 8 chunk_cnt= 7 mod=PackStateMap [MEMORY] hold= 5120 used= 3392 count= 8 avg_used= 424 block_cnt= 8 chunk_cnt= 7 mod=SequenceMap [MEMORY] hold= 5120 used= 3392 count= 8 avg_used= 424 block_cnt= 8 chunk_cnt= 7 mod=ContextsMap [MEMORY] hold= 5056 used= 4656 count= 2 avg_used= 2328 block_cnt= 2 chunk_cnt= 2 mod=MViewService [MEMORY] hold= 4832 used= 4632 count= 1 avg_used= 4632 block_cnt= 1 chunk_cnt= 1 mod=PxPoolBkt [MEMORY] hold= 4832 used= 4632 count= 1 avg_used= 4632 block_cnt= 1 chunk_cnt= 1 mod=ExTabletCntMap [MEMORY] hold= 4832 used= 4632 count= 1 avg_used= 4632 block_cnt= 1 chunk_cnt= 1 mod=ResRuleIdMap [MEMORY] hold= 4832 used= 4632 count= 1 avg_used= 4632 block_cnt= 1 chunk_cnt= 1 mod=ResGrpIdMap [MEMORY] hold= 4288 used= 4096 count= 1 avg_used= 4096 block_cnt= 1 chunk_cnt= 1 mod=MacroFile [MEMORY] hold= 4160 used= 3520 count= 4 avg_used= 880 block_cnt= 3 chunk_cnt= 3 mod=PoolArenaArray [MEMORY] hold= 4096 used= 2328 count= 1 avg_used= 2328 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoMap [MEMORY] hold= 3328 used= 1728 count= 4 avg_used= 432 block_cnt= 4 chunk_cnt= 4 mod=NegParamIdex [MEMORY] hold= 3328 used= 1792 count= 8 avg_used= 224 block_cnt= 1 chunk_cnt= 1 mod=LogIOTask [MEMORY] hold= 3312 used= 3112 count= 1 avg_used= 3112 block_cnt= 1 chunk_cnt= 1 mod=PendingFreeMap [MEMORY] hold= 3312 used= 3112 count= 1 avg_used= 3112 block_cnt= 1 chunk_cnt= 1 mod=TFileFLCtx [MEMORY] hold= 3200 used= 3008 count= 1 avg_used= 3008 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlDfc [MEMORY] hold= 2528 used= 2328 count= 1 avg_used= 2328 block_cnt= 1 chunk_cnt= 1 mod=DASMemHashBuck [MEMORY] hold= 2528 used= 2328 count= 1 avg_used= 2328 block_cnt= 1 chunk_cnt= 1 mod=IRMMemHashBuck [MEMORY] hold= 2528 used= 2328 count= 1 avg_used= 2328 block_cnt= 1 chunk_cnt= 1 mod=MedCheckMap [MEMORY] hold= 2528 used= 2328 count= 1 avg_used= 2328 block_cnt= 1 chunk_cnt= 1 mod=VecIdxLSMgr [MEMORY] hold= 2240 used= 2048 count= 1 avg_used= 2048 block_cnt= 1 chunk_cnt= 1 mod=CreateContext [MEMORY] hold= 2048 used= 1272 count= 1 avg_used= 1272 block_cnt= 1 chunk_cnt= 1 mod=LSCheckMap [MEMORY] hold= 2048 used= 1856 count= 1 avg_used= 1856 block_cnt= 1 chunk_cnt= 1 mod=LogIOWS [MEMORY] hold= 2016 used= 1824 count= 1 avg_used= 1824 block_cnt= 1 chunk_cnt= 1 mod=PxResMgr [MEMORY] hold= 1920 used= 1728 count= 1 avg_used= 1728 block_cnt= 1 chunk_cnt= 1 mod=timer_service [MEMORY] hold= 1808 used= 1424 count= 2 avg_used= 712 block_cnt= 2 chunk_cnt= 2 mod=DetectManager [MEMORY] hold= 1744 used= 1544 count= 1 avg_used= 1544 block_cnt= 1 chunk_cnt= 1 mod=TXBFRetryId [MEMORY] hold= 1744 used= 1544 count= 1 avg_used= 1544 block_cnt= 1 chunk_cnt= 1 mod=TabStatMgrLock [MEMORY] hold= 1472 used= 1272 count= 1 avg_used= 1272 block_cnt= 1 chunk_cnt= 1 mod=HashBucApiGroup [MEMORY] hold= 1472 used= 1272 count= 1 avg_used= 1272 block_cnt= 1 chunk_cnt= 1 mod=DUP_LS_SET [MEMORY] hold= 1472 used= 1272 count= 1 avg_used= 1272 block_cnt= 1 chunk_cnt= 1 mod=GCMemtableMap [MEMORY] hold= 1280 used= 1088 count= 1 avg_used= 1088 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 1248 used= 1056 count= 1 avg_used= 1056 block_cnt= 1 chunk_cnt= 1 mod=LOG_HASH_MAP [MEMORY] hold= 1232 used= 1032 count= 1 avg_used= 1032 block_cnt= 1 chunk_cnt= 1 mod=BFLoadTask [MEMORY] hold= 1200 used= 1000 count= 1 avg_used= 1000 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletMapLk [MEMORY] hold= 1120 used= 120 count= 5 avg_used= 24 block_cnt= 2 chunk_cnt= 2 mod=FreezeTask [MEMORY] hold= 1024 used= 832 count= 1 avg_used= 832 block_cnt= 1 chunk_cnt= 1 mod=tenant_rm_mgr [MEMORY] hold= 976 used= 776 count= 1 avg_used= 776 block_cnt= 1 chunk_cnt= 1 mod=DepTabletSet [MEMORY] hold= 896 used= 696 count= 1 avg_used= 696 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfo [MEMORY] hold= 752 used= 552 count= 1 avg_used= 552 block_cnt= 1 chunk_cnt= 1 mod=LSFreeze [MEMORY] hold= 704 used= 208 count= 1 avg_used= 208 block_cnt= 1 chunk_cnt= 1 mod=[T]ObMacroBlock [MEMORY] hold= 672 used= 472 count= 1 avg_used= 472 block_cnt= 1 chunk_cnt= 1 mod=Election [MEMORY] hold= 640 used= 224 count= 2 avg_used= 112 block_cnt= 1 chunk_cnt= 1 mod=Coordinator [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=MSTXCTX [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=HAScheduler [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=VectorIndexVsag [MEMORY] hold= 544 used= 144 count= 2 avg_used= 72 block_cnt= 2 chunk_cnt= 2 mod=TntSrvObjPool [MEMORY] hold= 512 used= 120 count= 2 avg_used= 60 block_cnt= 1 chunk_cnt= 1 mod=UserResourceMgr [MEMORY] hold= 448 used= 64 count= 2 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfFSCbNode [MEMORY] hold= 416 used= 2 count= 2 avg_used= 1 block_cnt= 2 chunk_cnt= 2 mod=ObLogEXTHandler [MEMORY] hold= 400 used= 200 count= 1 avg_used= 200 block_cnt= 1 chunk_cnt= 1 mod=ExTabletMapLk [MEMORY] hold= 384 used= 102 count= 1 avg_used= 102 block_cnt= 1 chunk_cnt= 1 mod=TxLSLogBuf [MEMORY] hold= 304 used= 104 count= 1 avg_used= 104 block_cnt= 1 chunk_cnt= 1 mod=RedoLogBuckLock [MEMORY] hold= 256 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 256 used= 48 count= 1 avg_used= 48 block_cnt= 1 chunk_cnt= 1 mod=[T]TSITabletTab [MEMORY] hold= 256 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=[T]TSILastOper [MEMORY] hold= 240 used= 41 count= 1 avg_used= 41 block_cnt= 1 chunk_cnt= 1 mod=KeepAliveBuf [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=RebuildCbNode [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfRCCbNode [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=logservice [MEMORY] hold= 389771296 used= 379518723 count= 32899 avg_used= 11535 mod=SUMMARY [2025-08-14 09:06:22.750589] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10880] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 230686720 used= 228884480 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 block_cnt= 110 chunk_cnt= 110 mod=Memstore [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:06:22.750605] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1 ctx_id= EXECUTE_CTX_ID hold= 1163264 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 15 washed_size= 13516800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:22.750621] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 block_cnt= 1 chunk_cnt= 1 mod=PartTranCtxMgr [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:06:22.750643] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 51617792 used= 47452416 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 84 washed_blocks= 334 washed_size= 126640128 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 40241472 used= 36454879 count= 2829 avg_used= 12886 block_cnt= 2829 chunk_cnt= 78 mod=SqlPhyPlan [MEMORY] hold= 3768320 used= 3427728 count= 321 avg_used= 10678 block_cnt= 84 chunk_cnt= 49 mod=SqlPlanCache [MEMORY] hold= 1524288 used= 1400877 count= 158 avg_used= 8866 block_cnt= 158 chunk_cnt= 11 mod=SqlPhyPlObj [MEMORY] hold= 1379840 used= 1261568 count= 616 avg_used= 2048 block_cnt= 223 chunk_cnt= 58 mod=CreateContext [MEMORY] hold= 536448 used= 523776 count= 66 avg_used= 7936 block_cnt= 66 chunk_cnt= 24 mod=OB_SQLSTAT_OBJ [MEMORY] hold= 2048 used= 1664 count= 2 avg_used= 832 block_cnt= 1 chunk_cnt= 1 mod=PlanCache [MEMORY] hold= 47452416 used= 43070492 count= 3992 avg_used= 10789 mod=SUMMARY [2025-08-14 09:06:22.750657] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= WORK_AREA hold= 1458176 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 13 washed_blocks= 66 washed_size= 25804800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:22.750674] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 14204928 used= 10970224 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 277 washed_size= 31932416 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10953744 used= 6065548 count= 18570 avg_used= 326 block_cnt= 372 chunk_cnt= 12 mod=PlJit [MEMORY] hold= 16480 used= 13376 count= 3 avg_used= 4458 block_cnt= 2 chunk_cnt= 2 mod=PlCodeGen [MEMORY] hold= 10970224 used= 6078924 count= 18573 avg_used= 327 mod=SUMMARY [2025-08-14 09:06:22.750690] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 111149056 used= 108896256 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 block_cnt= 211 chunk_cnt= 53 mod=CoStack [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 mod=SUMMARY [2025-08-14 09:06:22.750709] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 56623104 used= 56623104 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 56623104 used= 0 count= 27 avg_used= 0 block_cnt= 0 chunk_cnt= 0 mod=KvstorCacheMb [MEMORY] hold= 56623104 used= 0 count= 27 avg_used= 0 mod=SUMMARY [2025-08-14 09:06:22.750728] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 34365376 limit= 429496720 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 29491200 used= 26236400 count= 400 avg_used= 65591 block_cnt= 400 chunk_cnt= 15 mod=L_TabletPool [MEMORY] hold= 4784128 used= 4765440 count= 73 avg_used= 65280 block_cnt= 73 chunk_cnt= 3 mod=N_TabletPool [MEMORY] hold= 90048 used= 84768 count= 2 avg_used= 42384 block_cnt= 2 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 34365376 used= 31086608 count= 475 avg_used= 65445 mod=SUMMARY [2025-08-14 09:06:22.750740] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 8003584 used= 4947392 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 10 washed_blocks= 23 washed_size= 12967936 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4947392 used= 4793344 count= 604 avg_used= 7936 block_cnt= 66 chunk_cnt= 8 mod=MemtableCallbac [MEMORY] hold= 4947392 used= 4793344 count= 604 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:22.750753] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=LobPersist [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:19.762815] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2983458, owner: ObLog, click_count: 7, time dist:[FORMAT_END=9, ObMalloc#start=12, ObMalloc#end=1983820, ObMalloc#start=975033, ObMalloc#end=24580, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:20.740366] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2008142, owner: ObLog, click_count: 7, time dist:[FORMAT_END=8, ObMalloc#start=2, ObMalloc#end=1021844, ObMalloc#start=950654, ObMalloc#end=11797, ALLOC_END=23834, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:18.750700] EDIAG [SHARE] refresh (ob_task_define.cpp:396) [280][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=872882][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3995814, owner: ObLog, click_count: 7, time dist:[FORMAT_END=1010096, ObMalloc#start=1, ObMalloc#end=1982069, ObMalloc#start=977307, ObMalloc#end=26336, ALLOC_END=2, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1fd6cd17 0x96c37e0 0x1fda2168 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:22.750765] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 65536 used= 31024 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 2031616 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 31024 used= 27058 count= 20 avg_used= 1352 block_cnt= 6 chunk_cnt= 1 mod=SqlPsCache [MEMORY] hold= 31024 used= 27058 count= 20 avg_used= 1352 mod=SUMMARY [2025-08-14 09:06:19.762815] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:22.750790] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1 ctx_id= RPC_CTX_ID hold= 14827520 used= 6135808 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 5947392 used= 3933952 count= 242 avg_used= 16256 block_cnt= 242 chunk_cnt= 5 mod=OB_WRS_CLUSTER_ [MEMORY] hold= 98304 used= 65024 count= 4 avg_used= 16256 block_cnt= 4 chunk_cnt= 2 mod=OB_LOG_REGISTER [MEMORY] hold= 24576 used= 11904 count= 3 avg_used= 3968 block_cnt= 2 chunk_cnt= 1 mod=[L]OB_WRS_CLUST [MEMORY] hold= 24576 used= 16256 count= 1 avg_used= 16256 block_cnt= 1 chunk_cnt= 1 mod=OB_TENANT_MGR [MEMORY] hold= 24576 used= 16256 count= 1 avg_used= 16256 block_cnt= 1 chunk_cnt= 1 mod=OB_LOG_GET_STAT [MEMORY] hold= 8192 used= 3968 count= 1 avg_used= 3968 block_cnt= 1 chunk_cnt= 1 mod=[L]OB_LOG_REGIS [MEMORY] hold= 8192 used= 3968 count= 1 avg_used= 3968 block_cnt= 1 chunk_cnt= 1 mod=[L]OB_TENANT_MG [MEMORY] hold= 6135808 used= 4051328 count= 253 avg_used= 16013 mod=SUMMARY [2025-08-14 09:06:22.750806] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2983458][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:22.750810] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 3481600 used= 2391232 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 10 washed_size= 4907008 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2121728 used= 2105152 count= 3 avg_used= 701717 block_cnt= 3 chunk_cnt= 3 mod=MEMTABLE_ARENA [MEMORY] hold= 261376 used= 253952 count= 32 avg_used= 7936 block_cnt= 21 chunk_cnt= 2 mod=TX_DATA_SLICE [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=TX_DATA_ARENA [MEMORY] hold= 2391232 used= 2367040 count= 36 avg_used= 65751 mod=SUMMARY [2025-08-14 09:06:22.750823] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:22.750827] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 3964928 used= 3080192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 3 washed_size= 229376 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3080192 used= 2983936 count= 376 avg_used= 7936 block_cnt= 24 chunk_cnt= 2 mod=MdsTable [MEMORY] hold= 3080192 used= 2983936 count= 376 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:22.750837] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:22.750844] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= SCHEMA_SERVICE hold= 212992 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1884160 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=NonPartTenCac [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:22.725676] INFO dump_info (ob_timer_service.cpp:728) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=29] dump info [summary](this=0x7f99230b0af0, this={this:0x7f99230b0af0, tenant_id_:1002, priority_task_queue_.size():41, running_task_set_.size():26, uncanceled_task_set_.size():0, worker_thread_pool_.get_queue_num():10}) [2025-08-14 09:06:22.750868] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:22.750881] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= MERGE_NORMAL_CTX_ID hold= 884736 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 6 washed_blocks= 9 washed_size= 11698176 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:22.750860] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=25145] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a5998, timer:0x7f9935f8df20, task:0x7f99200f5a78, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr15MediumCheckTaskE", scheduled_time:1755161721762022, pushed_time:0, delay:20000000}) [2025-08-14 09:06:22.750900] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant: 500, limit: 9223372036854775807 hold: 686923776 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 329371648 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 37240832 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 85983232 limit= 9223372036854775807 [MEMORY] ctx_id= LIBEASY hold_bytes= 3145728 limit= 9223372036854775807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 12582912 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 3104768 limit= 9223372036854775807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 37584896 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 78925824 limit= 9223372036854775807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 98983936 limit= 9223372036854775807 [2025-08-14 09:06:22.750904] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=43] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a6418, timer:0x7f994b1dfb70, task:0x7f9920110068, task_type:"N9oceanbase5share22ObIndexUsageReportTaskE", scheduled_time:1755161727099859, pushed_time:0, delay:900000000}) [2025-08-14 09:06:22.750913] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=9] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a6bf8, timer:0x7f994b1dfb70, task:0x7f99201e29f8, task_type:"N9oceanbase5table21ObTableGroupCommitMgr19ObTableGroupOpsTaskE", scheduled_time:1755161732717636, pushed_time:0, delay:1000000}) [2025-08-14 09:06:22.750930] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=17] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a6878, timer:0x7f994b1dfb70, task:0x7f9920110178, task_type:"N9oceanbase5share27ObIndexUsageRefreshConfTaskE", scheduled_time:1755161733727027, pushed_time:0, delay:2000000}) [2025-08-14 09:06:22.750938] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=8] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a6958, timer:0x7f994b1dfb70, task:0x7f9920110810, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161734741162, pushed_time:0, delay:5000000}) [2025-08-14 09:06:22.750947] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=9] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a6a38, timer:0x7f994b1dfb70, task:0x7f9920110af8, task_type:"N9oceanbase5share23ObStorageIOUsageRepoter23TenantIOUsageReportTaskE", scheduled_time:1755161734803774, pushed_time:0, delay:5000000}) [2025-08-14 09:06:22.750955] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=8] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a6e98, timer:0x7f994b1dfb70, task:0x7f99201075a8, task_type:"N9oceanbase3omt11ObTenantSrs27TenantSrsUpdatePeriodicTaskE", scheduled_time:1755161734810627, pushed_time:0, delay:0}) [2025-08-14 09:06:22.750964] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=9] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a64f8, timer:0x7f994b1dfb70, task:0x7f992010e1b0, task_type:"N9oceanbase5table20ObTableObjectPoolMgr26ObTableSessEliminationTaskE", scheduled_time:1755161735735271, pushed_time:0, delay:5000000}) [2025-08-14 09:06:22.750974] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=10] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a65d8, timer:0x7f994b1dfb70, task:0x7f992010e1c8, task_type:"N9oceanbase5table20ObTableObjectPoolMgr27ObTableSessSysVarUpdateTaskE", scheduled_time:1755161735797729, pushed_time:0, delay:5000000}) [2025-08-14 09:06:22.750983] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=8] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a66b8, timer:0x7f994b1dfb70, task:0x7f992010e1e0, task_type:"N9oceanbase5table20ObTableObjectPoolMgr28ObTableReqResEliminationTaskE", scheduled_time:1755161735811808, pushed_time:0, delay:5000000}) [2025-08-14 09:06:22.750991] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=8] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a4ff8, timer:0x7f994b1dfb70, task:0x7f9922ff3e50, task_type:"N9oceanbase3sql3dtl23ObDTLIntermResultGCTaskE", scheduled_time:1755161742809616, pushed_time:0, delay:10000000}) [2025-08-14 09:06:22.750999] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=8] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a5b58, timer:0x7f99535c3f30, task:0x7f9922ffee90, task_type:"N9oceanbase7storage21ObTenantFreezeInfoMgr24UpdateLSResvSnapshotTaskE", scheduled_time:1755161744768148, pushed_time:0, delay:10000000}) [2025-08-14 09:06:22.751007] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=8] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a7838, timer:0x7f994b1dfb70, task:0x7f9920110820, task_type:"N9oceanbase10rootserver16ObMViewMdsOpTaskE", scheduled_time:1755161749755625, pushed_time:0, delay:5000000}) [2025-08-14 09:06:22.751015] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=8] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a6db8, timer:0x7f994b1dfb70, task:0x7f99201e33a0, task_type:"N9oceanbase5table20ObTableClientInfoMgr27ObTableClientInfoRetireTaskE", scheduled_time:1755161750768689, pushed_time:0, delay:60000000}) [2025-08-14 09:06:22.751030] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=14] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a4d58, timer:0x7f994b1dfb70, task:0x7f991f004030, task_type:"N9oceanbase8observer20ObTableQueryASyncMgrE", scheduled_time:1755161750817104, pushed_time:0, delay:0}) [2025-08-14 09:06:22.751038] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=9] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a7678, timer:0x7f994b1dfb70, task:0x7f9920110450, task_type:"N9oceanbase10rootserver21ObMLogMaintenanceTaskE", scheduled_time:1755161754754755, pushed_time:0, delay:0}) [2025-08-14 09:06:22.751049] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=10] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a7758, timer:0x7f994b1dfb70, task:0x7f9920110530, task_type:"N9oceanbase10rootserver22ObMViewMaintenanceTaskE", scheduled_time:1755161754754764, pushed_time:0, delay:0}) [2025-08-14 09:06:22.751057] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=9] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a7bb8, timer:0x7f994b1dfb70, task:0x7f9920110628, task_type:"N9oceanbase10rootserver34ObMViewRefreshStatsMaintenanceTaskE", scheduled_time:1755161754754765, pushed_time:0, delay:0}) [2025-08-14 09:06:22.751075] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=17] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a7058, timer:0x7f994b1dfb70, task:0x7f98ebbf7c30, task_type:"N9oceanbase8observer26ObTableLoadResourceManager21ObRefreshAndCheckTaskE", scheduled_time:1755161774744853, pushed_time:0, delay:30000000}) [2025-08-14 09:06:22.751083] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=8] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a6b18, timer:0x7f994b1dfb70, task:0x7f9920110718, task_type:"N9oceanbase10rootserver25ObMViewPushRefreshScnTaskE", scheduled_time:1755161774754765, pushed_time:0, delay:30000000}) [2025-08-14 09:06:22.751091] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=8] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a73d8, timer:0x7f994b1dfb70, task:0x7f9920110748, task_type:"N9oceanbase10rootserver22ObReplicaSafeCheckTaskE", scheduled_time:1755161774754766, pushed_time:0, delay:0}) [2025-08-14 09:06:22.751100] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=8] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a6cd8, timer:0x7f994b1dfb70, task:0x7f99201107e0, task_type:"N9oceanbase10rootserver24ObCollectMvMergeInfoTaskE", scheduled_time:1755161774755618, pushed_time:0, delay:30000000}) [2025-08-14 09:06:22.751108] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=8] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a74b8, timer:0x7f994b1dfb70, task:0x7f9920110730, task_type:"N9oceanbase10rootserver23ObMViewPushSnapshotTaskE", scheduled_time:1755161804754766, pushed_time:0, delay:60000000}) [2025-08-14 09:06:22.751116] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=8] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a7218, timer:0x7f994b1dfb70, task:0x7f99201107f8, task_type:"N9oceanbase10rootserver24ObMViewCleanSnapshotTaskE", scheduled_time:1755161804755624, pushed_time:0, delay:60000000}) [2025-08-14 09:06:22.751124] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=8] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a4e38, timer:0x7f99535f3df0, task:0x7f9922ff6bf0, task_type:"N9oceanbase7obmysql15ObEliminateTaskE", scheduled_time:1755161931284195, pushed_time:0, delay:500000}) [2025-08-14 09:06:22.751135] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=10] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a6178, timer:0x7f99200ff180, task:0x7f99200ff220, task_type:"N9oceanbase8observer18ObTableLoadService21ObClientTaskPurgeTaskE", scheduled_time:1755162377739421, pushed_time:0, delay:1000000}) [2025-08-14 09:06:22.751143] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=8] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a5d18, timer:0x7f99200ff180, task:0x7f99200ff1a8, task_type:"N9oceanbase8observer18ObTableLoadService17ObCheckTenantTaskE", scheduled_time:1755162377761774, pushed_time:0, delay:1000000}) [2025-08-14 09:06:22.751151] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=8] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a5fb8, timer:0x7f99200ff180, task:0x7f99200ff1f0, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162378750218, pushed_time:0, delay:1000000}) [2025-08-14 09:06:22.751159] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=8] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a4498, timer:0x7f9966a63f10, task:0x7f9922e05010, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr11TableGCTaskE", scheduled_time:1755162378768413, pushed_time:0, delay:20000}) [2025-08-14 09:06:22.751171] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 329371648 used= 301978480 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 501 washed_blocks= 2232 washed_size= 926711808 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 52641792 used= 50996736 count= 3317 avg_used= 15374 block_cnt= 1499 chunk_cnt= 42 mod=TZInfoArray [MEMORY] hold= 35672064 used= 35651584 count= 1 avg_used= 35651584 block_cnt= 1 chunk_cnt= 1 mod=qdtable [MEMORY] hold= 33292288 used= 33275904 count= 16 avg_used= 2079744 block_cnt= 16 chunk_cnt= 16 mod=ObTxDesc [MEMORY] hold= 25067136 used= 24984064 count= 160 avg_used= 156150 block_cnt= 106 chunk_cnt= 50 mod=TenantCtxAlloca [MEMORY] hold= 12873728 used= 12856400 count= 1 avg_used= 12856400 block_cnt= 1 chunk_cnt= 1 mod=MemDumpContext [MEMORY] hold= 7397376 used= 7340256 count= 7 avg_used= 1048608 block_cnt= 7 chunk_cnt= 7 mod=ConcurHashMap [MEMORY] hold= 7275040 used= 6334546 count= 1107 avg_used= 5722 block_cnt= 571 chunk_cnt= 6 mod=CharsetInit [MEMORY] hold= 5373952 used= 5236736 count= 32 avg_used= 163648 block_cnt= 32 chunk_cnt= 15 mod=Iterator ? UNION SELECT task_id, (CASE WHEN status = ? AND (tablet_list is null OR tablet_list = ?) THEN ? ELSE status END) AS STATUS FROM __all_transfer_task_history WHERE task_id > ?) [2025-08-14 09:03:56.786285] WDIAG [DETECT] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [561][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=1991011][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:03:56.786285] EDIAG [DETECT] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [561][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=1991011][errcode=0] LOGGER COST TOO MUCH TIME, cost: 147740892, owner: ObLog, click_count: 7, time dist:[FORMAT_END=22, ObMalloc#start=0, ObMalloc#end=820, ObMalloc#start=147740040, ObMalloc#end=3, ALLOC_END=0, APPEND_END=7], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ea7c4 0x9818f9c 0x973a310 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:24.751356] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=19] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:24.751875] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:24.751887] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:24.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:24.752154] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:24.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:24.752143] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=35] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:24.752183] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:24.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:24.752223] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1962352] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161974728707, last_flushed_end_lsn:{lsn:1604090331}}]) [2025-08-14 09:06:24.752286] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:24.752269] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=41] current log_task status(palf_id=1, self="172.77.0.7:2882", log_id=8607756, log_task={header:{begin_lsn:{lsn:1604090090}, end_lsn:{lsn:1604090203}, log_id:8607756, min_scn:{val:1755161749741193001, v:0}, max_scn:{val:1755161749741193001, v:0}, data_len:57, proposal_id:50, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:1604090090}, data_checksum:2031283194, accum_checksum:759186326, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1755161750716165, freeze_ts:1755161750716173, submit_ts:1755161750716183, flushed_ts:1755161853765770, gen_to_freeze cost time:8, gen_to_submit cost time:18, submit_to_flush cost time:103049587}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161974728707, last_flushed_end_lsn:{lsn:1604090331}}]) [2025-08-14 09:06:24.752715] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [807][T1002_TimerWK3_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=27] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):258, sequence:1}) [2025-08-14 09:06:24.753117] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=36] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:24.753493] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:24.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:24.754338] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:24.756036] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:24.756131] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=14] PNIO [ratelimit] time: 1755162384756129, bytes: 0, bw: 0.000000 MB/s, add_ts: 1000120, add_bytes: 0 [2025-08-14 09:06:24.757417] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:57.758637] INFO [COMMON] loop_finished_dag_net_list (ob_tenant_dag_scheduler.cpp:3868) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8719] loop finsihed dag net list(ret=0, finished_dag_net_list_size=0) [2025-08-14 09:03:57.758637] EDIAG [COMMON] loop_finished_dag_net_list (ob_tenant_dag_scheduler.cpp:3868) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8719][errcode=0] LOGGER COST TOO MUCH TIME, cost: 146777648, owner: ObLog, click_count: 7, time dist:[FORMAT_END=981563, ObMalloc#start=1, ObMalloc#end=56, ObMalloc#start=145796019, ObMalloc#end=3, ALLOC_END=0, APPEND_END=6], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x200612d4 0x20070ebe 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:24.759761] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:24.761501] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:24.762216] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:24.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:24.762259] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:24.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:24.762915] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7277230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:24.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:24.762946] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7272330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a5090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:24.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:24.762965] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919822b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:24.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:24.762976] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:24.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:24.762992] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d71b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:24.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:24.763004] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:24.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:24.763017] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a27b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:24.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:24.763030] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a40b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:24.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:24.763042] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7faf30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:24.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:24.766222] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:24.770222] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:24.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:24.772346] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=37] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:06:24.772962] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7352330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196fc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:24.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:24.775174] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:24.782451] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:24.782459] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=19] gc stale ls task succ [2025-08-14 09:06:24.783018] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aca90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:24.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:24.783047] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f734fc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891918610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:24.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:24.783209] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:24.793095] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:24.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:24.793128] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:24.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:24.800396] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=55][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:24.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:24.803168] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:24.794, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:24.813220] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:24.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:24.813264] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:24.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:24.813274] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:24.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:24.813291] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7397b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:24.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:24.813319] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918299b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891916210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:24.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:24.827286] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=17] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:24.827334] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:24.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:24.833312] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:24.825, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:24.833344] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:24.826, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:24.833355] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:24.820, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:24.833379] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72879b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:24.828, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:24.851949] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:24.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:24.853362] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:24.853429] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:24.853463] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:24.846, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:24.853483] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:24.853523] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:24.853573] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:24.853597] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:24.853613] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:24.862068] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6b18, timer:0x7f994b1dfb70, task:0x7f9920110718, task_type:"N9oceanbase10rootserver25ObMViewPushRefreshScnTaskE", scheduled_time:1755161774754765, pushed_time:0, delay:30000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=610107301, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:24.875247] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:24.927405] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:24.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:24.952028] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:24.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:24.975305] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:24.995168] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:24.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.014697] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:25.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.014732] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:25.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.021420] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:25.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.027462] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:24.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.052073] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:24.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.075378] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:06:25.075414] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:25.095121] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:25.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.127525] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:25.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.152097] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:25.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.175436] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:25.215767] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:25.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.227589] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:25.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.228979] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:25.229014] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:25.229031] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:25.229047] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:25.229072] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:25.229094] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:25.229111] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:25.229131] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:25.229146] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:25.229161] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:25.229189] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:25.229202] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:25.229215] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:25.229245] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:25.229253] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:25.229266] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:25.229282] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:25.229301] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:25.229328] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:25.229502] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:25.229520] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:25.246535] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:25.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.252123] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:25.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.256592] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:25.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.266137] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:25.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.266169] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:25.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.272860] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:25.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.275499] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:25.303038] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:25.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.327651] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:25.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.352212] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:25.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.363389] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] EASY SLOW: thread: 7f995427d300, lastrun: 1755162250.789339 cost: 134.574047 loop:26617, slowcnt: 1020 [2025-08-14 09:06:25.367434] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=25] gc stale ls task succ [2025-08-14 09:06:25.375535] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:25.427665] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:25.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.452273] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=49][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:25.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.475565] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:25.498063] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:25.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.517556] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:25.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.517594] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:25.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.524368] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:25.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.527727] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:25.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.552326] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:25.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.575600] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:25.598005] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:25.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.604628] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEC7-0-0] [lt=27] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:06:25.627747] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:25.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.652378] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:25.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.675637] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:24.752756] WDIAG [PALF] runTimerTask (block_gc_timer_task.cpp:94) [807][T1002_TimerWK3_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4389] try_recycle_blocks cost too much time(ret=-4389, cost_time_us=36235, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.77.0.7:2882", log_dir:"/root/ob/observer/store/clog/tenant_1002", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):258, sequence:1}, log_alloc_mgr_:{flying_log_task:1, flying_meta_task:2}}) [2025-08-14 09:06:25.713675] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=959465][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a4818, timer:0x7f99335d9ee0, task:0x7f991c5e9398, task_type:"N9oceanbase4palf16BlockGCTimerTaskE", scheduled_time:1755162377798644, pushed_time:1755162377798713, delay:30000}, delay_in_thread_pool=6917802, threashold=500000) [2025-08-14 09:06:25.716104] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=24] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:25.716999] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=26] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:25.718589] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:25.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.720898] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:25.726317] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=22] gc stale ls task succ [2025-08-14 09:06:25.728407] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:25.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.728478] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72760b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.719, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.728568] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:25.729380] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=25] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:25.729691] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:25.729722] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:25.729748] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:25.729770] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:25.729780] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:25.729793] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:25.729810] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:25.729840] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:25.729852] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:25.729864] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:25.729896] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:25.729909] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:25.729931] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:25.729948] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:25.729961] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:25.729974] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:25.729988] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:25.730004] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:25.730015] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:25.730189] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:25.730207] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:25.718046] WDIAG [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:180) [561][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=147740892][errcode=-4016] ObLCLBatchSenderThread is too busy(end_ts=1755162384750923, begin_ts=1755162235787248, diff=148963675, *this={this:0x7f994b3ee2b0, is_inited:true, is_running:true, total_record_time:0, over_night_times:1}) [2025-08-14 09:06:25.731317] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [561][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=13246] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=100, total_constructed_detector=16, total_destructed_detector=16, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f994b3ee2b0, is_inited:true, is_running:true, total_record_time:148963675, over_night_times:1}) [2025-08-14 09:06:25.731341] WDIAG [DETECT] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [561][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=22][errcode=0] cost too much time:ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t), (this=|threshold=100.00ms|start at 09:06:25.731|total=980.43ms) [2025-08-14 09:06:25.740409] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:25.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.742033] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.742068] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.742101] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739be30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.742119] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.742182] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:25.742206] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=19] temporary file module use no blocks [2025-08-14 09:06:25.742218] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:25.742251] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=31] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:25.742269] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=17] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623287181890, flush_io_finished_ret_:0, flush_io_finished_round_:26442, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:25.745470] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:25.713706] WDIAG alarm_if_necessary (ob_timer_service.cpp:143) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4389] timer task's elapsed_time is greater than period(token={this:0x7f991b7a4818, timer:0x7f99335d9ee0, task:0x7f991c5e9398, task_type:"N9oceanbase4palf16BlockGCTimerTaskE", scheduled_time:1755162377798644, pushed_time:1755162377798713, delay:30000}, elapsed_time=997159, period=30000) [2025-08-14 09:06:25.749245] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a4c78, timer:0x7f99329e1b10, task:0x7f98e4e091f8, task_type:"N9oceanbase5table21ObTTLAllTaskSchedulerE", scheduled_time:1755162384727446, pushed_time:0, delay:15000000}, running_task_token={this:0x7f991b7a6f78, timer:0x7f99329e1b10, task:0x7f98e4e091c0, task_type:"N9oceanbase5table21ObClearTTLHistoryTaskE", scheduled_time:1755162380725332, pushed_time:1755162380726731, delay:30000000}, total_delay=1021797, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:24.841501] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=911810] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:06:24.830998] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=2915859] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:06:25.748593] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=2003889] blacklist_loop exec finished(cost_time=17, is_enabled=true, send_cnt=0) [2025-08-14 09:06:24.830897] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=2891249] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:06:25.749936] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:25.751105] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:25.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.751644] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=26] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:25.752901] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7edc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.752938] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:25.752947] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ebe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.752963] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.752972] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:25.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.755714] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=41] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:25.755767] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:25.755839] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:03:58.741771] INFO [STORAGE.TRANS] statistics (ob_location_adapter.cpp:70) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=960378] location adapter statistics(renew_access=312, total_access=5445, error_count=0, renew_rate=5.728975311e-02) [2025-08-14 09:06:25.756671] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:25.757257] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=44] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:25.757526] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:25.758922] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=35429][errcode=-4389] timer task too much delay in priority queue(token={this:0x7f991b7a6098, timer:0x7f99200ff180, task:0x7f99200ff208, task_type:"N9oceanbase8observer18ObTableLoadService25ObClientTaskAutoAbortTaskE", scheduled_time:1755162370767586, pushed_time:1755162378749107, delay:1000000}, delay_in_priority_queue=7981521, threashold=1000000) [2025-08-14 09:06:25.758954] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a6098, timer:0x7f99200ff180, task:0x7f99200ff208, task_type:"N9oceanbase8observer18ObTableLoadService25ObClientTaskAutoAbortTaskE", scheduled_time:1755162370767586, pushed_time:1755162378749107, delay:1000000}, delay_in_thread_pool=7000098, threashold=500000) [2025-08-14 09:06:25.761376] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:25.762355] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=30] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:25.763075] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=21] PNIO [ratelimit] time: 1755162385763073, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006944, add_bytes: 0 [2025-08-14 09:06:25.763173] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.750, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.763212] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.763228] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728d1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7291110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.766325] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:25.769649] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:25.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.769684] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:25.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.773245] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aa990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.773278] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.773289] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919afe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.773314] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d62b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.773327] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.773894] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:25.773924] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=25] temporary file module use no blocks [2025-08-14 09:06:25.773935] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:25.773951] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:25.773969] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=18] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623287213592, flush_io_finished_ret_:0, flush_io_finished_round_:26404, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:25.775684] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:25.776357] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:25.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.782472] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:25.782513] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=14] gc stale ls task succ [2025-08-14 09:06:25.782550] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=60][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a73d8, timer:0x7f994b1dfb70, task:0x7f9920110748, task_type:"N9oceanbase10rootserver22ObReplicaSafeCheckTaskE", scheduled_time:1755161774754766, pushed_time:0, delay:0}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=611027783, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:25.782574] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6cd8, timer:0x7f994b1dfb70, task:0x7f99201107e0, task_type:"N9oceanbase10rootserver24ObCollectMvMergeInfoTaskE", scheduled_time:1755161774755618, pushed_time:0, delay:30000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=611026956, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:25.783324] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:25.783322] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.783347] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.783358] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.783377] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7397a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.793403] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.793448] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.793469] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.803464] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.806533] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:25.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.813517] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.802, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.823564] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.814, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.823598] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.817, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.823624] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.823650] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.819, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.823665] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.813, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.827933] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=18] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:25.828470] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:25.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.833576] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919831b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.828, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.843623] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.835, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.853035] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:25.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.853683] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.847, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.853719] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.844, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.853730] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:25.844, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:25.855871] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fc8b0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcb90, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:1, time_interval:3.00s, expected_run_time:09:06:25.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.855941] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:25.855995] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:25.856044] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:25.856098] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:25.856142] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:25.856164] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:25.856180] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:25.872710] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:25.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.875720] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:25.928518] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:25.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.953101] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:25.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.972773] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:25.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.975793] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:25.992563] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:25.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.010956] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:26.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.010989] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:26.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.028047] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:26.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.028545] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:25.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.053118] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:25.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.072850] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:25.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.075903] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:26.091396] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:26.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.128607] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:26.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.153194] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:26.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.172871] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:26.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.175911] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:26.212031] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:26.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.228630] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:26.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.230387] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:26.230421] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:26.230445] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:26.230460] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:26.230474] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:26.230494] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:26.230510] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:26.230531] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:26.230546] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:26.230569] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:26.230589] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:26.230604] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:26.230620] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:26.230637] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:26.230657] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:26.230669] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:26.230681] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:26.230695] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:26.230706] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:26.230885] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:26.230910] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:26.244044] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:26.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.253245] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:26.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.254130] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:26.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.262329] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:26.255, task_priority:1, with_handle_protected:1}) [856]OB_MALLOC COST TOO MUCH TIME, cost_time=457399, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=457397], seq:[0] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=471352, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=471346], seq:[0] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=461702, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=461696], seq:[0] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=406071, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=406066], seq:[0] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=406080, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=406078], seq:[0, 1] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=381593, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=381588], seq:[0] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=381602, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=381601], seq:[0, 1] [2025-08-14 09:06:26.328692] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:26.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.735428] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=406117][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:26.328|207=406.73ms|210=0us|222=1us|total=406.76ms) [2025-08-14 09:06:26.735455] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4389] timer task use too much time(end=1755162386735454, start=1755162386328690, delta=406764) [2025-08-14 09:03:58.741771] EDIAG [STORAGE.TRANS] statistics (ob_location_adapter.cpp:70) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=960378][errcode=0] LOGGER COST TOO MUCH TIME, cost: 146792199, owner: ObLog, click_count: 7, time dist:[FORMAT_END=41105, ObMalloc#start=2, ObMalloc#end=47, ObMalloc#start=146751037, ObMalloc#end=2, ALLOC_END=0, APPEND_END=6], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1b52a633 0x1b52a9c4 0x95df39e 0x1b5009be 0x1b50387c 0x1b8905de 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [856]OB_MALLOC COST TOO MUCH TIME, cost_time=468261, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=457425], seq:[0, 1] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=471327, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=461722], seq:[0, 1] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=482775, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=471365], seq:[0, 1] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=471770, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=471768], seq:[0] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=387663, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=387657], seq:[0] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=387676, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=387671], seq:[0, 1] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=384000, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=383995], seq:[0] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=384008, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=384007], seq:[0, 1] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=256455, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=256446, ALLOC_CHUNK_END=4], seq:[0, 1] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=123986, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=123977, ALLOC_CHUNK_END=5], seq:[0, 1] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=124008, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=123995, WASH_BLOCK_END=13], seq:[0, 1, 2] [2025-08-14 09:06:26.353289] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:26.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.735511] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=381641][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:26.353|207=382.22ms|210=0us|222=1us|total=382.23ms) [2025-08-14 09:06:26.735526] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] timer task use too much time(end=1755162386735525, start=1755162386353287, delta=382238) [2025-08-14 09:06:25.743468] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:981) [830][T1002_GCCollect][T1001][YB42AC4D0007-00063C496069AFFA-0-0] [lt=17089][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1002"}, record_ret=-4012, ret=-4012) [2025-08-14 09:06:26.275974] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:26.744986] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=468298][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:26.272946] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:26.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.745022] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=471358][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:26.272|207=472.07ms|210=0us|222=2us|total=472.09ms) [2025-08-14 09:06:26.745041] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4389] timer task use too much time(end=1755162386745040, start=1755162386272945, delta=472095) [2025-08-14 09:06:26.262361] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:26.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.745896] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=482801][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:26.262|207=483.53ms|210=0us|222=1us|total=483.55ms) [2025-08-14 09:06:26.745912] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] timer task use too much time(end=1755162386745911, start=1755162386262361, delta=483550) [2025-08-14 09:06:26.746842] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a4658, timer:0x7f9966a63f10, task:0x7f9922e05038, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr12TabletGCTaskE", scheduled_time:1755162377770297, pushed_time:1755162378749140, delay:20000}, delay_in_thread_pool=7009833, threashold=500000) [2025-08-14 09:06:26.746862] WDIAG alarm_if_necessary (ob_timer_service.cpp:143) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task's elapsed_time is greater than period(token={this:0x7f991b7a4658, timer:0x7f9966a63f10, task:0x7f9922e05038, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr12TabletGCTaskE", scheduled_time:1755162377770297, pushed_time:1755162378749140, delay:20000}, elapsed_time=987869, period=20000) [2025-08-14 09:06:26.745925] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=0] task scheduled too delay(scheduled_delay=240436, expected_run_ts_us_=1755162386505487, ObClockGenerator::getRealClock()=1755162386745924, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:26.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.748119] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=2188][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:26.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.748147] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] task scheduled too delay(scheduled_delay=158830, expected_run_ts_us_=1755162386589316, ObClockGenerator::getRealClock()=1755162386748146, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:26.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.748160] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:26.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.748171] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:26.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.751925] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:26.363792] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] EASY SLOW: thread: 7f995427d300, lastrun: 1755162250.789339 cost: 135.574451 loop:26617, slowcnt: 1030 [2025-08-14 09:06:26.367505] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=24] gc stale ls task succ [2025-08-14 09:06:26.729435] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:26.752764] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23293][errcode=0] epoll handler too much time :23349 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:26.752822] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:25.772439] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=20] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [660]OB_MALLOC COST TOO MUCH TIME, cost_time=475368, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=471784], seq:[0, 1] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=268907, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=256461, WASH_BLOCK_END=114, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=12328], seq:[0, 1, 2, 3, 4] [2025-08-14 09:06:26.753168] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=36063][errcode=0] epoll handler too much time :36130 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:26.753197] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25575][errcode=0] epoll handler too much time :25626 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:26.731098] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:26.753245] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22094][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:26.753258] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:26.753267] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:26.753276] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:26.753288] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:26.753297] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:26.753309] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:26.753318] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:26.753326] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:26.753338] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:26.753346] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:26.753354] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:26.753364] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:26.753373] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:26.753381] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:26.753390] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:26.753401] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:26.753411] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:26.753572] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:26.753583] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:26.628069] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ebbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:26.615, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:26.755231] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=126963][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:26.628|207=127.16ms|210=0us|222=2us|total=127.19ms) [2025-08-14 09:06:26.755260] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4389] timer task use too much time(end=1755162386755259, start=1755162386628067, delta=127192) [2025-08-14 09:06:26.755271] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919abd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:26.721, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:26.755284] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919abb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:26.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:26.755294] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:26.726, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:26.279473] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:26.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.755611] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=475415][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:26.279|207=476.14ms|210=0us|222=1us|total=476.15ms) [2025-08-14 09:06:26.755624] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task use too much time(end=1755162386755623, start=1755162386279471, delta=476152) [2025-08-14 09:06:26.755631] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=5][errcode=0] task scheduled too delay(scheduled_delay=463281, expected_run_ts_us_=1755162386292349, ObClockGenerator::getRealClock()=1755162386755630, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:26.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.755649] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:26.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.755310] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fdc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:26.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:26.758343] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=3023][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:26.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:26.758368] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728be10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:26.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:26.758379] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72847b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:26.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:26.758671] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=327] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:26.758840] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:26.758961] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:26.759015] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=16] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:26.759115] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:26.742565] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=38] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:26.759456] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=16859] temporary file module use no blocks [2025-08-14 09:06:26.759464] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:26.743389] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:26.759534] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16120][errcode=0] epoll handler too much time :16154 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:26.743419] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12054] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:26.558037] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=1767][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fe030, caller:ob_table_lock_service.cpp:start:276, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE6$_1273, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe310, time_wheel:0x7f99535f3990, thread_pool:0x7f99535f3170, is_running:1, time_interval:10.00s, expected_run_time:09:06:26.426, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.761726] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=203375][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:26.558|207=203.69ms|210=0us|222=1us|total=203.71ms) [2025-08-14 09:06:26.761744] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4389] timer task use too much time(end=1755162386761744, start=1755162386558034, delta=203710) [2025-08-14 09:06:26.759817] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=16371] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:26.762308] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:26.759475] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:26.762354] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=2875] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623288182076, flush_io_finished_ret_:0, flush_io_finished_round_:26443, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:26.495269] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:26.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.764642] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=268962][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:26.495|207=269.37ms|210=1us|222=1us|total=269.39ms) [2025-08-14 09:06:26.764658] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] timer task use too much time(end=1755162386764658, start=1755162386495266, delta=269392) [2025-08-14 09:06:26.764670] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:26.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.762116] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=2294] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:26.765137] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=3015] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:26.765149] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:26.765158] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:26.765166] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:26.765175] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:26.765183] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:26.765409] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:26.765448] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:26.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:26.765468] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891806f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:26.751, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:26.765480] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:26.750, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:26.766441] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=146792199][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:06:26.766467] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=26] get number(ret=-4038, service_type_=0, range=1, base_id=1755162386765414000, start_id=0, end_id=140295968341544) [2025-08-14 09:06:26.766765] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [807][T1002_TimerWK3_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=9] ====== check clog disk timer task ====== [2025-08-14 09:06:26.767083] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:26.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.767116] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:26.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.770673] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=30] PNIO [ratelimit] time: 1755162386770671, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007598, add_bytes: 0 [2025-08-14 09:06:26.774060] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:26.774126] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=62] temporary file module use no blocks [2025-08-14 09:06:26.774131] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:26.774139] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=8] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:26.774156] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623288213763, flush_io_finished_ret_:0, flush_io_finished_round_:26405, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:26.775509] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73539b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7396b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:26.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:26.775881] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:26.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.782412] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:26.782536] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=11] gc stale ls task succ [2025-08-14 09:06:26.783391] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:26.805653] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:26.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:26.806051] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:26.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.815713] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d22b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eaf90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:26.808, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:26.825723] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:26.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:26.825765] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:26.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:26.828609] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=23] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:26.835493] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:26.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.835599] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:26.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.845045] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:06:26.845077] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:26.845838] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:26.831, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:26.845868] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:26.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:26.855900] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ac10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:26.849, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:26.859159] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:26.859214] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:26.859274] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:26.859324] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:26.859365] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:26.859379] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:26.859400] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:26.865971] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:26.852, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:26.873477] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc4658, timer:0x7f9970f40f80, task:0x564d6d07a468, task_type:"N9oceanbase8observer23ObSchemaReleaseTimeTaskE", scheduled_time:1755161716573087, pushed_time:0, delay:0}, running_task_token={this:0x7f9970fc5fb8, timer:0x7f9970f40f80, task:0x564d727182f0, task_type:"N9oceanbase5share16ObAshRefreshTaskE", scheduled_time:1755161611717613, pushed_time:1755161614757742, delay:120000000}, total_delay=670300388, threashold=1000000, tenant_id=500) [2025-08-14 09:06:26.873520] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc5618, timer:0x564d6efbde60, task:0x564d6efbde20, task_type:"N9oceanbase12blocksstable14ObBlockManager13MarkBlockTaskE", scheduled_time:1755162196749035, pushed_time:0, delay:30000000}, running_task_token={this:0x7f9970fc5538, timer:0x564d6efbde60, task:0x564d6efbde38, task_type:"N9oceanbase12blocksstable14ObBlockManager19InspectBadBlockTaskE", scheduled_time:1755162049755883, pushed_time:1755162166774569, delay:1000000}, total_delay=190124485, threashold=1000000, tenant_id=500) [2025-08-14 09:06:26.876036] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:26.861, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:26.906159] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919859b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:26.896, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:26.935555] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:26.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.935646] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:26.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:26.945084] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:26.996012] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:26.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.018370] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:27.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.018408] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:27.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.027337] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:27.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.035570] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:26.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.035700] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:26.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.045143] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:27.098866] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:27.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.135632] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:27.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.135763] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:27.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.145220] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:27.219506] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:27.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.235694] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:27.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.235804] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:27.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.245253] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:27.247524] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=73][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:27.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.253746] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:27.253775] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:27.253795] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:27.253811] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:27.253827] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:27.253842] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:27.253868] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:27.253883] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:27.253897] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:27.253913] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:27.253928] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:27.253948] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:27.253963] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:27.253984] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:27.253995] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:27.254010] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:27.254032] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:27.254044] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:27.254062] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:27.254079] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:27.254276] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:27.254295] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:27.257586] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:27.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.269695] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:27.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.269731] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:27.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.278709] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:27.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.308888] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:27.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.335721] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:27.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.335867] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:27.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.345289] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:27.367568] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=384033] gc stale ls task succ [2025-08-14 09:06:27.435781] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:27.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.435929] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:27.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.442722] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=1635183][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 6.698101 loop:26707, slowcnt: 66 [2025-08-14 09:06:27.445310] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:27.499092] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:27.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.510932] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:27.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.510975] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:27.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.530018] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:27.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.535846] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:27.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.535990] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:27.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.545371] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:27.591373] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:27.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.635885] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:27.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.636011] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:27.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.645405] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:27.711961] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:27.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.716608] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:26.726337] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=22] gc stale ls task succ [2025-08-14 09:06:26.776474] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [412][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=968455] [sql nio session](*s={this:0x7f98f733a030, session_id:3221590255, trace_id:YB42AC4D0007-00063C49651A3232-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:257}, err:0, last_decode_time:1755161952736711, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1716, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:06:25.782983] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [411][sql_nio4][T0][Y0-0000000000000000-0-0] [lt=15] [sql nio session](*s={this:0x7f98f72c3030, session_id:3221590336, trace_id:YB42AC4D0007-00063C49634A0941-0-0, sql_handling_stage:3, sql_initiative_shutdown:false, reader:{fd:255}, err:5, last_decode_time:1755162354747599, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1422, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:06:25.782983] EDIAG [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [411][sql_nio4][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1935642, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1935630, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=7], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a5bbf2 0x94da542 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:27.721597] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:27.726283] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:06:27.726326] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=23] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:06:27.726422] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=989456] gc stale ls task succ [2025-08-14 09:06:27.728154] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:27.730600] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:27.724, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:27.733643] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:763) [830][T1002_GCCollect][T1002][YB42AC4D0007-00063C496069AFFA-0-0] [lt=999175] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1002"}, process_time=450046224) [2025-08-14 09:06:27.733676] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1018) [830][T1002_GCCollect][T1002][YB42AC4D0007-00063C496069AFFA-0-0] [lt=32][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1002"}, aret=-4012, ret=-4012) [2025-08-14 09:06:26.766485] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:06:27.736146] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:27.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.736384] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:27.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.740620] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:27.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:25.753885] EDIAG [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=953][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1985604, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1985594, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x989c577 0x989bc96 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:25.753885] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=953] [thread pool stat](pool_stat={pool:0x564d6e8922c0, last_check_time:1755162385753884, last_idle_time:102477764956, last_thread_count:12}, *pool={name:TimerWK, this:0x564d6e8922c0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:10, threads_idle_time:102477764956, tenant_id:500}, qsize=10, current_idle_time=102477764956) [2025-08-14 09:06:27.742534] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=1985604] [thread pool stat](pool_stat={pool:0x7f99667cbcf0, last_check_time:1755162387742533, last_idle_time:34996878383, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x7f99667cbcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:34996878383, tenant_id:500}, qsize=-1, current_idle_time=34996878383) [2025-08-14 09:06:27.742553] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=18] [thread pool stat](pool_stat={pool:0x7f99667ccaf0, last_check_time:1755162387742553, last_idle_time:26771363534, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x7f99667ccaf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:26771363534, tenant_id:500}, qsize=-1, current_idle_time=26771363534) [2025-08-14 09:06:27.742572] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=17] [thread pool stat](pool_stat={pool:0x7f9969de56b0, last_check_time:1755162387742571, last_idle_time:102938681542, last_thread_count:17}, *pool={name:TimerWK, this:0x7f9969de56b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:16, threads_idle_time:102938681542, tenant_id:1}, qsize=10, current_idle_time=102938681542) [2025-08-14 09:06:27.742590] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=17] [thread pool stat](pool_stat={pool:0x7f99356b0e70, last_check_time:1755162387742589, last_idle_time:102366343177, last_thread_count:15}, *pool={name:TimerWK, this:0x7f99356b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:12, threads_idle_time:102366343177, tenant_id:1001}, qsize=12, current_idle_time=102366343177) [2025-08-14 09:06:27.742600] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] [thread pool stat](pool_stat={pool:0x7f99230b0e70, last_check_time:1755162387742600, last_idle_time:102534969216, last_thread_count:17}, *pool={name:TimerWK, this:0x7f99230b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:16, threads_idle_time:102534969216, tenant_id:1002}, qsize=11, current_idle_time=102534969216) [2025-08-14 09:06:27.745444] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:27.745526] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:27.750412] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:27.750683] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:27.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.751008] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:27.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:27.751037] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:27.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:27.751893] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=168] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:27.751929] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:27.752709] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:27.753185] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [807][T1002_TimerWK3_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=14] get_disk_usage(ret=0, capacity(MB):=24883, used(MB):=8642) [2025-08-14 09:06:27.753542] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=979108] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:06:27.753595] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=36940] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:27.753634] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=23693] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:27.753737] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=1962898] log compressor cycles once. (ret=0, cost_time=45504, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25862041600, fast_delete_log_mode=false) [2025-08-14 09:06:27.754262] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:27.754394] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=7] finish to resize log external storage handler(current_ts=1755162387744642, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-08-14 09:06:27.754420] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=22] snapshot_traffic_info once [2025-08-14 09:06:27.754476] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:27.754506] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:27.754541] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:27.754556] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:27.754571] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:27.754585] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:27.754603] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:27.754621] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:27.754639] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:27.754652] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:27.754663] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:27.754677] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:27.754687] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:27.754706] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:27.754720] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:27.754733] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:27.754746] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:27.754760] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:27.754774] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:27.754789] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:26.767776] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=146777648] [COMPACTION DAG STATUS] (start_time=1755162234737808, end_time=1755162386767774, dag_status={}) [2025-08-14 09:06:27.754862] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=985590] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:27.754951] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:27.754976] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:27.755076] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:27.755658] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=248] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:27.756983] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:27.757439] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=428] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:27.755502] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] get_fetch_log_dst failed or invalid(ret=0, palf_id=1001, self="172.77.0.7:2882", fetch_log_dst="172.77.0.7:2882") [2025-08-14 09:06:27.758592] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1808) [830][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1001, sql=SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1002) [2025-08-14 09:06:27.754892] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=30] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:27.758616] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1079) [830][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1001) [2025-08-14 09:06:27.758613] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=3100] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:06:27.759079] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=9] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:27.760941] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] [COMPACTION DAG STATUS] (start_time=1755162377755572, end_time=1755162387760940, dag_status={}) [2025-08-14 09:06:27.760965] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=23] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:27.760974] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:27.760984] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:27.760997] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:27.761008] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:27.761014] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:27.761025] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:27.761037] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:27.761046] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:27.761061] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=12] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:06:27.761067] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:06:27.761085] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:27.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:27.758632] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1748) [830][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1753069619, tenant_id=1001) [2025-08-14 09:06:27.761458] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:27.762448] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:27.762476] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=21] temporary file module use no blocks [2025-08-14 09:06:27.762486] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:27.762501] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:27.762602] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=100] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623289202149, flush_io_finished_ret_:0, flush_io_finished_round_:26444, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:27.762983] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:27.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.763016] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:27.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.765437] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:27.767917] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a4578, timer:0x7f9966a63f10, task:0x7f9922e05028, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr17RefreshConfigTaskE", scheduled_time:1755162386750680, pushed_time:0, delay:10000000}, running_task_token={this:0x7f991b7a4498, timer:0x7f9966a63f10, task:0x7f9922e05010, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr11TableGCTaskE", scheduled_time:1755162378768413, pushed_time:1755162386746888, delay:20000}, total_delay=1017236, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:27.767950] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a5df8, timer:0x7f99200ff180, task:0x7f99200ff1c0, task_type:"N9oceanbase8observer18ObTableLoadService15ObHeartBeatTaskE", scheduled_time:1755162386750869, pushed_time:0, delay:10000000}, running_task_token={this:0x7f991b7a6178, timer:0x7f99200ff180, task:0x7f99200ff220, task_type:"N9oceanbase8observer18ObTableLoadService21ObClientTaskPurgeTaskE", scheduled_time:1755162377739421, pushed_time:1755162385762403, delay:1000000}, total_delay=1017081, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:27.767963] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6098, timer:0x7f99200ff180, task:0x7f99200ff208, task_type:"N9oceanbase8observer18ObTableLoadService25ObClientTaskAutoAbortTaskE", scheduled_time:1755162386758966, pushed_time:0, delay:1000000}, running_task_token={this:0x7f991b7a6178, timer:0x7f99200ff180, task:0x7f99200ff220, task_type:"N9oceanbase8observer18ObTableLoadService21ObClientTaskPurgeTaskE", scheduled_time:1755162377739421, pushed_time:1755162385762403, delay:1000000}, total_delay=1008997, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:27.767982] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a4658, timer:0x7f9966a63f10, task:0x7f9922e05038, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr12TabletGCTaskE", scheduled_time:1755162386766872, pushed_time:0, delay:20000}, running_task_token={this:0x7f991b7a4498, timer:0x7f9966a63f10, task:0x7f9922e05010, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr11TableGCTaskE", scheduled_time:1755162378768413, pushed_time:1755162386746888, delay:20000}, total_delay=1001110, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:27.769859] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a5298, timer:0x7f9922ffd5d0, task:0x7f9922ffd618, task_type:"N9oceanbase7storage10checkpoint19ObCheckPointService24ObCheckClogDiskUsageTaskE", scheduled_time:1755162378750746, pushed_time:1755162378759456, delay:2000000}, delay_in_thread_pool=7987426, threashold=500000) [2025-08-14 09:06:27.769922] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][YB42AC4D0007-00063C495F8A197A-0-0] [lt=34][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a7ad8, timer:0x7f9935f59f10, task:0x7f9935f6c710, task_type:"N9oceanbase5table22OBTTLTimerPeriodicTaskE", scheduled_time:1755162378769704, pushed_time:1755162378770082, delay:10000000}, delay_in_thread_pool=8999819, threashold=500000) [2025-08-14 09:06:27.769936] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][YB42AC4D0007-00063C495F8A197B-0-0] [lt=11][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a79f8, timer:0x7f99535c95e0, task:0x7f9935f6c340, task_type:"N9oceanbase5table22OBTTLTimerPeriodicTaskE", scheduled_time:1755162378769747, pushed_time:1755162378770086, delay:10000000}, delay_in_thread_pool=8999849, threashold=500000) [2025-08-14 09:06:27.758621] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3721] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:27.770147] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11507] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:27.770157] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:27.770165] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:27.770174] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:27.770182] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:27.770190] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:27.770201] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:06:27.770209] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:06:27.770221] INFO [COMMON] loop_running_dag_net_list (ob_tenant_dag_scheduler.cpp:3845) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] loop running dag net list(ret=0, running_dag_net_list_size=0, slow_dag_net_cnt=0, finished_dag_net_list_size=0) [2025-08-14 09:06:27.770245] INFO [COMMON] loop_finished_dag_net_list (ob_tenant_dag_scheduler.cpp:3868) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] loop finsihed dag net list(ret=0, finished_dag_net_list_size=0) [2025-08-14 09:06:27.770277] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=25] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:06:25.798630] INFO [ARCHIVE] do_thread_task_ (ob_archive_timer.cpp:125) [734][T1001_ArcTimer][T1001][YB42AC4D0007-00063C495CC99C16-0-0] [lt=1900723] archive round not in doing status, just skip(key={incarnation:-1, dest_id:-1, round:-1}, state={status:"INVALID"}) [2025-08-14 09:06:27.771147] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:27.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:27.771182] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:27.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:27.771197] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d36b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ebb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:27.767, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:27.771214] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:27.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:27.774586] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:27.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.778271] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=25] PNIO [ratelimit] time: 1755162387778268, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007597, add_bytes: 0 [2025-08-14 09:06:27.781212] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:27.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:27.781257] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72739b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a5e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:27.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:27.781274] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72779b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eab10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:27.777, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:27.781288] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728bc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:27.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:27.782408] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:27.782600] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=17] gc stale ls task succ [2025-08-14 09:06:27.783397] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:27.791244] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919902b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:27.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:27.791278] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:27.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:27.791290] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:27.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:27.801278] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7277730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:27.794, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:27.801312] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:27.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:27.804764] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:27.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.821387] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:27.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:27.829252] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=18] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:27.831447] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:27.820, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:27.836208] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:27.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.836451] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:27.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.841510] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:27.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:27.845559] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:27.936250] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:27.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.936514] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:27.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:27.945630] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:27.954862] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:27.954919] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:27.954964] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:27.955005] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:27.955041] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:27.955053] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:27.955065] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:27.962213] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196aa90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:27.952, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:27.962266] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=51][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:27.953, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:27.982313] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891992fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196de90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:27.970, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:27.992136] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:27.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.014278] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:28.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.014319] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:28.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.026052] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:28.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.036306] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:27.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.036575] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:27.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.045689] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:28.094735] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:28.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.136392] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:28.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.136638] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:28.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.145712] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:06:28.145757] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:28.215432] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:28.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.236453] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:28.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.236702] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:28.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.243672] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:28.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.245775] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:28.253724] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:28.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.255165] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:28.255194] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:28.255215] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:28.255227] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:28.255256] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:28.255269] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:28.255285] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:28.255306] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:28.255331] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:28.255342] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:28.255353] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:28.255369] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:28.255390] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:28.255404] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:28.255417] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:28.255429] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:28.255441] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:28.255453] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:28.255467] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:28.255481] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:28.255646] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:28.255660] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:28.265730] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:28.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.265766] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:28.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.277495] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=51][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:28.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.307692] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:28.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.336513] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:28.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.336769] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:28.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.345832] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:28.367590] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:06:28.436532] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:28.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.436833] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:28.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.443251] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 7.698629 loop:26707, slowcnt: 76 [2025-08-14 09:06:28.445857] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:28.495146] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:28.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.517161] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:28.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.517202] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:28.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.529026] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:28.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.536595] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:28.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.536901] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:28.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.545917] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:28.597519] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:28.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.606374] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EEC9-0-0] [lt=25] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:06:28.636641] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:28.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.636966] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:28.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.645967] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:28.717496] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:28.719604] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:28.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.721745] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:28.726445] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=13] gc stale ls task succ [2025-08-14 09:04:02.729328] EDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1008) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1025725, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1025718, ObMalloc#start=0, ObMalloc#end=3, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x27a5a6c0 0x94da840 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:28.729295] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:28.717821] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [830][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=2769][errcode=-4012] query failed(ret=-4012, conn=0x7f98f673fad0, start=1755161935772262, sql=SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1002) [2025-08-14 09:06:28.729416] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [830][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=11574][errcode=-4012] read failed(ret=-4012) [2025-08-14 09:06:28.729292] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1001, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:8571658, max_lsn:7323687369, max_scn:{val:1755161749741193002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:7278610047}, reuse_lsn:{lsn:7323686131}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:7281744329}}, last_submit_lsn:{lsn:7323686131}, last_submit_end_lsn:{lsn:7323687369}, last_submit_log_id:8571658, last_submit_log_pid:49, max_flushed_lsn:{lsn:7323686009}, max_flushed_end_lsn:{lsn:7323686131}, max_flushed_log_pid:49, committed_end_lsn:{lsn:7323686009}, last_slide_log_id:8571656, last_slide_scn:{val:1755161747742105001, v:0}, last_slide_lsn:{lsn:7323685887}, last_slide_end_lsn:{lsn:7323686009}, last_slide_log_pid:49, last_slide_log_accum_checksum:2505458373, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f990112a530}, [StateMgr]={this:0x7f990112e9b0, self:"172.77.0.7:2882", palf_id:1001, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"PENDING", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:49}, leader:"0.0.0.0:0", leader_epoch:-1, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:7323687369}, scan_disk_log_finished:true, last_check_start_id:8571657, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161767744317, palf_role_change_cb:0x7f9901135470, allow_vote:true}, [ConfigMgr]={palf_id:1001, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:49, prev:{version:1, config:{config_version:{proposal_id:48, config_seq:47}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:49, config_seq:48}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_mode_pid:49}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_end_lsn:{lsn:7323685240}, prev_mode_pid:49}, reconfig_barrier:{prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_end_lsn:{lsn:7323685240}, prev_mode_pid:49}, persistent_config_version:{proposal_id:49, config_seq:48}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755161739755312, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755161739755312, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f990112bc70}, [ModeMgr]={palf_id:1001, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, accepted_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1001, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1753069671378640001, v:0}, base_lsn_for_block_gc:{lsn:7247314944}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:49}, log_config_meta:{version:3, proposal_id:49, prev:{version:1, config:{config_version:{proposal_id:48, config_seq:47}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:49, config_seq:48}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_mode_pid:49}, log_snapshot_meta:{version:2, base_lsn:{lsn:7247314944}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1753069671378640000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}}, log_meta_storage:{log_tail:{lsn:765952}, readable_log_tail:{lsn:765952}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7381524480}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/meta", dir_fd:232, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66338816, block_header_serialize_buf_:0x7f9901130bb8, flashback_version:0}, log_storage:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, palf_epoch:2, last_purge_throttling_ts:1755161769777679, this:0x7f990112f0f0}, [Reconfirm]={palf_id:1001, self:"172.77.0.7:2882", state:"WAITING_LOG_FLUSHED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:1755161769777679, this:0x7f990112eb30}) [2025-08-14 09:06:28.729514] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=215] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [2025-08-14 09:06:28.729531] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14] current log_task status(palf_id=1001, self="172.77.0.7:2882", log_id=8571657, log_task={header:{begin_lsn:{lsn:7323686009}, end_lsn:{lsn:7323686131}, log_id:8571657, min_scn:{val:1755161749741193000, v:0}, max_scn:{val:1755161749741193000, v:0}, data_len:66, proposal_id:49, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:7323686009}, data_checksum:2222604892, accum_checksum:1531236791, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1755161749745909, freeze_ts:1755161749745914, submit_ts:1755161749745940, flushed_ts:1755161785758706, gen_to_freeze cost time:5, gen_to_submit cost time:31, submit_to_flush cost time:36012766}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [2025-08-14 09:06:28.728230] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:213) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-08-14 09:06:28.736639] INFO [COMMON] handle (memory_dump.cpp:544) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8394] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-08-14 09:06:28.737091] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:28.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.726943] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [807][T1002_TimerWK3_TabletShell][T1002][Y0-0000000000000000-0-0] [lt=10] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-08-14 09:06:28.737321] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [807][T1002_TimerWK3_TabletShell][T1002][Y0-0000000000000000-0-0] [lt=10355] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=2, times=1264) [2025-08-14 09:06:28.737339] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a5458, timer:0x7f9922ffd748, task:0x7f9922ffd770, task_type:"N9oceanbase7storage10checkpoint16ObEmptyShellTaskE", scheduled_time:1755162379762654, pushed_time:1755162379762955, delay:5000000}, delay_in_thread_pool=8006992, threashold=500000) [2025-08-14 09:06:28.737375] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][YB42AC4D0007-00063C495F8A197C-0-0] [lt=17][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a6f78, timer:0x7f99329e1b10, task:0x7f98e4e091c0, task_type:"N9oceanbase5table21ObClearTTLHistoryTaskE", scheduled_time:1755162380725332, pushed_time:1755162380726731, delay:30000000}, delay_in_thread_pool=8010633, threashold=500000) [2025-08-14 09:06:28.737693] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=14] update_check_range(min_check_version=2540, max_check_version=2540, global_version=2648) [2025-08-14 09:06:28.723914] INFO [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:171) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=966129] handle gts request statistics(total_rt=967715, total_cnt=4, avg_rt=1.935430000000000000e+05, avg_cnt=4.000000000000000222e-01) [2025-08-14 09:06:28.738726] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=14784][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162386766429], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:06:28.739867] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc190, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc470, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:10.00s, expected_run_time:09:06:28.720, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.738755] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162386766429], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:06:28.740120] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1344] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162386766429]}) [2025-08-14 09:06:28.740452] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:28.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.740755] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=24] LogLoopThread round_cost_time(us)(round_cost_time=9988968) [2025-08-14 09:06:28.740791] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] get_fetch_log_dst failed or invalid(ret=0, palf_id=1, self="172.77.0.7:2882", fetch_log_dst="172.77.0.7:2882") [2025-08-14 09:06:28.743454] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:28.746027] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:28.746649] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8030, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8310, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:10.00s, expected_run_time:09:06:28.730, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.746685] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:28.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.747092] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:28.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:28.747131] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:28.737, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:28.747151] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e71b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:28.739, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:28.747170] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918119b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:28.737, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:28.740810] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:06:28.749318] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:28.749855] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:980) [830][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4012] failed to read(ret=-4012, ret="OB_TIMEOUT", exec_tenant_id=1001, sql=SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1002) [2025-08-14 09:06:28.740150] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=26] gts statistics(tenant_id=1002, gts_rpc_cnt=0, get_gts_cache_cnt=66, get_gts_with_stc_cnt=2, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:06:28.750068] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9900] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162386766429]}) [2025-08-14 09:06:28.751660] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=18] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:28.752044] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:28.753672] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:06:28.753727] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=18] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:28.753769] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=13] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:28.755093] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:28.755275] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:28.755653] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:28.756626] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:28.756654] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:28.756677] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:28.756701] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:28.756714] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:28.756711] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:28.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.756727] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:28.756744] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:28.756755] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:28.756776] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:28.756791] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:28.756807] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:28.756824] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:28.756839] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:28.756855] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:28.756873] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:28.756885] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:28.756902] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:28.756918] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:28.756936] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:28.756953] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:28.756983] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:28.757118] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:28.757145] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:28.757270] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72774b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:28.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:28.757293] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fbe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:28.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:28.758391] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:28.759473] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=31] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:28.760270] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:28.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.760303] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:28.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.762853] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=30] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:28.762900] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:28.765390] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:00:53.754508] EDIAG [SHARE] end_run (ob_tenant_base.cpp:389) [15373][T1002_L0_G1][T0][Y0-0000000000000000-0-0] [lt=29941886][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3985692, owner: ObLog, click_count: 3, time dist:[FORMAT_END=3977533, ALLOC_END=8158, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x20333146 0x20332f46 0x2668bbfa 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:28.766291] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [854][T1002_ObXAHbWor][T1002][Y0-0000000000000000-0-0] [lt=5044] XA scheduler heartbeat task statistics(avg_time=2) [2025-08-14 09:06:28.768866] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=1025725] kill and revert session(conn.sessid_=3221590977, proxy_sessid=0, ret=0) [2025-08-14 09:06:28.768942] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=35] [sql nio session](*s={this:0x7f98f733f030, session_id:3221589417, trace_id:YB42AC4D0007-00063C496359B268-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:252}, err:0, last_decode_time:1755161813752329, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1778, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:06:28.768971] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=29] [sql nio session](*s={this:0x7f98f733e030, session_id:3221589980, trace_id:YB42AC4D0007-00063C496359B261-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:251}, err:0, last_decode_time:1755161748778070, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1681, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:06:28.768985] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=14] [sql nio session](*s={this:0x7f98f7332030, session_id:3221590132, trace_id:YB42AC4D0007-00063C496359B265-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:266}, err:0, last_decode_time:1755161768783802, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:2125, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:06:28.769002] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=17] [sql nio session](*s={this:0x7f98f733b030, session_id:3221590977, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:10, sql_initiative_shutdown:false, reader:{fd:211}, err:5, last_decode_time:1755162219768794, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:06:28.769450] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:28.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:28.769489] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:28.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:28.769517] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:28.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:28.774328] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:28.774356] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=24] temporary file module use no blocks [2025-08-14 09:06:28.774366] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:28.774379] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:28.774393] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623290214023, flush_io_finished_ret_:0, flush_io_finished_round_:26407, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:28.779585] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919aef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:28.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:28.779614] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7270a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a4190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:28.762, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:28.779627] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:28.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:28.779640] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:28.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:28.779651] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:28.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:28.780662] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:28.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.782623] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=18] gc stale ls task succ [2025-08-14 09:06:28.783399] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:28.785905] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=23] PNIO [ratelimit] time: 1755162388785903, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007635, add_bytes: 0 [2025-08-14 09:06:28.787263] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:28.789648] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f734de30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891919810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:28.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:28.789673] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:28.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:28.789685] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:28.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:28.789697] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d59b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:28.773, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:28.789708] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efa10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:28.777, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:28.795828] INFO handle (memory_dump.cpp:671) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=15] statistics: tenant_cnt: 6, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 168 270 0 1 2 110 1 0 1 3 7 0 0 1 4 1 1 0 1 5 85 6 0 1 6 13 0 0 1 7 22 2 0 1 8 53 1 0 1 12 17 3 0 1 13 10 1 0 1 14 1 1 0 1 15 1 1 0 1 16 8 7 0 1 18 4 3 0 1 20 2 1 0 1 22 1 1 0 1 24 1 0 0 1 25 6 0 0 500 0 562 276 0 500 7 190 5 0 500 8 41 1 0 500 9 2 1 0 500 10 2 1 0 500 16 58 4 0 500 17 60 7 0 500 22 38 64 0 500 23 27 10 0 508 0 2 7 0 508 8 5 1 0 509 0 2 7 0 509 8 5 1 0 1001 0 147 252 0 1001 2 26 1 0 1001 3 5 0 0 1001 4 1 1 0 1001 5 85 6 0 1001 6 11 0 0 1001 7 30 2 0 1001 8 36 1 0 1001 12 17 3 0 1001 13 3 1 0 1001 14 1 1 0 1001 16 7 3 0 1001 18 4 3 0 1001 20 4 1 0 1001 25 8 0 0 1002 0 263 279 0 1002 2 198 1 0 1002 3 4 0 0 1002 4 1 1 0 1002 5 101 6 0 1002 6 14 0 0 1002 7 6 2 0 1002 8 73 1 0 1002 12 18 3 0 1002 13 7 1 0 1002 14 1 2 0 1002 16 10 7 0 1002 18 9 3 0 1002 20 2 1 0 1002 21 1 0 0 1002 24 1 0 0 1002 25 7 0 0 cost_time: 58166 [2025-08-14 09:06:28.795874] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=25] [MEMORY] tenant: 1, limit: 2147483648 hold: 1013174272 cache_hold: 56623104 cache_used: 56623104 cache_item_count: 27 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 444993536 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 230686720 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 1163264 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 483328 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 51617792 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 1458176 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 14204928 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 111149056 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35651584 limit= 429496720 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 8003584 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= PS_CACHE_CTX_ID hold_bytes= 65536 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 14827520 limit= 9223372036854775807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 3481600 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3964928 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 212992 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 884736 limit= 9223372036854775807 [2025-08-14 09:06:28.799714] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eba10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:28.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:28.799762] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:28.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:28.800805] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:28.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.806782] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] [sql nio session](*s={this:0x7f98f72d0030, session_id:3221589415, trace_id:YB42AC4D0007-00063C4965FA43DC-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:245}, err:0, last_decode_time:1755161751714366, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1662, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:06:28.806813] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=31] [sql nio session](*s={this:0x7f98f72d5030, session_id:3221590200, trace_id:YB42AC4D0007-00063C4965FA43E0-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:248}, err:0, last_decode_time:1755161776777167, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1593, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:06:28.809761] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:28.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:28.809794] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:28.790, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:28.810860] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40b7f0, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40bad0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:10.00s, expected_run_time:09:06:28.802, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.829841] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=19] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:28.830308] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:28.827, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:28.837141] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:28.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.840380] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739af30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196bc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:28.837, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:28.840517] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:28.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.840860] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8357] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:06:28.846087] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:28.850433] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:28.841, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:28.850464] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:28.849, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:28.850479] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:28.849, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:28.858395] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fc8b0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcb90, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:1, time_interval:3.00s, expected_run_time:09:06:28.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.900719] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:28.892, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:28.920860] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:28.917, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:28.930923] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:28.922, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:28.937184] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:28.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.940584] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:28.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:28.946146] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:28.963128] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:28.963196] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:28.963270] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:28.963318] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:28.963372] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:28.963390] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:28.963404] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:28.970771] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=23][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:06:28.971052] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=22][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:06:28.991483] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:28.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.011644] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:29.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.011682] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:29.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.022058] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:29.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.037245] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:28.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.040648] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:28.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.046196] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:29.092070] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:29.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.137310] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:29.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.140711] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:29.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.146288] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:29.212882] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:29.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.237372] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:29.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.240775] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:29.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.242870] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:29.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.246319] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:29.252933] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:29.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.257322] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:29.257352] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:29.257368] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:29.257390] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:29.257406] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:29.257424] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:29.257441] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:29.257455] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:29.257475] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:29.257490] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:29.257506] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:29.257522] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:29.257536] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:29.257545] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:29.257562] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:29.257584] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:29.257600] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:29.257617] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:29.257634] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:29.257651] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:29.257823] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:29.257840] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:29.263140] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:29.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.263171] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:29.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.273667] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:29.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.303851] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:29.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.337435] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:29.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.340838] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:29.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.346383] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:29.367639] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=23] gc stale ls task succ [2025-08-14 09:06:29.437494] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:29.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.440901] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:29.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.443796] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 8.699175 loop:26707, slowcnt: 86 [2025-08-14 09:06:29.446437] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:29.494393] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:29.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.517007] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:29.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.517047] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:29.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.525123] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:29.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.537556] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:29.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.540967] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:29.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.546502] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:29.597540] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:29.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.637617] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:29.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.641030] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:29.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.646561] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:29.717499] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:29.718340] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:29.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.721793] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:29.726487] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=21] gc stale ls task succ [2025-08-14 09:06:29.728334] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:29.732481] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:29.728, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:29.732518] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f734c030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891807390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:29.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:29.737689] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:29.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.741097] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:29.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.741284] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=65] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:06:29.742537] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aae10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:29.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:29.742572] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:29.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:29.742588] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e67b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:29.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:29.742603] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:29.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:29.744109] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:29.746621] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:29.748514] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:29.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.748685] INFO [SERVER] check_tenant_memory (ob_plugin_vector_index_scheduler.cpp:493) [807][T1002_TimerWK3_TTLTabletMgr][T1002][YB42AC4D0007-00063C495F8A197E-0-0] [lt=17] get vector mem limit size(ret=0, ret="OB_SUCCESS", tenant_id=1002, current_memory_config=0) [2025-08-14 09:06:29.749334] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:29.751687] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=16] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:29.751897] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:29.752074] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:29.753023] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:29.753046] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919bfe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aa810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:29.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:29.753079] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:29.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:29.753098] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:29.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:29.753124] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:29.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:29.753135] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:29.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:29.754031] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:29.755182] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=21] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:29.755255] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:06:29.755318] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=11] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:29.755613] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:29.757060] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:29.758371] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:29.758455] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:29.758483] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:29.758502] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:29.758515] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:29.758539] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:29.758552] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:29.758568] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:29.758580] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:29.758588] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:29.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.758610] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:29.758623] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:29.758632] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:29.758647] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:29.758669] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:29.758679] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:29.758706] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:29.758722] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:29.758745] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:29.758760] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:29.758772] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:29.758784] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:29.758945] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:29.758961] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:29.759930] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=21] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:29.760818] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:29.762744] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:29.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.762779] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:29.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.763076] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:29.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:29.765471] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:29.766726] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=31] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:29.766761] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26] temporary file module use no blocks [2025-08-14 09:06:29.766775] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:29.766795] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=19] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:29.766860] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=64] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623291206032, flush_io_finished_ret_:0, flush_io_finished_round_:26446, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:29.776873] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=59][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:29.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.782685] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=24] gc stale ls task succ [2025-08-14 09:06:29.783202] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:29.783205] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72707b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a4010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:29.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:29.783418] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:29.793261] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:29.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:29.793502] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=28] PNIO [ratelimit] time: 1755162389793499, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007596, add_bytes: 0 [2025-08-14 09:06:29.803311] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739acb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196bb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:29.798, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:29.807046] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:29.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.813373] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=49][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:29.806, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:29.813408] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:29.809, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:29.823432] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:29.815, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:29.830654] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=19] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:29.833145] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=917976] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:06:29.837750] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:29.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.839063] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=917534] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:06:29.841162] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:29.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.841331] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=25] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:06:29.843508] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:29.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:29.843540] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:29.832, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:29.845681] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=907033] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:06:29.846665] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:06:29.846695] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:29.853569] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:29.841, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:29.863587] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:29.859, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:29.937767] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:29.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.941228] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:29.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:29.946727] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:29.964100] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:29.964170] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=31] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:29.964231] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:29.964290] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:29.964344] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:29.964371] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:29.964388] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:29.990240] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:29.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.014036] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:30.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.014085] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:30.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.028176] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:30.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.037783] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:29.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.041293] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:29.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.046787] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:30.075457] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc4578, timer:0x7f9970f40f80, task:0x564d6d07a498, task_type:"N9oceanbase8observer26ObRemoteMasterRsUpdateTaskE", scheduled_time:1755162028742204, pushed_time:0, delay:600000000}, running_task_token={this:0x7f9970fc5fb8, timer:0x7f9970f40f80, task:0x564d727182f0, task_type:"N9oceanbase5share16ObAshRefreshTaskE", scheduled_time:1755161611717613, pushed_time:1755161614757742, delay:120000000}, total_delay=361333251, threashold=1000000, tenant_id=500) [2025-08-14 09:06:30.094486] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:30.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.137845] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:30.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.141315] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:30.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.146846] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:30.215158] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:30.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.237871] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:30.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.241379] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:30.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.241523] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:30.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.246904] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:30.251585] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:30.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.259134] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:30.259166] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:30.259192] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:30.259206] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:30.259229] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:30.259262] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:30.259278] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:30.259294] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:30.259325] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:30.259337] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:30.259349] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:30.259362] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:30.259387] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:30.259400] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:30.259414] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:30.259429] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:30.259441] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:30.259455] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:30.259477] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:30.259490] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:30.259656] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:30.259676] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:30.265482] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:30.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.265523] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:30.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.279962] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:30.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.300053] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:30.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.337944] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:30.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.341442] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:30.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.346933] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:30.367701] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=24] gc stale ls task succ [2025-08-14 09:06:30.438006] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:30.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.441506] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:30.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.445011] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 9.700390 loop:26707, slowcnt: 96 [2025-08-14 09:06:30.447012] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:30.492893] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:30.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.516866] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:30.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.516904] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:30.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.521221] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:30.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.538030] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:30.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.541568] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:30.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.547068] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:30.597296] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:30.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.638093] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:30.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.641594] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:30.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.647135] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:30.717551] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:30.717871] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:30.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.719782] WDIAG print_alloc_failed_msg (alloc_failed_reason.cpp:146) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14128][errcode=-4013] [OOPS]: alloc failed reason is that server memory has reached the upper limit(server_hold: 4294098944, server_limit: 4294967296, alloc_size: 2097152). detailed info: tenant_id=500, ctx_id=17, ctx_name=PKT_NIO, ctx_hold=37543936, ctx_limit=9223372036854775807, tenant_hold=692756480, tenant_limit=9223372036854775807, backtrace=0x26a1806d 0xf98a0db 0xf989b06 0x9327c08 0x9315ac5 0x27a8a305 0x984531c 0x9844f4c 0x1c047a9f 0x1c0340a3 0x1c02b713 0x97a97cd 0x97a1ba3 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:30.719823] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=CLIENT_REQ_CHUNK, ctx_id=17, prio=0, nbyte=16288) [2025-08-14 09:06:30.722531] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:30.726640] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=21] gc stale ls task succ [2025-08-14 09:06:30.728973] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:30.729020] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73516b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196f510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:30.716, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:30.738156] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:30.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.739082] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:30.720, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:30.739115] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:30.727, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:30.739139] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:30.727, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:30.739149] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72851b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:30.721, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:30.741833] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=24] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:06:30.741837] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:30.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.120861] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:233) [709][T1001_RFLWorker][T1001][YB42AC4D0007-00063C495BE99C16-0-0] [lt=1960946] ObRemoteFetchWorker is running(thread_index=0) [2025-08-14 09:06:30.743360] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:30.744267] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:30.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.748147] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:30.749139] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:30.737, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:30.749172] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:30.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:30.749355] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:30.751014] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:288) [732][T1001_ArcFetche][T1001][YB42AC4D0007-00063C495CA99C16-0-0] [lt=6681] ObArchiveFetcher is running(thread_index=0) [2025-08-14 09:06:30.751253] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=17] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 444993536 used= 389771296 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 70 washed_blocks= 287 washed_size= 70221824 [MEMORY] request_cached_chunk_cnt= 6 [MEMORY] hold= 389771296 used= 379518723 count= 32899 avg_used= 11535 mod=SUMMARY [2025-08-14 09:06:30.751294] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=20] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 230686720 used= 228884480 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 block_cnt= 110 chunk_cnt= 110 mod=Memstore [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:06:30.751310] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1 ctx_id= EXECUTE_CTX_ID hold= 1163264 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 15 washed_size= 13516800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:30.751324] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:06:30.751340] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 51617792 used= 47452416 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 84 washed_blocks= 334 washed_size= 126640128 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 47452416 used= 43070492 count= 3992 avg_used= 10789 mod=SUMMARY [2025-08-14 09:06:30.751354] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= WORK_AREA hold= 1458176 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 13 washed_blocks= 66 washed_size= 25804800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:30.751369] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 14204928 used= 10970224 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 277 washed_size= 31932416 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10970224 used= 6078924 count= 18573 avg_used= 327 mod=SUMMARY [2025-08-14 09:06:30.751384] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 111149056 used= 108896256 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 block_cnt= 211 chunk_cnt= 53 mod=CoStack [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 mod=SUMMARY [2025-08-14 09:06:30.751404] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 56623104 used= 56623104 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 56623104 used= 0 count= 27 avg_used= 0 mod=SUMMARY [2025-08-14 09:06:30.751418] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 34365376 limit= 429496720 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 34365376 used= 31086608 count= 475 avg_used= 65445 mod=SUMMARY [2025-08-14 09:06:30.751444] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 8003584 used= 4947392 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 10 washed_blocks= 23 washed_size= 12967936 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4947392 used= 4793344 count= 604 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:30.751459] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:30.751474] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 65536 used= 31024 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 2031616 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 31024 used= 27058 count= 20 avg_used= 1352 mod=SUMMARY [2025-08-14 09:06:30.751492] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1 ctx_id= RPC_CTX_ID hold= 14827520 used= 6135808 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 6135808 used= 4051328 count= 253 avg_used= 16013 mod=SUMMARY [2025-08-14 09:06:30.751518] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=17] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 3481600 used= 2391232 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 10 washed_size= 4907008 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2391232 used= 2367040 count= 36 avg_used= 65751 mod=SUMMARY [2025-08-14 09:06:30.751536] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 3964928 used= 3080192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 3 washed_size= 229376 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3080192 used= 2983936 count= 376 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:30.751552] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= SCHEMA_SERVICE hold= 212992 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1884160 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:30.751568] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:30.751581] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= MERGE_NORMAL_CTX_ID hold= 884736 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 6 washed_blocks= 9 washed_size= 11698176 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:30.751669] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=19] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:30.752066] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:30.752092] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:30.753066] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:30.754345] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:30.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.754578] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:30.755904] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=36] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:30.755961] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:06:30.756010] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:30.756047] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:30.757164] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:30.728133] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [561][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=18] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=16, total_destructed_detector=16, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f994b3ee2b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-08-14 09:06:30.758587] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:30.759176] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:30.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:30.759209] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891807210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:30.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:30.759223] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:30.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:30.759253] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:30.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:30.716503] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=16] new mysql sessid created(conn.sessid_=3221590982, support_ssl=false) [2025-08-14 09:06:30.759846] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:30.759871] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:30.759889] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:30.759901] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:30.759929] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:30.759942] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:30.759958] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:30.759986] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:30.760002] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:30.760013] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:30.760025] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:30.760041] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:30.760053] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:30.760066] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:30.760080] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:30.760094] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:30.760105] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:30.760125] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:30.760139] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:30.760152] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:30.760283] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=20] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:30.760326] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:30.760342] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:30.719836] WDIAG stat_cleanup (debug.c:35) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=0] PNIO [delay_warn] cost too much time: 145961029us, procedure: mod_alloc [2025-08-14 09:06:30.763028] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:30.765435] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:30.766965] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=37] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:30.766992] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=22] temporary file module use no blocks [2025-08-14 09:06:30.767002] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:30.767017] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:30.767031] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623292206665, flush_io_finished_ret_:0, flush_io_finished_round_:26447, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:30.769152] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:30.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.769188] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:30.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.769261] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d67b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:30.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:30.773128] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:30.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.778266] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:30.778300] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=30] temporary file module use no blocks [2025-08-14 09:06:30.778307] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:30.778322] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:30.778340] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=18] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623292214255, flush_io_finished_ret_:0, flush_io_finished_round_:26409, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:30.782730] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=22] gc stale ls task succ [2025-08-14 09:06:30.783243] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:30.783441] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:30.799467] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891807090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:30.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:30.799503] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739aa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:30.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:30.801124] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=21] PNIO [ratelimit] time: 1755162390801116, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007617, add_bytes: 0 [2025-08-14 09:06:30.803298] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:30.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.809527] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:30.791, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:30.809560] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ecf10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:30.791, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:30.809575] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:30.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:30.809592] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:30.799, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:30.819589] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:30.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:30.819621] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7351e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196f990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:30.803, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:30.819636] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a22b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:30.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:30.819660] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:30.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:30.838218] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:30.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.839713] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ebe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:30.826, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:30.839742] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7350530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ea90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:30.824, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:30.841898] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=33] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:06:30.841900] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:30.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.848214] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:30.880036] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fbbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:30.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:30.880069] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72860b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:30.873, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:30.938247] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:30.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.941963] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:30.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:30.948275] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:30.965105] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:30.965183] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:30.965252] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:30.965308] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:30.965361] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:30.965384] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:30.965398] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:30.995758] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:30.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.010629] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:31.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.010663] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:31.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.024485] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:31.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.038299] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:30.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.041988] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:30.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.048310] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:31.091099] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:31.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.138381] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:31.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.142048] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:31.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.148327] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:06:31.148358] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:31.211731] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:31.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.238398] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:31.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.242150] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:31.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.247193] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:31.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.248389] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:31.257257] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:31.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.260521] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:31.260549] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:31.260576] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:31.260592] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:31.260608] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:31.260619] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:31.260637] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:31.260647] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:31.260665] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:31.260680] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:31.260692] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:31.260707] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:31.260719] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:31.260741] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:31.260754] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:31.260773] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:31.260785] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:31.260798] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:31.260812] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:31.260825] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:31.260998] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:31.261012] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:31.262032] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:31.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.262061] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:31.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.275940] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:31.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.306108] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:31.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.332701] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7faa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:31.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:31.332734] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72842b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:31.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:31.338473] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:31.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.342213] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:31.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.348448] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:31.367724] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=21] gc stale ls task succ [2025-08-14 09:06:31.438531] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:31.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.442274] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:31.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.445608] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 10.700987 loop:26707, slowcnt: 106 [2025-08-14 09:06:31.448511] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:31.498794] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:31.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.513369] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:31.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.513420] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:31.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.527284] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:31.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.538560] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:31.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.542319] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:31.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.548571] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:31.593813] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:31.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.608094] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EECB-0-0] [lt=25] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:06:31.624384] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eef90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:31.614, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:31.624425] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eaf30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:31.614, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:31.638583] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:31.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.642389] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:31.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.648605] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:31.714900] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:31.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.717537] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:31.721554] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:31.726339] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant: 500, limit: 9223372036854775807 hold: 686645248 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 331468800 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 43253760 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 85983232 limit= 9223372036854775807 [MEMORY] ctx_id= LIBEASY hold_bytes= 3145728 limit= 9223372036854775807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 4194304 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 3104768 limit= 9223372036854775807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 37584896 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 78925824 limit= 9223372036854775807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 98983936 limit= 9223372036854775807 [2025-08-14 09:06:31.726393] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=27] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 331468800 used= 301898224 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 501 washed_blocks= 2232 washed_size= 926711808 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 301898224 used= 294456821 count= 12729 avg_used= 23132 mod=SUMMARY [2025-08-14 09:06:31.726421] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 500 ctx_id= GLIBC hold= 43253760 used= 21979888 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 180 washed_blocks= 800 washed_size= 355205120 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 21979888 used= 10373317 count= 131705 avg_used= 78 mod=SUMMARY [2025-08-14 09:06:31.726438] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= CO_STACK hold= 85983232 used= 83091456 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 83091456 used= 82936896 count= 161 avg_used= 515136 block_cnt= 161 chunk_cnt= 41 mod=CoStack [MEMORY] hold= 83091456 used= 82936896 count= 161 avg_used= 515136 mod=SUMMARY [2025-08-14 09:06:31.726454] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= LIBEASY hold= 3145728 used= 3070816 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1048576 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3070816 used= 2859784 count= 223 avg_used= 12824 mod=SUMMARY [2025-08-14 09:06:31.726475] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=15] [MEMORY] tenant_id= 500 ctx_id= LOGGER_CTX_ID hold= 6291456 used= 4161536 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4161536 used= 4159488 count= 2 avg_used= 2079744 mod=SUMMARY [2025-08-14 09:06:31.726499] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= RPC_CTX_ID hold= 3104768 used= 1507328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 58 washed_blocks= 71 washed_size= 118530048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1507328 used= 995584 count= 62 avg_used= 16057 mod=SUMMARY [2025-08-14 09:06:31.726513] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 500 ctx_id= PKT_NIO hold= 37584896 used= 36013568 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 50 washed_blocks= 71 washed_size= 99016704 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 36013568 used= 35017984 count= 125 avg_used= 280143 mod=SUMMARY [2025-08-14 09:06:31.726545] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=15] [MEMORY] tenant_id= 500 ctx_id= SCHEMA_SERVICE hold= 78925824 used= 68945536 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 22 washed_size= 5767168 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 68945536 used= 67682783 count= 4861 avg_used= 13923 mod=SUMMARY [2025-08-14 09:06:31.726559] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= UNEXPECTED_IN_500 hold= 98983936 used= 97219920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 97219920 used= 86819136 count= 3016 avg_used= 28786 mod=SUMMARY [2025-08-14 09:06:31.726574] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 508, limit: 1073741824 hold: 15613952 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 5128192 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10485760 limit= 9223372036854775807 [2025-08-14 09:06:31.726585] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 508 ctx_id= DEFAULT_CTX_ID hold= 5128192 used= 3765328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3765328 used= 3723912 count= 84 avg_used= 44332 mod=SUMMARY [2025-08-14 09:06:31.726609] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 508 ctx_id= CO_STACK hold= 10485760 used= 10321920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10321920 used= 10302720 count= 20 avg_used= 515136 mod=SUMMARY [2025-08-14 09:06:29.715514] EDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4038] LOGGER COST TOO MUCH TIME, cost: 2008586, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2008571, ObMalloc#start=9, ObMalloc#end=2, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1ba227a3 0x1b89078a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:30.723376] EDIAG [SERVER] execute_adapter_maintenance (ob_plugin_vector_index_scheduler.cpp:477) [807][T1002_TimerWK3_TTLTabletMgr][T1002][YB42AC4D0007-00063C495F8A197E-0-0] [lt=34][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1002811, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1002803, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1fb9131c 0x1fb8f4ec 0x1fb98f17 0x1fb8617a 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:31.726657] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant: 509, limit: 1073741824 hold: 15613952 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 5128192 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10485760 limit= 9223372036854775807 [2025-08-14 09:06:31.726670] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 509 ctx_id= DEFAULT_CTX_ID hold= 5128192 used= 3765328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3765328 used= 3723912 count= 84 avg_used= 44332 mod=SUMMARY [2025-08-14 09:06:31.726705] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 509 ctx_id= CO_STACK hold= 10485760 used= 10321920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10321920 used= 10302720 count= 20 avg_used= 515136 mod=SUMMARY [2025-08-14 09:06:31.726754] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 1001, limit: 1073741824 hold: 608747520 cache_hold: 44040192 cache_used: 44040192 cache_item_count: 21 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 302309376 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 54525952 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 3080192 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 483328 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 45514752 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 5521408 limit= 157286400 [MEMORY] ctx_id= GLIBC hold_bytes= 15499264 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 73424896 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35651584 limit= 214748360 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 3252224 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 14680064 limit= 1073741800 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 5570560 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3866624 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 1179648 limit= 9223372036854775807 [2025-08-14 09:06:31.726785] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= DEFAULT_CTX_ID hold= 302309376 used= 269196048 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 81 washed_blocks= 406 washed_size= 113410048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 269196048 used= 263064350 count= 3790 avg_used= 69410 mod=SUMMARY [2025-08-14 09:06:31.726797] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= MEMSTORE_CTX_ID hold= 54525952 used= 54099968 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 54099968 used= 54075216 count= 26 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:06:31.726814] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=14] [MEMORY] tenant_id= 1001 ctx_id= EXECUTE_CTX_ID hold= 3080192 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 8 washed_size= 7405568 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:31.726824] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:06:31.726843] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=16] [MEMORY] tenant_id= 1001 ctx_id= PLAN_CACHE_CTX_ID hold= 45514752 used= 36552320 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 77 washed_blocks= 201 washed_size= 132743168 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 36552320 used= 33009690 count= 3323 avg_used= 9933 mod=SUMMARY [2025-08-14 09:06:31.726853] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= WORK_AREA hold= 5521408 used= 0 limit= 157286400 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 9 washed_blocks= 40 washed_size= 17547264 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:31.726863] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= GLIBC hold= 15499264 used= 10889728 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 25 washed_blocks= 164 washed_size= 47415296 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10889728 used= 6020141 count= 18313 avg_used= 328 mod=SUMMARY [2025-08-14 09:06:31.726875] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= CO_STACK hold= 73424896 used= 71221248 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 32 washed_blocks= 34 washed_size= 2072576 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 71221248 used= 71088768 count= 138 avg_used= 515136 block_cnt= 138 chunk_cnt= 36 mod=CoStack [MEMORY] hold= 71221248 used= 71088768 count= 138 avg_used= 515136 mod=SUMMARY [2025-08-14 09:06:31.726890] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= KVSTORE_CACHE_ID hold= 44040192 used= 44040192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 44040192 used= 0 count= 21 avg_used= 0 mod=SUMMARY [2025-08-14 09:06:31.726900] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 33906624 limit= 214748360 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33906624 used= 30629648 count= 468 avg_used= 65447 mod=SUMMARY [2025-08-14 09:06:31.726918] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=15] [MEMORY] tenant_id= 1001 ctx_id= TX_CALLBACK_CTX_ID hold= 3252224 used= 2342592 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 3 washed_blocks= 8 washed_size= 3039232 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2342592 used= 2269696 count= 286 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:31.726929] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:31.726941] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= RPC_CTX_ID hold= 14680064 used= 4202496 limit= 1073741800 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4202496 used= 2779776 count= 171 avg_used= 16256 mod=SUMMARY [2025-08-14 09:06:31.726953] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= TX_DATA_TABLE hold= 5570560 used= 2342784 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 8 washed_size= 2818048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2342784 used= 2319424 count= 30 avg_used= 77314 mod=SUMMARY [2025-08-14 09:06:31.726974] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= MDS_DATA_ID hold= 3866624 used= 2785280 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 5 washed_size= 4521984 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2785280 used= 2698240 count= 340 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:31.726976] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=24] gc stale ls task succ [2025-08-14 09:06:31.726991] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= MERGE_NORMAL_CTX_ID hold= 1179648 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 8 washed_blocks= 24 washed_size= 15597568 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:31.727010] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant: 1002, limit: 8589934592 hold: 1945841664 cache_hold: 48234496 cache_used: 48234496 cache_item_count: 23 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 1009610752 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 415236096 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 1114112 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 950272 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 211812352 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 1441792 limit= 429496725 [MEMORY] ctx_id= GLIBC hold_bytes= 12582912 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 149913600 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 37748736 limit= 1717986900 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 7766016 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 5783552 limit= 8589934500 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 5718016 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3047424 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 1032192 limit= 9223372036854775807 [2025-08-14 09:06:30.831269] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=22] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:29.715514] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1002}) [2025-08-14 09:06:31.727174] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=2008586] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1002}) [2025-08-14 09:06:30.723376] INFO [SERVER] execute_adapter_maintenance (ob_plugin_vector_index_scheduler.cpp:477) [807][T1002_TimerWK3_TTLTabletMgr][T1002][YB42AC4D0007-00063C495F8A197E-0-0] [lt=34] finish generate tenant tablet tasks(ret=0, ret="OB_SUCCESS", tenant_id=1002) [2025-08-14 09:06:31.727738] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][YB42AC4D0007-00063C495F8A197E-0-0] [lt=1002811][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a4b98, timer:0x7f99535c9ed0, task:0x7f99004c9710, task_type:"N9oceanbase5share32ObPluginVectorIndexLoadSchedulerE", scheduled_time:1755162384751200, pushed_time:1755162384751925, delay:10000000}, delay_in_thread_pool=3985468, threashold=500000) [2025-08-14 09:06:31.729667] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:31.729821] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [365][TbltRefreshSer][T1002][YB42AC4D0007-00063C4960B99C42-0-0] [lt=22555][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:06:31.729839] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [365][TbltRefreshSer][T1002][YB42AC4D0007-00063C4960B99C42-0-0] [lt=17] get number(ret=-4038, service_type_=0, range=1, base_id=1755162391729262000, start_id=0, end_id=94890572229104) [2025-08-14 09:06:31.727044] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= DEFAULT_CTX_ID hold= 1009610752 used= 919021120 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 14 [MEMORY] hold= 185860096 used= 185534336 count= 114 avg_used= 1627494 block_cnt= 114 chunk_cnt= 104 mod=MysqlRequesReco [MEMORY] hold= 167792640 used= 167772192 count= 1 avg_used= 167772192 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 83927040 used= 83886080 count= 2 avg_used= 41943040 block_cnt= 2 chunk_cnt= 2 mod=LogGroupBuffer [MEMORY] hold= 67190784 used= 67109344 count= 4 avg_used= 16777336 block_cnt= 4 chunk_cnt= 4 mod=MrgMemPoolChk [MEMORY] hold= 919021120 used= 901764910 count= 9514 avg_used= 94782 mod=SUMMARY [2025-08-14 09:06:31.730977] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=3908] [MEMORY] tenant_id= 1002 ctx_id= MEMSTORE_CTX_ID hold= 415236096 used= 411992064 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 411992064 used= 411803568 count= 198 avg_used= 2079816 block_cnt= 198 chunk_cnt= 198 mod=Memstore [MEMORY] hold= 411992064 used= 411803568 count= 198 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:06:31.730991] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1002 ctx_id= EXECUTE_CTX_ID hold= 1114112 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 7 washed_size= 7274496 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:31.731000] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1002 ctx_id= TRANS_CTX_MGR_ID hold= 950272 used= 933888 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1146880 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 933888 used= 921984 count= 2 avg_used= 460992 mod=SUMMARY [2025-08-14 09:06:31.731011] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1002 ctx_id= PLAN_CACHE_CTX_ID hold= 211812352 used= 201995008 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 176930880 used= 160228962 count= 13450 avg_used= 11912 block_cnt= 13450 chunk_cnt= 101 mod=SqlPhyPlan [MEMORY] hold= 201995008 used= 183048143 count= 17906 avg_used= 10222 mod=SUMMARY [2025-08-14 09:06:31.731020] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1002 ctx_id= WORK_AREA hold= 1441792 used= 0 limit= 429496725 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 14 washed_blocks= 54 washed_size= 27918336 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:31.731030] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= GLIBC hold= 12582912 used= 11425280 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 11425280 used= 6322556 count= 19216 avg_used= 329 mod=SUMMARY [2025-08-14 09:06:31.731041] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= CO_STACK hold= 149913600 used= 147087360 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 68 washed_blocks= 70 washed_size= 3178496 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 147087360 used= 146813760 count= 285 avg_used= 515136 block_cnt= 285 chunk_cnt= 73 mod=CoStack [MEMORY] hold= 147087360 used= 146813760 count= 285 avg_used= 515136 mod=SUMMARY [2025-08-14 09:06:31.731054] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1002 ctx_id= KVSTORE_CACHE_ID hold= 48234496 used= 48234496 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 48234496 used= 0 count= 23 avg_used= 0 mod=SUMMARY [2025-08-14 09:06:31.731063] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1002 ctx_id= META_OBJ_CTX_ID hold= 37748736 used= 35741632 limit= 1717986900 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 35741632 used= 32457488 count= 496 avg_used= 65438 mod=SUMMARY [2025-08-14 09:06:31.731071] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1002 ctx_id= TX_CALLBACK_CTX_ID hold= 7766016 used= 6934400 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 17 washed_size= 6914048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 6934400 used= 6729728 count= 848 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:31.731080] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1002 ctx_id= LOB_CTX_ID hold= 147456 used= 16384 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 16384 used= 15872 count= 2 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:31.731091] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1002 ctx_id= RPC_CTX_ID hold= 5783552 used= 5029888 limit= 8589934500 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 10 washed_blocks= 10 washed_size= 15187968 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 5029888 used= 3315456 count= 210 avg_used= 15787 mod=SUMMARY [2025-08-14 09:06:31.731108] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1002 ctx_id= TX_DATA_TABLE hold= 5718016 used= 4521856 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 9 washed_blocks= 16 washed_size= 13156352 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4521856 used= 4480128 count= 40 avg_used= 112003 mod=SUMMARY [2025-08-14 09:06:31.731119] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1002 ctx_id= MDS_DATA_ID hold= 3047424 used= 1998848 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 4 washed_size= 1146880 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1998848 used= 1936384 count= 244 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:31.731135] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=14] [MEMORY] tenant_id= 1002 ctx_id= MDS_CTX_ID hold= 147456 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:31.731147] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1002 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:31.731156] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= MERGE_NORMAL_CTX_ID hold= 1032192 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 12 washed_size= 13647872 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:31.733161] INFO [CLOG] update_upstream (ob_remote_location_adaptor.cpp:76) [829][T1002_LogRessvr][T1002][YB42AC4D0007-00063C495D199C16-0-0] [lt=935852] primary tenant, just skip [2025-08-14 09:06:31.738646] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:31.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.740867] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:31.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.748668] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:31.750952] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:31.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.753005] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:31.754582] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:31.755993] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=23] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:31.756084] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=17] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:06:31.757148] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:31.761185] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:31.761213] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:31.761229] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:31.761253] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:31.761262] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:31.761272] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:31.761285] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:31.761295] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:31.761317] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:31.761326] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:31.761335] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:31.761347] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:31.761355] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:31.761365] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:31.761375] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:31.761392] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:31.761400] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:31.761409] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:31.761420] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:31.761431] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:31.761600] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:31.761611] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:31.747176] WDIAG [RPC] post (ob_poc_rpc_proxy.h:266) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=42029][errcode=-4013] pn_send fail(sys_err=12, addr="172.77.0.7:2882", pcode=1815) [2025-08-14 09:06:31.741556] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a41f8, timer:0x7f9922df93b0, task:0x7f9922ffab60, task_type:"N9oceanbase3sql27ObSqlPlanMonitorRecycleTaskE", scheduled_time:1755162384755452, pushed_time:1755162384761995, delay:1000000}, delay_in_thread_pool=6965780, threashold=500000) [2025-08-14 09:06:31.742337] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:06:31.742452] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:31.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.743415] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:31.771670] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28211][errcode=0] epoll handler too much time :28264 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:31.749402] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:31.771706] WDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22269][errcode=0] epoll handler too much time :22314 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:31.751901] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=14] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:31.771731] WDIAG ev_invoke_pending (ev.c:2136) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=19799][errcode=0] epoll handler too much time :19834 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:31.752057] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:31.752162] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:31.771787] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19595][errcode=0] epoll handler too much time :19633 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:31.755733] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:31.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:31.771824] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16063][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:31.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:31.771854] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919827b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:31.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:31.771868] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:31.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:31.771881] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72742b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:31.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:31.771903] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919840b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:31.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:31.771916] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180ef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:31.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:31.771929] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180ecb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:31.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:31.771942] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:31.767, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:31.771955] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:31.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:31.756103] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=13] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:31.758247] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:31.772009] WDIAG ev_invoke_pending (ev.c:2136) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13741][errcode=0] epoll handler too much time :13786 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:31.729853] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:283) [365][TbltRefreshSer][T1002][YB42AC4D0007-00063C4960B99C42-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="172.77.0.7:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:06:31.772073] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:223) [365][TbltRefreshSer][T1002][YB42AC4D0007-00063C4960B99C42-0-0] [lt=42153][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.77.0.7:2882", server="172.77.0.7:2882", tmp_ret=-4038) [2025-08-14 09:06:31.772108] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [365][TbltRefreshSer][T1002][YB42AC4D0007-00063C4960B99C42-0-0] [lt=28] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162386766429]}) [2025-08-14 09:06:31.765408] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:31.765477] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:31.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.767373] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:31.772244] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=4849] temporary file module use no blocks [2025-08-14 09:06:31.772243] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=6742][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:31.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.772257] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:31.760562] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=14] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:31.772270] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:31.772284] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623293206826, flush_io_finished_ret_:0, flush_io_finished_round_:26448, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:31.778445] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:31.778478] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=29] temporary file module use no blocks [2025-08-14 09:06:31.778486] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:31.778504] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=17] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:31.778515] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623293218137, flush_io_finished_ret_:0, flush_io_finished_round_:26410, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:31.778589] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:31.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.781932] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919925b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196d890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:31.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:31.781970] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:31.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:31.781987] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:31.777, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:31.783297] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:31.783731] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:31.791976] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728ca30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7291590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:31.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:31.802037] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739ee10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:31.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:31.808739] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=23] PNIO [ratelimit] time: 1755162391808737, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007621, add_bytes: 0 [2025-08-14 09:06:31.808775] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:31.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.812096] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ebd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:31.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:31.812158] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=59][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739ec90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:31.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:31.822157] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72874b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:31.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:31.832219] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:31.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:31.832266] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739dc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:31.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:31.838684] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:31.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.842414] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=29252] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:06:31.848694] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:31.860333] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fc8b0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcb90, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:1, time_interval:3.00s, expected_run_time:09:06:31.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.902611] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a59b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:31.895, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:31.927282] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=894468] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:31.938745] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:31.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.942539] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=29147][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:31.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:31.948755] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:31.966181] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:31.966252] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:31.966314] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:31.966365] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:31.966417] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:31.966435] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:31.966448] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:31.992522] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:31.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.016941] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:32.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.016987] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:32.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.030020] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:32.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.038768] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:31.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.042580] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:31.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.048783] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:32.097330] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:32.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.138797] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:32.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.142640] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:32.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.148896] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:32.218010] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:32.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.238860] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:32.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.242704] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:32.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.244073] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:32.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.248921] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:32.254141] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:32.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.261757] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:32.261790] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:32.261815] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:32.261832] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:32.261860] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:32.261878] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:32.261895] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:32.261913] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:32.261934] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:32.261950] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:32.261972] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:32.261992] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:32.262010] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:32.262029] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:32.262055] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:32.262071] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:32.262084] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:32.262101] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:32.262120] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:32.262146] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:32.262338] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:32.262358] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:32.268306] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:32.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.268342] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:32.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.271264] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:32.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.301444] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:32.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.338922] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:32.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.342766] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:32.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.348999] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:32.367730] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:06:32.438982] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:32.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.442827] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:32.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.446256] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 11.701636 loop:26707, slowcnt: 116 [2025-08-14 09:06:32.449049] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:32.495910] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:32.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.519722] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:32.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.519781] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=56][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:32.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.522675] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:32.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.539051] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:32.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.542859] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:32.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.549109] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:32.590073] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:32.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.639116] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:32.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.642920] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:32.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.649177] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:32.710774] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:32.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.718670] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:31.771765] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19677][errcode=0] epoll handler too much time :19716 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:31.739161] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=42711] sm conn init succ(conn.sessid_=3221590982, sess.client_addr_="172.77.0.1:48114") [2025-08-14 09:06:32.722031] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1090) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=981370] accept one succ(*s={this:0x7f98f7338030, session_id:3221590982, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:276}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:06:32.722104] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1008) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x7f98f7338030, session_id:3221590982, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:276}, err:5, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:06:32.722126] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=17] kill and revert session(conn.sessid_=3221590982, proxy_sessid=0, ret=0) [2025-08-14 09:06:32.722146] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:983) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=19] sock ref clean, do destroy(*s={this:0x7f98f7338030, session_id:3221590982, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:276}, err:5, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:06:32.722167] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:221) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221590982) [2025-08-14 09:06:32.726626] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:32.727048] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=14] gc stale ls task succ [2025-08-14 09:06:32.728349] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:31.782767] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=22] gc stale ls task succ [2025-08-14 09:06:32.734072] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][YB42AC4D0007-00063C495F8A197F-0-0] [lt=29450][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a5c38, timer:0x7f9935f93f20, task:0x7f99200fe2d0, task_type:"N9oceanbase8datadict17ObDataDictServiceE", scheduled_time:1755162385716508, pushed_time:1755162385716581, delay:1000000}, delay_in_thread_pool=6054486, threashold=500000) [2025-08-14 09:06:32.726490] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=31] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25862037504, fast_delete_log_mode=false) [376]OB_MALLOC COST TOO MUCH TIME, cost_time=1003791, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=41133, WASH_BLOCK_END=962652, ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=2], seq:[0, 1, 2, 3, 4] [2025-08-14 09:06:30.760475] EDIAG [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=1649][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1971635, owner: ObLog, click_count: 5, time dist:[FORMAT_END=967796, ObMalloc#start=2, ObMalloc#end=1003817, ALLOC_END=1, APPEND_END=19], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1fce4e91 0x1fce455d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:30.760475] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=1649] blacklist_loop exec finished(cost_time=18, is_enabled=true, send_cnt=0) [384]OB_MALLOC COST TOO MUCH TIME, cost_time=977614, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=15917, WASH_BLOCK_END=961697], seq:[0, 1, 2] [2025-08-14 09:06:32.734221] INFO [STORAGE.TRANS] get_gts (ob_gts_source.cpp:257) [365][TbltRefreshSer][T1002][YB42AC4D0007-00063C4960B99C42-0-0] [lt=13] push gts task success(*task={:""}) [395]OB_MALLOC COST TOO MUCH TIME, cost_time=973048, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=10967, WASH_BLOCK_END=962081], seq:[0, 1, 2] [2025-08-14 09:06:32.736574] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:32.726, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:32.736654] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=77][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739eb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:32.721, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:32.739182] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:32.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.726829] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] [COMPACTION DAG STATUS] (start_time=1755162381748287, end_time=1755162392726828, dag_status={}) [2025-08-14 09:06:32.742892] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=32] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:06:32.742990] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:32.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.746475] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:32.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.746664] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:32.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:32.749210] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:32.749550] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:32.742506] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:31.756058] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:32.749909] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=992347][errcode=0] epoll handler too much time :993858 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:31.761066] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:32.749941] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=987380][errcode=0] epoll handler too much time :988884 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:32.751088] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=947126] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:32.751729] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=7] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:32.751884] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4389] timer task too much delay in priority queue(token={this:0x7f991b7a6178, timer:0x7f99200ff180, task:0x7f99200ff220, task_type:"N9oceanbase8observer18ObTableLoadService21ObClientTaskPurgeTaskE", scheduled_time:1755162377739421, pushed_time:1755162385762403, delay:1000000}, delay_in_priority_queue=8022982, threashold=1000000) [2025-08-14 09:06:32.751901] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a6178, timer:0x7f99200ff180, task:0x7f99200ff220, task_type:"N9oceanbase8observer18ObTableLoadService21ObClientTaskPurgeTaskE", scheduled_time:1755162377739421, pushed_time:1755162385762403, delay:1000000}, delay_in_thread_pool=6971710, threashold=500000) [2025-08-14 09:06:32.751929] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][YB42AC4D0007-00063C495F8A1981-0-0] [lt=12][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a7918, timer:0x7f99329e19d0, task:0x7f98f55f1250, task_type:"N9oceanbase5share24ObVectorIndexHistoryTaskE", scheduled_time:1755162385763094, pushed_time:1755162385772479, delay:10000000}, delay_in_thread_pool=6979444, threashold=500000) [2025-08-14 09:06:32.751959] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [807][T1002_TimerWK3_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=10] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):258, sequence:1}) [2025-08-14 09:06:32.751991] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:97) [807][T1002_TimerWK3_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=22] BlockGCTimerTask success(ret=0, cost_time_us=39, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.77.0.7:2882", log_dir:"/root/ob/observer/store/clog/tenant_1002", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):258, sequence:1}, log_alloc_mgr_:{flying_log_task:1, flying_meta_task:2}}) [2025-08-14 09:06:32.752011] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a4818, timer:0x7f99335d9ee0, task:0x7f991c5e9398, task_type:"N9oceanbase4palf16BlockGCTimerTaskE", scheduled_time:1755162385779190, pushed_time:1755162385782588, delay:30000}, delay_in_thread_pool=6969359, threashold=500000) [2025-08-14 09:06:32.752472] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:32.752505] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:32.720354] WDIAG [STORAGE.TRANS] post_cluster_heartbeat_rpc_ (ob_tenant_weak_read_service.cpp:909) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20346][errcode=-4013] post cluster heartbeat fail(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", cluster_service_master="172.77.0.7:2882", tenant_id_=1001, req={req_server:"172.77.0.7:2882", version:{val:1755161719775179002, v:0}, valid_part_count:1, total_part_count:1, generate_timestamp:1755162244758751}, cluster_service_tablet_id={id:226}) [2025-08-14 09:06:32.756285] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=35870][errcode=-4013] tenant weak read service cluster heartbeat RPC fail(ret=-4013, rcode={code:-4013, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:06:32.756551] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:32.757399] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:32.757577] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=12] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:32.757625] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=88] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:06:32.758117] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:32.758178] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:32.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.760317] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:32.762583] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:32.762617] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:32.762637] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:32.762649] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:32.762662] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:32.762675] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:32.762701] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:32.762714] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:32.762731] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:32.762742] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:32.762755] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:32.762771] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:32.762782] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:32.762795] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:32.762807] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:32.762819] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:32.762831] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:32.762844] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:32.762858] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:32.762872] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:32.763035] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:32.763050] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:32.764708] WDIAG [STORAGE.TRANS] run1 (ob_ts_mgr.cpp:488) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=0] gts is not used for a long time(tenant_id=1001) [2025-08-14 09:06:32.765144] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] timer task too much delay in priority queue(token={this:0x7f991b7a4498, timer:0x7f9966a63f10, task:0x7f9922e05010, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr11TableGCTaskE", scheduled_time:1755162378768413, pushed_time:1755162386746888, delay:20000}, delay_in_priority_queue=7978475, threashold=1000000) [2025-08-14 09:06:32.765164] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a4498, timer:0x7f9966a63f10, task:0x7f9922e05010, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr11TableGCTaskE", scheduled_time:1755162378768413, pushed_time:1755162386746888, delay:20000}, delay_in_thread_pool=6005137, threashold=500000) [2025-08-14 09:06:32.765466] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:865) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=-4013] post cluster heartbeat rpc fail(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id_=1001, local_server_version={val:1755161719775179002, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1755162244758751) [2025-08-14 09:06:32.765498] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:875) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=-4013] tenant weak read service do cluster heartbeat fail(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1755162241717184, cluster_heartbeat_interval_=50000, cluster_service_tablet_id={id:226}, cluster_service_master="172.77.0.7:2882") [2025-08-14 09:06:32.765557] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119797572, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755162138745740) [2025-08-14 09:06:32.724123] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [830][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] long time to hold one guard(schema_mgr=0x7f98ad7818f0, tenant_id=1, version=1753073979411448, cur_timestamp=1755162392724110, ref_timestamp=1755161936718968, lbt()="0x26a1806d 0x974f3e5 0x974e159 0x974e084 0x948f9df 0x97257c0 0x97256f4 0x9725691 0x1ee94121 0x1ee92221 0x1ee928a5 0xfd80a58 0xfd91a9a 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3") [2025-08-14 09:06:32.765958] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6792] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:32.765968] WDIAG [SHARE.SCHEMA] revert (ob_schema_mgr_cache.cpp:119) [830][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=41771][errcode=0] long time to hold one guard(schema_mgr=0x7f988fae9820, tenant_id=1001, version=1753074886385104, cur_timestamp=1755162392765959, ref_timestamp=1755161936718968, lbt()="0x26a1806d 0x974f3e5 0x974e159 0x974e084 0x948fab3 0x97257c0 0x97256f4 0x9725691 0x1ee94121 0x1ee92221 0x1ee928a5 0xfd80a58 0xfd91a9a 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3") [2025-08-14 09:06:32.766608] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:32.766976] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:32.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.767010] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:32.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.768260] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:32.750, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:32.768298] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:32.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:32.768308] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d80b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:32.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:32.768318] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:32.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:32.768327] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891992330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196d710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:32.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:32.768351] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:32.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:32.772082] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=15872] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:32.772363] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=11703] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:32.772413] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=32] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:32.772447] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26] temporary file module use no blocks [2025-08-14 09:06:32.772458] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:32.772473] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:32.772489] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623294212085, flush_io_finished_ret_:0, flush_io_finished_round_:26449, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:32.778260] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:32.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.778322] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:32.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:32.778348] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:32.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:32.782321] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:32.782643] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:32.782830] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=949500] gc stale ls task succ [2025-08-14 09:06:32.788395] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:32.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:32.788416] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72751b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:32.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:32.808472] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:32.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.816353] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=25] PNIO [ratelimit] time: 1755162392816350, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007613, add_bytes: 0 [2025-08-14 09:06:32.839223] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:32.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.843051] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:32.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.849267] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:06:32.849297] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:32.928033] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=24] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:32.939275] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:32.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.943103] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:32.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:32.949335] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:32.967121] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:32.967202] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=31] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:32.967263] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:32.967311] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:32.967353] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:32.967380] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:32.967394] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:32.999631] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:32.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.018397] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:33.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.018433] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:33.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.029837] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:33.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.039337] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:32.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.043167] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:32.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.049363] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:33.098843] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:33.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.139409] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:33.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.143230] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:33.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.149408] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:33.219517] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:33.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.239471] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:33.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.241292] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:33.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.243290] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:33.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.249470] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:33.251342] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:33.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.263218] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:33.269755] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:33.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.269788] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:33.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.271468] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:33.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.301636] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:33.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.339533] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:33.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.343352] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:33.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.349529] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:33.367788] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=24] gc stale ls task succ [2025-08-14 09:06:33.439595] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:33.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.443427] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:33.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.446935] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 12.702314 loop:26707, slowcnt: 126 [2025-08-14 09:06:33.449586] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:33.492883] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:33.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.511172] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:33.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.511215] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:33.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.522973] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:33.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.539608] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:33.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.543481] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:33.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.549647] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:33.591565] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:33.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.639670] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:33.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.643544] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:33.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.649718] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:33.712208] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:33.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.717669] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=29] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:32.749913] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:244) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221590982, proxy_sessid=0, tenant_id=0, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:0], disconnect_state=0) [2025-08-14 09:06:33.721852] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:33.727096] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=14] gc stale ls task succ [374]OB_MALLOC COST TOO MUCH TIME, cost_time=949188, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=949185], seq:[0] [2025-08-14 09:06:33.739734] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:33.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.744359] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:33.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.746535] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:33.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.729611] WDIAG [SERVER] set_timeout (ob_inner_sql_connection.cpp:2049) [360][UserLocAsyncUp0][T0][YB42AC4D0007-00063C495D499F18-0-0] [lt=9][errcode=-4012] already timeout(ret=-4012, ctx={abs_timeout_us:1755162375756821}, abs_timeout_us=1755162375756821) [2025-08-14 09:06:33.750937] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [374]OB_MALLOC COST TOO MUCH TIME, cost_time=964380, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=949206], seq:[0, 1] [2025-08-14 09:06:33.752510] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:32.788426] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d31b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:32.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:33.729545] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:33.755201] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25615][errcode=0] epoll handler too much time :25665 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:33.756177] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:33.756205] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:33.757061] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:33.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.757587] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [372]OB_MALLOC COST TOO MUCH TIME, cost_time=503858, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=503857], seq:[0] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=945418, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=945415], seq:[0] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=1473839, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=503874], seq:[0, 1] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=991111, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=991104], seq:[0] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=991106, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=41, WASH_BLOCK_END=15177, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=945449], seq:[0, 1, 2, 3, 4] [2025-08-14 09:06:33.759665] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=20] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:33.759706] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=17] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:06:33.778698] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:33.778726] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16] temporary file module use no blocks [2025-08-14 09:06:33.778733] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:33.778744] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:33.778755] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623295218378, flush_io_finished_ret_:0, flush_io_finished_round_:26412, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:33.839813] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:33.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.852555] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:33.939863] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:33.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:34.039890] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:33.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:34.139951] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:34.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.856677] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a4498, timer:0x7f9966a63f10, task:0x7f9922e05010, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr11TableGCTaskE", scheduled_time:1755162392785176, pushed_time:0, delay:20000}, running_task_token={this:0x7f991b7a4578, timer:0x7f9966a63f10, task:0x7f9922e05028, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr17RefreshConfigTaskE", scheduled_time:1755162386750680, pushed_time:1755162392765831, delay:10000000}, total_delay=1071500, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:33.990905] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:33.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:34.240012] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:34.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:34.722598] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:34.447748] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 13.703127 loop:26707, slowcnt: 136 [2025-08-14 09:06:33.952613] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:34.727274] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=23] gc stale ls task succ [2025-08-14 09:06:34.717268] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:34.741846] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24539][errcode=0] epoll handler too much time :24597 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:34.722578] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=481835][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:34.240|207=482.57ms|210=0us|222=1us|total=504.65ms) [2025-08-14 09:06:33.828318] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [880][T1002_ArcSender][T1002][YB42AC4D0007-00063C495EB99C16-0-0] [lt=17] ObArchiveSender is running(thread_index=5) [395]OB_MALLOC COST TOO MUCH TIME, cost_time=994435, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=994428], seq:[0] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=992127, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=992103, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=992170, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=992141, WASH_BLOCK_END=29], seq:[0, 1, 2] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=993071, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=993058, ALLOC_CHUNK_END=5], seq:[0, 1] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=993063, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=993047, ALLOC_CHUNK_END=4], seq:[0, 1] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=993143, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=993088, WASH_BLOCK_END=55], seq:[0, 1, 2] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=993116, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=993076, WASH_BLOCK_END=40], seq:[0, 1, 2] [389]OB_MALLOC COST TOO MUCH TIME, cost_time=993129, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=993120, ALLOC_CHUNK_END=4], seq:[0, 1] [389]OB_MALLOC COST TOO MUCH TIME, cost_time=1003226, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=30, WASH_BLOCK_END=10058, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=993138], seq:[0, 1, 2, 3, 4] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=989163, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=989155, ALLOC_CHUNK_END=4], seq:[0, 1] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=989182, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=989172, WASH_BLOCK_END=10], seq:[0, 1, 2] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=989223, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=989214, ALLOC_CHUNK_END=4], seq:[0, 1] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=989240, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=989231, WASH_BLOCK_END=9], seq:[0, 1, 2] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=983921, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=983911, ALLOC_CHUNK_END=5], seq:[0, 1] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=983937, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=983928, WASH_BLOCK_END=9], seq:[0, 1, 2] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=980549, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=980540, ALLOC_CHUNK_END=4], seq:[0, 1] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=980569, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=980558, WASH_BLOCK_END=11], seq:[0, 1, 2] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=980253, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=980243, ALLOC_CHUNK_END=5], seq:[0, 1] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=980273, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=980261, WASH_BLOCK_END=11], seq:[0, 1, 2] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=980367, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=980357, ALLOC_CHUNK_END=5], seq:[0, 1] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=980400, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=980390, WASH_BLOCK_END=10], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=939868, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=939859, ALLOC_CHUNK_END=4], seq:[0, 1] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=939886, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=939878, WASH_BLOCK_END=8], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=919538, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=919528, ALLOC_CHUNK_END=4], seq:[0, 1] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=919555, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=919546, WASH_BLOCK_END=9], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=832898, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=832889, ALLOC_CHUNK_END=4], seq:[0, 1] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=832915, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=832907, WASH_BLOCK_END=8], seq:[0, 1, 2] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=812984, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=812975, ALLOC_CHUNK_END=5], seq:[0, 1] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=813001, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=812991, WASH_BLOCK_END=10], seq:[0, 1, 2] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=793924, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=793915, ALLOC_CHUNK_END=4], seq:[0, 1] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=793941, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=793932, WASH_BLOCK_END=9], seq:[0, 1, 2] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=397288, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=397278, ALLOC_CHUNK_END=5], seq:[0, 1] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=397304, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=397295, WASH_BLOCK_END=9], seq:[0, 1, 2] [2025-08-14 09:06:34.722550] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=730534][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:33.990|207=731.64ms|210=0us|222=2us|total=754.23ms) [2025-08-14 09:06:33.752588] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=34] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:34.745215] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=991125][errcode=0] epoll handler too much time :992634 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:34.751255] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:34.753131] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:34.754216] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:34.761522] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=11] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:06:33.823980] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=23] PNIO [ratelimit] time: 1755162393823978, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007628, add_bytes: 0 [2025-08-14 09:06:33.931300] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=20] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:34.732173] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:34.765887] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=33662][errcode=0] epoll handler too much time :33726 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:34.768139] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:34.070924] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=27][errcode=-4018] try free send task failed(ret=-4018) [772]OB_MALLOC COST TOO MUCH TIME, cost_time=999457, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=999444], seq:[0] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=999478, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=999474], seq:[0, 1] [2025-08-14 09:06:33.767743] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1001007, owner: ObLog, click_count: 6, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=992178, ObMalloc#start=22, ObMalloc#end=34, FREE_END=8763], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:21.734663] EDIAG [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:244) [407][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3029287, owner: ObLog, click_count: 3, time dist:[FORMAT_END=3029286, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0xfa06846 0xfa05f62 0xfa05c84 0xfa05a50 0x13553ff2 0x13552f70 0x27a475ba 0x94da9e6 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:33.756053] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1015027, owner: ObLog, click_count: 6, time dist:[FORMAT_END=10, ObMalloc#start=0, ObMalloc#end=13748, ObMalloc#start=19, ObMalloc#end=993124, FREE_END=8126], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:33.755567] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1015684, owner: ObLog, click_count: 6, time dist:[FORMAT_END=8, ObMalloc#start=1, ObMalloc#end=14191, ObMalloc#start=29, ObMalloc#end=993153, FREE_END=8302], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [401]OB_MALLOC COST TOO MUCH TIME, cost_time=1019489, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=991126], seq:[0, 1] [2025-08-14 09:06:33.752560] EDIAG [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1019518, owner: ObLog, click_count: 5, time dist:[FORMAT_END=9, ObMalloc#start=0, ObMalloc#end=1019503, ALLOC_END=3, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a1f867 0x27a135c3 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:33.766809] EDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=964412][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1005749, owner: ObLog, click_count: 8, time dist:[FORMAT_END=971487, ObMalloc#start=0, ObMalloc#end=26153, ObMalloc#start=20, ObMalloc#end=33, ObMalloc#start=125, ObMalloc#end=43, FREE_END=7887], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ed35c 0x97aead0 0x97ad969 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:33.759754] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=49][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1012868, owner: ObLog, click_count: 6, time dist:[FORMAT_END=10, ObMalloc#start=0, ObMalloc#end=1003233, ObMalloc#start=33, ObMalloc#end=58, FREE_END=9534], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:33.263263] EDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4016] LOGGER COST TOO MUCH TIME, cost: 1509364, owner: ObLog, click_count: 5, time dist:[FORMAT_END=8, ObMalloc#start=0, ObMalloc#end=1473864, ALLOC_END=35488, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x13b06ffb 0x958a39e 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:33.768879] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:33.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:34.769911] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=999514][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:33.768|207=1.00s|210=0us|222=3us|total=1.00s) [2025-08-14 09:06:34.769939] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4389] timer task use too much time(end=1755162394769936, start=1755162393768877, delta=1001059) [2025-08-14 09:06:34.769957] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] task scheduled too delay(scheduled_delay=1014467, expected_run_ts_us_=1755162393755487, ObClockGenerator::getRealClock()=1755162394769956, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:33.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:34.769986] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:33.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:34.770004] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=680688, expected_run_ts_us_=1755162394089316, ObClockGenerator::getRealClock()=1755162394770004, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:34.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:34.770021] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:34.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:34.770035] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=565737, expected_run_ts_us_=1755162394204297, ObClockGenerator::getRealClock()=1755162394770035, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:34.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:34.770047] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:34.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:34.770065] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd110, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd3f0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:10.00s, expected_run_time:09:06:34.726, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:34.744844] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=80] get_max_applied_scn(scn={val:1755161719775179002, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:45, palf_committed_end_lsn_:{lsn:8461098908}, palf_committed_end_scn_:{val:1755161719775179002, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161719775179002, v:0}}) [2025-08-14 09:06:34.770302] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1001007][errcode=0] epoll handler too much time :1002569 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:34.770336] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:34.772457] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=22050][errcode=-4389] timer task use too much time(end=1755162394744662, start=1755162394240010, delta=504652) [2025-08-14 09:06:34.772680] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1015027][errcode=0] epoll handler too much time :1016636 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:34.772701] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:34.772808] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1015684][errcode=0] epoll handler too much time :1017250 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:34.772825] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:34.773481] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=992571][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:33.779|207=994.08ms|210=0us|222=1us|total=994.10ms) [2025-08-14 09:06:34.773504] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4389] timer task use too much time(end=1755162394773504, start=1755162393779402, delta=994102) [2025-08-14 09:06:34.773517] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=981167, expected_run_ts_us_=1755162393792349, ObClockGenerator::getRealClock()=1755162394773517, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:33.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:34.773536] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:33.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.752560] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=13] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:34.773642] WDIAG ev_invoke_pending (ev.c:2136) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=1019518][errcode=0] epoll handler too much time :1021084 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:34.773658] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=9] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:34.722103] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=25] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:06:34.722490] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1055) [360][UserLocAsyncUp0][T0][YB42AC4D0007-00063C495D499F18-0-0] [lt=21305][errcode=-4012] set timeout failed(ret=-4012) [2025-08-14 09:06:33.782895] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=39] gc stale ls task succ [2025-08-14 09:06:34.367849] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=22] gc stale ls task succ [2025-08-14 09:06:34.773936] WDIAG ev_invoke_pending (ev.c:2136) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24642][errcode=0] epoll handler too much time :24689 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:34.774059] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=989304][errcode=0] epoll handler too much time :990809 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:34.774103] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1005749][errcode=-4389] timer task use too much time(end=1755162394774102, start=1755162392788425, delta=1985677) [2025-08-14 09:06:34.774174] WDIAG ev_invoke_pending (ev.c:2136) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1012868][errcode=0] epoll handler too much time :1014427 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:34.774192] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:33.263263] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:34.774957] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=1509364][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:34.774973] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:34.774986] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:34.774998] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:34.775012] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:34.775024] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:34.775039] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:34.775049] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:34.775060] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:34.775074] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:34.775085] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:34.775097] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:34.775109] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:34.775120] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:34.775132] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:34.775144] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:34.775165] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:34.775178] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:34.775339] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:34.775354] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:34.773188] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=22547][errcode=-4389] timer task use too much time(end=1755162394745132, start=1755162393990903, delta=754229) [2025-08-14 09:06:34.775448] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=2248][errcode=0] task scheduled too delay(scheduled_delay=530813, expected_run_ts_us_=1755162394244633, ObClockGenerator::getRealClock()=1755162394775447, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:34.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:34.775466] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:34.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:34.775488] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:34.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:34.723205] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=769423] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:06:34.775537] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=52249][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:34.789246] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=989329] gc stale ls task succ [2025-08-14 09:06:34.822603] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=864542][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6178, timer:0x7f99200ff180, task:0x7f99200ff220, task_type:"N9oceanbase8observer18ObTableLoadService21ObClientTaskPurgeTaskE", scheduled_time:1755162393751914, pushed_time:0, delay:1000000}, running_task_token={this:0x7f991b7a5d18, timer:0x7f99200ff180, task:0x7f99200ff1a8, task_type:"N9oceanbase8observer18ObTableLoadService17ObCheckTenantTaskE", scheduled_time:1755162377761774, pushed_time:1755162392751930, delay:1000000}, total_delay=1070687, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:34.823318] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:34.831593] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=939933] PNIO [ratelimit] time: 1755162394831591, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007613, add_bytes: 0 [2025-08-14 09:06:34.872549] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:34.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:34.923407] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:34.965690] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=832957] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:34.996707] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:34.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:35.023458] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:35.072672] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:34.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:35.123515] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:35.248032] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:35.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:35.258081] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:35.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:35.273264] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:35.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:35.275522] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:35.275545] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:35.275569] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:35.275579] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:35.275591] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:35.275602] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:35.367887] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=405288] gc stale ls task succ [2025-08-14 09:06:35.499535] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:35.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.742865] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7383][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1970364, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=4, FORMAT_END=1970343, ObMalloc#start=2, ObMalloc#end=5, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:33.739090] EDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1974655, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1974630, ALLOC_END=5, APPEND_END=18], seq:[0, 1, 2], BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0xfc0e73d 0x993f560 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:34.780635] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:35.716099] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=934042] temporary file module use no blocks [2025-08-14 09:06:35.716119] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=19] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:35.716138] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=16] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:35.716174] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=35] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623296213798, flush_io_finished_ret_:0, flush_io_finished_round_:26451, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:33.742865] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7383] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:35.716302] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1970364][errcode=0] epoll handler too much time :1973446 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:34.774211] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=928381][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:33.844|207=929.79ms|210=0us|222=1us|total=929.80ms) [2025-08-14 09:06:35.716333] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=940697][errcode=-4389] timer task use too much time(end=1755162395716332, start=1755162393844421, delta=1871911) [2025-08-14 09:06:35.716359] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=46] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:35.716356] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] task scheduled too delay(scheduled_delay=851512, expected_run_ts_us_=1755162394864842, ObClockGenerator::getRealClock()=1755162395716355, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:34.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:35.716398] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:34.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:34.774244] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=821693][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:33.951|207=822.94ms|210=0us|222=1us|total=822.95ms) [2025-08-14 09:06:35.716434] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=940765][errcode=-4389] timer task use too much time(end=1755162395716434, start=1755162393951303, delta=1765131) [2025-08-14 09:06:35.716448] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=922514, expected_run_ts_us_=1755162394793934, ObClockGenerator::getRealClock()=1755162395716448, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:34.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:35.716463] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:34.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:33.739090] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:8607757, max_lsn:1604090331, max_scn:{val:1755161756730847001, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:1556737504}, reuse_lsn:{lsn:1604090331}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:1562147291}}, last_submit_lsn:{lsn:1604090203}, last_submit_end_lsn:{lsn:1604090331}, last_submit_log_id:8607757, last_submit_log_pid:50, max_flushed_lsn:{lsn:1604090203}, max_flushed_end_lsn:{lsn:1604090331}, max_flushed_log_pid:50, committed_end_lsn:{lsn:1604090090}, last_slide_log_id:8607755, last_slide_scn:{val:1755161749741193000, v:0}, last_slide_lsn:{lsn:1604089968}, last_slide_end_lsn:{lsn:1604090090}, last_slide_log_pid:50, last_slide_log_accum_checksum:1446576759, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f990bfe4530}, [StateMgr]={this:0x7f990bfe89b0, self:"172.77.0.7:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"PENDING", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:51}, leader:"0.0.0.0:0", leader_epoch:-1, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:1604090331}, scan_disk_log_finished:true, last_check_start_id:8607756, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161901828039, palf_role_change_cb:0x7f990bfef470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:50, prev:{version:1, config:{config_version:{proposal_id:49, config_seq:45}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:50, config_seq:46}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_mode_pid:50}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_end_lsn:{lsn:1604088821}, prev_mode_pid:50}, reconfig_barrier:{prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_end_lsn:{lsn:1604088821}, prev_mode_pid:50}, persistent_config_version:{proposal_id:50, config_seq:46}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755161738854289, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755161738854289, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f990bfe5c70}, [ModeMgr]={palf_id:1, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:50, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:1543409664}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:51}, log_config_meta:{version:3, proposal_id:50, prev:{version:1, config:{config_version:{proposal_id:49, config_seq:45}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:50, config_seq:46}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_mode_pid:50}, log_snapshot_meta:{version:2, base_lsn:{lsn:1543409664}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:712704}, readable_log_tail:{lsn:712704}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:1610514432}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/meta", dir_fd:225, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66392064, block_header_serialize_buf_:0x7f990bfeabb8, flashback_version:0}, log_storage:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1755161901828095, this:0x7f990bfe90f0}, [Reconfirm]={palf_id:1, self:"172.77.0.7:2882", state:"RECONFIRM_MODE_META", new_proposal_id:51, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"172.77.0.7:2882", majority_max_accept_pid:50, majority_max_lsn:{lsn:1604090331}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:1755161901828373, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f990bfe8b30}) [395]OB_MALLOC COST TOO MUCH TIME, cost_time=1949101, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=994449], seq:[0, 1] [2025-08-14 09:06:33.764962] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1949139, owner: ObLog, click_count: 5, time dist:[FORMAT_END=18, ObMalloc#start=0, ObMalloc#end=1949113, ALLOC_END=1, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:33.764962] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:35.469519] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=387700][errcode=0] EASY SLOW: thread: 7f9954779300, lastrun: 1755162394.782138 cost: 0.687378 loop:26745, slowcnt: 6 [2025-08-14 09:06:35.717070] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1949139][errcode=0] epoll handler too much time :1952119 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:35.717089] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:35.717527] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=51500][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162393739175], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:06:34.774045] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=802276] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:34.774136] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=989352][errcode=0] epoll handler too much time :990862 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:35.718010] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=942490] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:35.718042] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=942456] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:35.718058] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:35.718093] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:35.718104] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:35.718112] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:34.774115] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=2000165, expected_run_ts_us_=1755162392773949, ObClockGenerator::getRealClock()=1755162394774114, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73959b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:32.773, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.718294] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=942750][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73959b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:32.773, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.718329] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:34.774|207=944.21ms|210=0us|222=2us|total=944.23ms) [2025-08-14 09:06:35.718341] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task use too much time(end=1755162395718341, start=1755162394774114, delta=944227) [2025-08-14 09:06:35.718350] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] task scheduled too delay(scheduled_delay=2931015, expected_run_ts_us_=1755162392787334, ObClockGenerator::getRealClock()=1755162395718349, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:32.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.718438] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=88][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:32.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.718448] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=2930943, expected_run_ts_us_=1755162392787504, ObClockGenerator::getRealClock()=1755162395718448, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72856b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:32.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.718458] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72856b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:32.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.718469] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=2930493, expected_run_ts_us_=1755162392787976, ObClockGenerator::getRealClock()=1755162395718469, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:32.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.718479] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:32.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.718488] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=2928326, expected_run_ts_us_=1755162392790161, ObClockGenerator::getRealClock()=1755162395718488, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919920b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196d590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:32.790, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.718497] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919920b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196d590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:32.790, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.718507] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=2878422, expected_run_ts_us_=1755162392840084, ObClockGenerator::getRealClock()=1755162395718507, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:32.840, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.718527] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:32.840, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.718537] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=2872412, expected_run_ts_us_=1755162392846124, ObClockGenerator::getRealClock()=1755162395718536, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d54b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ecd90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:32.846, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.718546] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d54b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ecd90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:32.846, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.718570] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] task scheduled too delay(scheduled_delay=2870516, expected_run_ts_us_=1755162392848053, ObClockGenerator::getRealClock()=1755162395718570, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ecc10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:32.848, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.718579] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ecc10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:32.848, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.718603] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1996425, expected_run_ts_us_=1755162393722177, ObClockGenerator::getRealClock()=1755162395718603, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.722, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.718612] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.722, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.718622] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1987740, expected_run_ts_us_=1755162393730881, ObClockGenerator::getRealClock()=1755162395718622, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.718644] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.718653] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1987126, expected_run_ts_us_=1755162393731526, ObClockGenerator::getRealClock()=1755162395718653, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891826cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891918190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.718664] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891826cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891918190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.718674] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1971336, expected_run_ts_us_=1755162393747338, ObClockGenerator::getRealClock()=1755162395718674, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:34.782159] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:35.718783] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=935209][errcode=0] epoll handler too much time :936643 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:34.803704] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:34.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:35.718813] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=913716][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:34.803|207=915.11ms|210=0us|222=1us|total=915.12ms) [2025-08-14 09:06:35.718832] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4389] timer task use too much time(end=1755162395718832, start=1755162394803702, delta=915130) [2025-08-14 09:06:35.718843] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=0] task scheduled too delay(scheduled_delay=708811, expected_run_ts_us_=1755162395010031, ObClockGenerator::getRealClock()=1755162395718843, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:35.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:35.718856] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:35.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:35.718878] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.718894] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] task scheduled too delay(scheduled_delay=1972801, expected_run_ts_us_=1755162393746092, ObClockGenerator::getRealClock()=1755162395718894, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919845b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.718913] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919845b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.718932] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] task scheduled too delay(scheduled_delay=1964414, expected_run_ts_us_=1755162393754518, ObClockGenerator::getRealClock()=1755162395718932, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.718942] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.718951] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1964228, expected_run_ts_us_=1755162393754723, ObClockGenerator::getRealClock()=1755162395718951, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.718961] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.718970] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1963566, expected_run_ts_us_=1755162393755403, ObClockGenerator::getRealClock()=1755162395718970, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.718988] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:34.748253] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=16] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:06:35.718997] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1959621, expected_run_ts_us_=1755162393759376, ObClockGenerator::getRealClock()=1755162395718997, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.719033] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.719042] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1961456, expected_run_ts_us_=1755162393757586, ObClockGenerator::getRealClock()=1755162395719042, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eacb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.719050] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eacb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.719060] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1951070, expected_run_ts_us_=1755162393767990, ObClockGenerator::getRealClock()=1755162395719060, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.767, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.719070] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.767, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.719081] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1939681, expected_run_ts_us_=1755162393779400, ObClockGenerator::getRealClock()=1755162395719081, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.719112] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.719167] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1940834, expected_run_ts_us_=1755162393778332, ObClockGenerator::getRealClock()=1755162395719166, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.719176] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.719185] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1940603, expected_run_ts_us_=1755162393778581, ObClockGenerator::getRealClock()=1755162395719185, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.719195] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.719204] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1949050, expected_run_ts_us_=1755162393770154, ObClockGenerator::getRealClock()=1755162395719204, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.719218] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.719497] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:35.011428] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:35.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:35.720242] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=707727][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:35.114|207=708.80ms|210=1us|222=1us|total=708.83ms) [2025-08-14 09:06:35.720263] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task use too much time(end=1755162395720262, start=1755162395011425, delta=708837) [2025-08-14 09:06:35.720272] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=0] task scheduled too delay(scheduled_delay=714784, expected_run_ts_us_=1755162395005487, ObClockGenerator::getRealClock()=1755162395720271, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:35.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:35.720292] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:35.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:35.720311] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=630995, expected_run_ts_us_=1755162395089316, ObClockGenerator::getRealClock()=1755162395720311, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:35.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:35.720322] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:35.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:35.720333] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=516036, expected_run_ts_us_=1755162395204297, ObClockGenerator::getRealClock()=1755162395720333, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:35.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:35.720343] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:35.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:35.275617] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:35.720550] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=444255][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:35.720570] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:35.720580] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:35.720591] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:35.720605] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:35.720613] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:35.720623] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:35.720633] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:35.720641] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:35.720649] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:35.720657] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:35.720667] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:35.720677] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:35.720836] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:35.720853] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:35.473396] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:35.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:35.720943] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=247159][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:35.473|207=247.54ms|210=0us|222=2us|total=247.56ms) [2025-08-14 09:06:35.720966] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4389] timer task use too much time(end=1755162395720965, start=1755162395473394, delta=247571) [2025-08-14 09:06:33.766396] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [854][T1002_ObXAHbWor][T1002][Y0-0000000000000000-0-0] [lt=29] XA scheduler heartbeat task statistics(avg_time=2) [2025-08-14 09:06:33.766396] EDIAG [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [854][T1002_ObXAHbWor][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1951668, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1951660, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b64d8ba 0x1b64d154 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:35.721424] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:35.719227] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1935575, expected_run_ts_us_=1755162393783651, ObClockGenerator::getRealClock()=1755162395719227, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72716b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a4910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.722097] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=2864][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72716b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a4910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.722121] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] task scheduled too delay(scheduled_delay=1934093, expected_run_ts_us_=1755162393788027, ObClockGenerator::getRealClock()=1755162395722121, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.722142] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.722168] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] task scheduled too delay(scheduled_delay=1933900, expected_run_ts_us_=1755162393788267, ObClockGenerator::getRealClock()=1755162395722168, ret=0, ret="OB_SUCCESS", *this={this:0x7f98918100b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.722186] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918100b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.722204] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] task scheduled too delay(scheduled_delay=1940991, expected_run_ts_us_=1755162393781212, ObClockGenerator::getRealClock()=1755162395722204, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.722311] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=106][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.722351] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] task scheduled too delay(scheduled_delay=1928556, expected_run_ts_us_=1755162393793794, ObClockGenerator::getRealClock()=1755162395722350, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.793, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.722370] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.793, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.722402] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] task scheduled too delay(scheduled_delay=1930001, expected_run_ts_us_=1755162393792400, ObClockGenerator::getRealClock()=1755162395722402, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180fe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.722415] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180fe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.722428] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=1916680, expected_run_ts_us_=1755162393805747, ObClockGenerator::getRealClock()=1755162395722427, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.805, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.727393] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=21] gc stale ls task succ [2025-08-14 09:06:35.728348] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:35.742442] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:35.744638] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:35.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:35.746894] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1748) [360][UserLocAsyncUp0][T0][YB42AC4D0007-00063C495D499F18-0-0] [lt=51124][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1753069619, tenant_id=1001) [2025-08-14 09:06:35.747924] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162393739175], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:06:35.747958] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=25] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162393739175]}) [2025-08-14 09:06:34.869251] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fcc90, caller:mds_tenant_service.cpp:mtl_start:117, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_145, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcf70, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:35.748881] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=878297][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:34.869|207=879.63ms|210=0us|222=1us|total=879.64ms) [2025-08-14 09:06:35.748897] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4389] timer task use too much time(end=1755162395748897, start=1755162394869249, delta=879648) [2025-08-14 09:06:35.748908] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=955587, expected_run_ts_us_=1755162394793320, ObClockGenerator::getRealClock()=1755162395748907, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fc8b0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcb90, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:1, time_interval:3.00s, expected_run_time:09:06:34.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:35.748923] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fc8b0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcb90, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:1, time_interval:3.00s, expected_run_time:09:06:34.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:35.748966] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1974655] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161974728707, last_flushed_end_lsn:{lsn:1604090331}}]) [2025-08-14 09:06:35.748993] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20] current log_task status(palf_id=1, self="172.77.0.7:2882", log_id=8607756, log_task={header:{begin_lsn:{lsn:1604090090}, end_lsn:{lsn:1604090203}, log_id:8607756, min_scn:{val:1755161749741193001, v:0}, max_scn:{val:1755161749741193001, v:0}, data_len:57, proposal_id:50, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:1604090090}, data_checksum:2031283194, accum_checksum:759186326, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1755161750716165, freeze_ts:1755161750716173, submit_ts:1755161750716183, flushed_ts:1755161853765770, gen_to_freeze cost time:8, gen_to_submit cost time:18, submit_to_flush cost time:103049587}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161974728707, last_flushed_end_lsn:{lsn:1604090331}}]) [2025-08-14 09:06:35.750190] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:35.753258] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:35.753289] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:35.753631] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=9] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:35.754920] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:35.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:35.758309] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:35.758704] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:35.759666] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:35.759707] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:35.759744] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=34] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:35.760561] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:35.762290] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=13] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:06:35.758133] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=67, get_gts_with_stc_cnt=1, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:06:35.763778] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5634] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162393739175]}) [2025-08-14 09:06:35.765253] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:35.767346] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:35.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:35.767383] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:35.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:35.770108] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EECD-0-0] [lt=24][errcode=0] cost too much time:mds_tenant_service.cpp:int oceanbase::storage::mds::ObTenantMdsTimer::get_tablet_oldest_scn_(oceanbase::storage::ObTablet &, share::SCN &), (this=|threshold=5.00ms|start at 09:06:34.607|344=0us|total=1.16s) [2025-08-14 09:06:35.748119] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=29] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:06:35.770403] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=9302] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:35.770433] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EECD-0-0] [lt=26][errcode=0] cost too much time:mds_tenant_service.cpp:int oceanbase::storage::mds::ObTenantMdsTimer::process_with_tablet_(oceanbase::storage::ObTablet &), (this=|threshold=10.00ms|start at 09:06:34.607|315=1us|317=1.16s|319=5us|total=1.16s) [2025-08-14 09:06:35.751907] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=28] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:06:35.774724] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:35.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:35.760536] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=18] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:06:35.774954] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=550586][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:35.774966] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:35.776747] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=998492] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:35.776786] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=998459] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:35.722440] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.805, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.777288] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=54760][errcode=0] task scheduled too delay(scheduled_delay=1964732, expected_run_ts_us_=1755162393812555, ObClockGenerator::getRealClock()=1755162395777288, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73927b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.812, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.777316] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73927b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.812, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.777331] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=1961864, expected_run_ts_us_=1755162393815466, ObClockGenerator::getRealClock()=1755162395777331, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7399090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.815, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.777345] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7399090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.815, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.777358] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1961059, expected_run_ts_us_=1755162393816298, ObClockGenerator::getRealClock()=1755162395777358, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.777370] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.777383] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1950053, expected_run_ts_us_=1755162393827329, ObClockGenerator::getRealClock()=1755162395777382, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73954b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.827, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.777394] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73954b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.827, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.777409] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=1903724, expected_run_ts_us_=1755162393873684, ObClockGenerator::getRealClock()=1755162395777409, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.873, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.777420] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:33.873, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.777437] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1529264, expected_run_ts_us_=1755162394248172, ObClockGenerator::getRealClock()=1755162395777437, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.248, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.777448] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.248, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.777466] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1041937, expected_run_ts_us_=1755162394735529, ObClockGenerator::getRealClock()=1755162395777466, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7270030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a3b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.735, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.777479] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7270030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a3b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.735, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.777490] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1037584, expected_run_ts_us_=1755162394739906, ObClockGenerator::getRealClock()=1755162395777490, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ecb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.739, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.777501] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ecb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.739, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.777513] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1044753, expected_run_ts_us_=1755162394732759, ObClockGenerator::getRealClock()=1755162395777513, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7353730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7396990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.777524] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7353730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7396990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.777538] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=1040676, expected_run_ts_us_=1755162394736861, ObClockGenerator::getRealClock()=1755162395777537, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.777549] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.777560] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1037404, expected_run_ts_us_=1755162394740156, ObClockGenerator::getRealClock()=1755162395777560, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ea30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.777571] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ea30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.777583] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1023283, expected_run_ts_us_=1755162394754299, ObClockGenerator::getRealClock()=1755162395777583, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919afbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.777595] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919afbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.777607] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1019074, expected_run_ts_us_=1755162394758532, ObClockGenerator::getRealClock()=1755162395777607, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919854b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.777618] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919854b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.777629] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1018896, expected_run_ts_us_=1755162394758733, ObClockGenerator::getRealClock()=1755162395777629, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.777640] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.780086] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=2437][errcode=0] task scheduled too delay(scheduled_delay=1021114, expected_run_ts_us_=1755162394758970, ObClockGenerator::getRealClock()=1755162395780085, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.780100] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.780112] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1027203, expected_run_ts_us_=1755162394752908, ObClockGenerator::getRealClock()=1755162395780112, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d45b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.780123] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d45b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.780135] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1013543, expected_run_ts_us_=1755162394766592, ObClockGenerator::getRealClock()=1755162395780135, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72747b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.780148] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72747b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.780172] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] task scheduled too delay(scheduled_delay=1013763, expected_run_ts_us_=1755162394766408, ObClockGenerator::getRealClock()=1755162395780172, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.780184] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.780197] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1004577, expected_run_ts_us_=1755162394775620, ObClockGenerator::getRealClock()=1755162395780197, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.780209] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.780223] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=1004567, expected_run_ts_us_=1755162394775655, ObClockGenerator::getRealClock()=1755162395780223, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.780244] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.780256] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1008355, expected_run_ts_us_=1755162394771901, ObClockGenerator::getRealClock()=1755162395780256, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.780268] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.780277] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1007305, expected_run_ts_us_=1755162394772971, ObClockGenerator::getRealClock()=1755162395780276, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.780288] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.780298] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=994438, expected_run_ts_us_=1755162394785859, ObClockGenerator::getRealClock()=1755162395780298, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.780308] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.780317] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=995157, expected_run_ts_us_=1755162394785160, ObClockGenerator::getRealClock()=1755162395780317, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.780327] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.780339] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=976260, expected_run_ts_us_=1755162394804079, ObClockGenerator::getRealClock()=1755162395780339, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919836b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.780349] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919836b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.780366] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] task scheduled too delay(scheduled_delay=949966, expected_run_ts_us_=1755162394830400, ObClockGenerator::getRealClock()=1755162395780366, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.780378] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.780388] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=949362, expected_run_ts_us_=1755162394831025, ObClockGenerator::getRealClock()=1755162395780387, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.831, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.780396] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.831, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.780412] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=918278, expected_run_ts_us_=1755162394862133, ObClockGenerator::getRealClock()=1755162395780412, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.862, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.780423] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:34.862, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:35.780447] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:35.782139] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:35.787311] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:35.789309] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=23] gc stale ls task succ [2025-08-14 09:06:35.804972] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:35.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:35.816419] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:35.816512] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:35.818361] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:35.818411] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:35.818451] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:35.818491] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:35.818528] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:35.818542] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:35.818552] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:35.828327] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:35.839207] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=20] PNIO [ratelimit] time: 1755162395839206, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007615, add_bytes: 0 [2025-08-14 09:06:35.875027] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:06:35.875057] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:35.916498] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:35.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:35.916579] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:35.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:35.931266] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:35.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:35.969395] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=18] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:35.975092] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:35.991868] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:35.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.011972] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:36.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.012013] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:36.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.016564] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=60][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:35.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.016639] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:35.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.026534] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:36.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.031325] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:35.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.075167] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:36.092548] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:36.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.116634] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:36.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.116700] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:36.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.131405] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:36.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.175226] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:36.213256] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:36.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.216693] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:36.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.216759] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:36.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.221016] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:36.221044] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:36.221063] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:36.221076] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:36.221089] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:36.221100] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:36.221115] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:36.221134] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:36.221149] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:36.221168] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:36.221185] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:36.221199] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:36.221209] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:36.221221] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:36.221245] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:36.221256] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:36.221264] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:36.221273] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:36.221283] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:36.221292] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:36.221450] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:36.221475] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:36.231465] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:36.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.243470] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:36.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.253529] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:36.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.263541] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:36.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.263581] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:36.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.275285] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:36.278393] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:36.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.308646] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:36.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.316754] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:36.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.316829] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:36.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.331525] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:36.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.334939] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:36.367952] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=19] gc stale ls task succ [2025-08-14 09:06:36.375353] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:36.416813] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:36.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.416874] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:36.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.431596] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:36.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.475376] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:36.495025] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:36.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.515059] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:36.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.515090] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:36.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.516871] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:36.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.516932] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:36.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.518041] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=247157][errcode=0] EASY SLOW: thread: 7f995427d300, lastrun: 1755162250.789339 cost: 145.728700 loop:26617, slowcnt: 1125 [2025-08-14 09:06:36.520208] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:36.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.531657] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:36.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.575429] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:36.599464] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:36.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.616926] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:36.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.617042] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:36.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.633295] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:36.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.675457] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:36.710434] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:36.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.716998] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=52][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:36.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.717158] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:36.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.719267] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:36.721666] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:36.721701] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:36.721719] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:36.721730] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:36.721742] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:36.721754] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:36.721769] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:36.721787] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:36.721802] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:36.721813] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:36.721824] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:36.721839] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:36.721856] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:36.721869] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:36.721882] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:36.721892] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:36.721903] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:36.721914] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:36.721926] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:36.721945] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:36.722111] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:36.722125] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:36.723261] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:36.724809] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=23] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:36.724839] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=24] temporary file module use no blocks [2025-08-14 09:06:36.724849] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:36.724870] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=20] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:36.724884] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623297218644, flush_io_finished_ret_:0, flush_io_finished_round_:26414, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:36.726072] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=31] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:36.726098] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=21] temporary file module use no blocks [2025-08-14 09:06:36.726107] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:36.726120] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:36.726133] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623298155974, flush_io_finished_ret_:0, flush_io_finished_round_:26452, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:36.727906] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:06:36.729802] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:36.733778] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=63][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:36.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:35.774549] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EECD-0-0] [lt=16] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:06:36.737630] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] get_fetch_log_dst failed or invalid(ret=0, palf_id=1001, self="172.77.0.7:2882", fetch_log_dst="172.77.0.7:2882") [2025-08-14 09:06:36.737658] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=23] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:06:36.739974] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11] LogLoopThread round_cost_time(us)(round_cost_time=3896434) [2025-08-14 09:06:36.739995] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] get_fetch_log_dst failed or invalid(ret=0, palf_id=1, self="172.77.0.7:2882", fetch_log_dst="172.77.0.7:2882") [2025-08-14 09:06:36.740006] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:06:36.741081] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:36.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.751379] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:36.751577] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:36.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.753341] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:36.754254] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:35.746920] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:389) [360][UserLocAsyncUp0][T0][YB42AC4D0007-00063C495D499F18-0-0] [lt=21][errcode=-4012] query failed(ret=-4012, conn=0x7f98eb024a40, start=1755162376738599, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 AND ls_id IN (1001) ORDER BY tenant_id, ls_id, svr_ip, svr_port, cluster_id=1753069619) [2025-08-14 09:06:36.757461] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:35.746920] EDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:389) [360][UserLocAsyncUp0][T0][YB42AC4D0007-00063C495D499F18-0-0] [lt=21][errcode=-4012] LOGGER COST TOO MUCH TIME, cost: 1006295, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1006290, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x2615fb09 0x9a04720 0x1eeb9e64 0x1eeb94f7 0x201f419e 0x2020e480 0x9a194f6 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:36.758306] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:36.758354] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=39] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:36.758466] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72756b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:36.762680] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:36.763933] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=56] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:06:36.758542] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=73][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.724, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:36.764366] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=5811][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a36b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:36.764395] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919bfbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aa690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:36.764406] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919911b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196cc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:36.764415] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d40b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:36.766276] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:36.764424] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:36.766349] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1917][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:36.766370] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:36.766380] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:36.766391] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.744, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:36.767611] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EECD-0-0] [lt=959148][errcode=-4389] run this task use too much time(*p_task_={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:0, time_interval:3.00s, expected_run_time:09:06:37.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.767634] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EECD-0-0] [lt=21][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:06:36.767|total=2.16s) [2025-08-14 09:06:35.882320] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [280][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:29, tid:660}]) [2025-08-14 09:06:36.766401] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aaf90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:36.771150] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=4737][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aba10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:36.771180] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:36.771189] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196cb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:36.771199] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:36.771209] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.751, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:36.776303] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=5082][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:36.777080] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=12] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:36.777117] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=9] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:36.749896] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:36.777460] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27520][errcode=0] epoll handler too much time :27573 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:36.766323] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25279] get_min_unreplayed_log_info(lsn={lsn:8461098908}, scn={val:1755100947600130001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161920778323, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:23084275, ret_code_:0}}, next_to_submit_lsn_:{lsn:8461098908}, next_to_submit_scn_:{val:1755100947600130001, v:0}, base_lsn_:{lsn:7515734016}, base_scn_:{val:1753840867171901045, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6e05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:8461098908}, end_lsn:{lsn:8461098908}, read_buf:{buf_len_:3747840, buf_:0x7f98f6e05000}, block_size:67104768, log_storage_:0x7f99297a3bf0, log_storage_:{log_tail:{lsn:8461098908}, readable_log_tail:{lsn:8461098908}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8455200768}, min_scn:{val:1755160384851871002, v:0}, curr_block_id:126, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:215, min_block_id:93, max_block_id:127, curr_writable_block_id:126}, logical_block_size_:67104768, curr_block_writable_size_:61206628, block_header_serialize_buf_:0x7f99297a48b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:198, read_io_size:699822080, read_disk_cost_ts:4178035, start_lsn:{lsn:7515734016}}}}}}) [2025-08-14 09:06:36.744316] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=1971635] blacklist_loop exec finished(cost_time=16, is_enabled=true, send_cnt=0) [2025-08-14 09:06:36.782278] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919acf10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:36.782304] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:36.782723] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:32.765208] EDIAG [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4012621, owner: ObLog, click_count: 7, time dist:[FORMAT_END=4011558, ObMalloc#start=1, ObMalloc#end=105, ObMalloc#start=498, ObMalloc#end=458, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xff22be4 0x97a9cd7 0xff1a920 0x982e1ac 0xfefdb03 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:36.770514] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=13] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:36.783845] WDIAG ev_invoke_pending (ev.c:2136) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=32388][errcode=0] epoll handler too much time :32440 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:36.783405] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=26253][errcode=0] epoll handler too much time :26299 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:32.765208] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=11] get_min_unreplayed_log_info(lsn={lsn:1604088821}, scn={val:1755100948296325001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873479, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:12492179, ret_code_:0}}, next_to_submit_lsn_:{lsn:1604088821}, next_to_submit_scn_:{val:1755100948296325001, v:0}, base_lsn_:{lsn:1409200128}, base_scn_:{val:1753840802540914000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:1604088821}, end_lsn:{lsn:1604088821}, read_buf:{buf_len_:3747840, buf_:0x7f98f6a05000}, block_size:67104768, log_storage_:0x7f990bfebbf0, log_storage_:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:42, read_io_size:147709952, read_disk_cost_ts:1061450, start_lsn:{lsn:1409200128}}}}}}) [2025-08-14 09:06:36.783285] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26198][errcode=0] epoll handler too much time :26248 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:35.792419] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [561][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=29154] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=16, total_destructed_detector=16, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f994b3ee2b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-08-14 09:06:36.786099] WDIAG [DETECT] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [561][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=992174][errcode=0] cost too much time:ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t), (this=|threshold=100.00ms|start at 09:06:36.786|total=993.70ms) [2025-08-14 09:06:36.789039] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:36.789264] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:36.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.794249] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919abe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:36.795313] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=18] gc stale ls task succ [2025-08-14 09:06:36.796329] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:36.809392] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:36.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.819363] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:36.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.833827] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:36.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.846814] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=20] PNIO [ratelimit] time: 1755162396846811, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007605, add_bytes: 0 [2025-08-14 09:06:36.896346] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:36.919426] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:36.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.933899] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:36.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.971408] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=19] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:36.996403] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:36.997207] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:36.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:37.019487] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:36.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:37.025195] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:37.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:37.033975] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:36.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:37.096432] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:37.119547] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:37.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:37.134038] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:37.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:37.196491] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:37.219607] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:37.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:37.222305] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:37.222328] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:37.222343] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:37.222352] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:37.222360] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:37.222369] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:37.222380] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:37.222390] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:37.248861] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:37.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:37.258931] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:37.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:37.276730] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:37.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:37.296550] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:37.306936] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:37.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:37.319665] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:37.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:37.367989] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=18] gc stale ls task succ [2025-08-14 09:06:37.396610] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:37.419725] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:37.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:37.490437] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:37.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:37.496668] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:37.519785] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:37.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:37.528435] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:37.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:37.596728] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:37.619846] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:37.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:37.696834] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:37.523643] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] EASY SLOW: thread: 7f995427d300, lastrun: 1755162250.789339 cost: 146.734301 loop:26617, slowcnt: 1135 [2025-08-14 09:06:37.720252] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:37.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.794286] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:37.720972] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=925280][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:36.794|207=926.68ms|210=0us|222=2us|total=926.71ms) [2025-08-14 09:06:37.720995] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4389] timer task use too much time(end=1755162397720994, start=1755162396794285, delta=926709) [2025-08-14 09:06:36.796629] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=55][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:36.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:37.721272] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:36.819607] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:36.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:37.222403] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:37.723133] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=499969][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:37.723150] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:37.723165] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:37.723184] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:37.723194] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:37.234099] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:37.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:37.750574] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:37.751815] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=9] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:37.751847] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=482] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:37.751869] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=641] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:37.753302] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:37.754254] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:37.755608] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:37.756993] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:37.759311] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:37.717242] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:36.581995] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1180264, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1177331, ALLOC_END=2929, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:37.761619] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1085) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=4012621] replay status is not follower(min_unreplayed_lsn={lsn:1604088821}, base_lsn={lsn:1409200128}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873479, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:12492179, ret_code_:0}}, next_to_submit_lsn_:{lsn:1604088821}, next_to_submit_scn_:{val:1755100948296325001, v:0}, base_lsn_:{lsn:1409200128}, base_scn_:{val:1753840802540914000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:1604088821}, end_lsn:{lsn:1604088821}, read_buf:{buf_len_:3747840, buf_:0x7f98f6a05000}, block_size:67104768, log_storage_:0x7f990bfebbf0, log_storage_:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:42, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:42, read_io_size:147709952, read_disk_cost_ts:1061450, start_lsn:{lsn:1409200128}}}}}}) [2025-08-14 09:06:37.721008] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=935243, expected_run_ts_us_=1755162396785763, ObClockGenerator::getRealClock()=1755162397721007, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199fe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919acd90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:37.761723] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=40651][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199fe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919acd90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:37.761749] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] task scheduled too delay(scheduled_delay=978996, expected_run_ts_us_=1755162396782752, ObClockGenerator::getRealClock()=1755162397761749, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e80b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:37.723101] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=902124][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:36.819|207=903.49ms|210=1us|222=1us|total=903.51ms) [2025-08-14 09:06:37.761773] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=98] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:37.723204] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:37.761811] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=38545][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:37.761828] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:37.761841] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:37.761855] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:37.761868] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:37.723220] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=488379][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:37.234|207=489.12ms|210=0us|222=1us|total=489.13ms) [2025-08-14 09:06:36.581995] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fe030, caller:ob_table_lock_service.cpp:start:276, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE6$_1273, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe310, time_wheel:0x7f99535f3990, thread_pool:0x7f99535f3170, is_running:1, time_interval:10.00s, expected_run_time:09:06:36.426, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:37.721036] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=923007][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:36.796|207=924.40ms|210=0us|222=2us|total=924.42ms) [2025-08-14 09:06:37.729485] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=17] gc stale ls task succ [2025-08-14 09:06:37.766458] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=14] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:06:37.766596] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:37.770912] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:37.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:36.762792] EDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4038] LOGGER COST TOO MUCH TIME, cost: 1009769, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1009758, ObMalloc#start=1, ObMalloc#end=4, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1ba227a3 0x1b89078a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:36.762792] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:06:37.761764] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e80b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:37.777695] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15902][errcode=0] task scheduled too delay(scheduled_delay=988163, expected_run_ts_us_=1755162396789530, ObClockGenerator::getRealClock()=1755162397777694, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a31b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ede10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:37.777720] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a31b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ede10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:37.777735] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=971535, expected_run_ts_us_=1755162396806200, ObClockGenerator::getRealClock()=1755162397777735, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.806, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:37.766037] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=44930][errcode=-4389] timer task use too much time(end=1755162397766036, start=1755162396796627, delta=969409) [2025-08-14 09:06:37.762038] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=38759][errcode=-4389] timer task use too much time(end=1755162397762038, start=1755162397234097, delta=527941) [2025-08-14 09:06:37.764080] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=1180264][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:36.581|207=1.18s|210=0us|222=1us|total=1.18s) [2025-08-14 09:06:37.762023] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:37.779466] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17410][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:37.780269] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=6] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:37.784532] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=13301] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:37.795431] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=12] gc stale ls task succ [2025-08-14 09:06:37.798297] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:06:37.798335] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:37.801323] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:37.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:37.820324] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:37.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:37.854469] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=16] PNIO [ratelimit] time: 1755162397854467, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007656, add_bytes: 0 [2025-08-14 09:06:37.860526] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fc8b0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcb90, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:1, time_interval:3.00s, expected_run_time:09:06:37.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:37.879242] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=17100][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:37.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:37.894413] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc4738, timer:0x7f9970f40f80, task:0x564d6cd5b940, task_type:"N9oceanbase3sql15ObSQLSessionMgrE", scheduled_time:1755161613781775, pushed_time:0, delay:5000000}, running_task_token={this:0x7f9970fc5fb8, timer:0x7f9970f40f80, task:0x564d727182f0, task_type:"N9oceanbase5share16ObAshRefreshTaskE", scheduled_time:1755161611717613, pushed_time:1755161614757742, delay:120000000}, total_delay=784112637, threashold=1000000, tenant_id=500) [2025-08-14 09:06:37.900256] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:37.920394] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:37.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:37.973920] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=12] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:38.001286] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:38.020460] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:37.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:38.026770] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:38.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:38.079363] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:37.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:38.101336] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:38.120527] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:38.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:38.202247] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:38.220601] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:38.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:38.276590] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=52][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:38.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:38.279484] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:38.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:38.279610] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:38.279638] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:38.279657] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:38.279667] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:38.279678] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:06:37.726, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:06:07.726], idx=471, next_allow_print_ts="09:07:07.726") [2025-08-14 09:06:38.279689] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:38.279701] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:38.279722] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:38.279733] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:38.279748] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:38.279758] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:38.279769] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:38.279783] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:38.279793] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:38.279810] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:38.279830] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:38.279841] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:38.279851] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:38.279862] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:38.279875] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:38.279887] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:38.280050] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:38.280063] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:38.302309] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:38.306739] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:38.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:38.320666] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:38.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:38.368052] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=11] gc stale ls task succ [2025-08-14 09:06:38.402378] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:38.420731] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:38.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:38.479603] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:38.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:38.502413] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:38.520798] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:38.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:38.525709] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=274453][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 17.781088 loop:26707, slowcnt: 174 [2025-08-14 09:06:38.528130] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:38.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:38.602446] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:38.620860] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:38.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:38.679733] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:38.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:38.702463] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:38.721485] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=38] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:38.721530] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:38.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:38.730669] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=36856] gc stale ls task succ [2025-08-14 09:06:38.737163] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=9] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:37.750416] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:37.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:38.737226] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=985315][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:37.750|207=986.81ms|210=0us|222=1us|total=986.84ms) [2025-08-14 09:06:38.737252] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4389] timer task use too much time(end=1755162398737251, start=1755162397750410, delta=986841) [2025-08-14 09:06:37.777747] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.806, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737265] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=992630, expected_run_ts_us_=1755162397744633, ObClockGenerator::getRealClock()=1755162398737264, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:37.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:38.737282] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:37.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:38.737270] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=958068][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:37.777|207=959.53ms|210=1us|222=1us|total=959.55ms) [2025-08-14 09:06:38.737295] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4389] timer task use too much time(end=1755162398737295, start=1755162397777734, delta=959561) [2025-08-14 09:06:38.737309] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1929863, expected_run_ts_us_=1755162396807444, ObClockGenerator::getRealClock()=1755162398737308, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737324] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737336] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1932956, expected_run_ts_us_=1755162396804380, ObClockGenerator::getRealClock()=1755162398737336, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:37.727829] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1010667, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:38.737345] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737358] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1936918, expected_run_ts_us_=1755162396800439, ObClockGenerator::getRealClock()=1755162398737357, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.800, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737366] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.800, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737375] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1932030, expected_run_ts_us_=1755162396805344, ObClockGenerator::getRealClock()=1755162398737375, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728e330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7290690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.805, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737383] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728e330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7290690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.805, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737392] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1933666, expected_run_ts_us_=1755162396803726, ObClockGenerator::getRealClock()=1755162398737392, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.803, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737401] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.803, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737410] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1912794, expected_run_ts_us_=1755162396824616, ObClockGenerator::getRealClock()=1755162398737410, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.824, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737419] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.824, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737428] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1905094, expected_run_ts_us_=1755162396832333, ObClockGenerator::getRealClock()=1755162398737428, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.832, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737436] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.832, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737445] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1872515, expected_run_ts_us_=1755162396864930, ObClockGenerator::getRealClock()=1755162398737445, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ade10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.864, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737454] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ade10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.864, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737463] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1872373, expected_run_ts_us_=1755162396865089, ObClockGenerator::getRealClock()=1755162398737463, ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737471] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737482] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1872059, expected_run_ts_us_=1755162396865422, ObClockGenerator::getRealClock()=1755162398737481, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919adc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737490] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919adc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737499] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1871957, expected_run_ts_us_=1755162396865541, ObClockGenerator::getRealClock()=1755162398737499, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919adb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737507] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919adb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737516] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1859404, expected_run_ts_us_=1755162396878111, ObClockGenerator::getRealClock()=1755162398737516, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72865b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.878, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737526] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72865b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.878, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737535] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1837695, expected_run_ts_us_=1755162396899839, ObClockGenerator::getRealClock()=1755162398737534, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728de30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7290b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.899, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737543] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728de30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7290b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.899, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737552] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1834696, expected_run_ts_us_=1755162396902855, ObClockGenerator::getRealClock()=1755162398737552, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.902, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737560] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:36.902, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737573] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1625916, expected_run_ts_us_=1755162397111656, ObClockGenerator::getRealClock()=1755162398737573, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.111, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737581] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.111, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737598] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1006876, expected_run_ts_us_=1755162397730722, ObClockGenerator::getRealClock()=1755162398737598, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737607] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737623] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] task scheduled too delay(scheduled_delay=1006412, expected_run_ts_us_=1755162397731210, ObClockGenerator::getRealClock()=1755162398737623, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737633] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737641] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1005395, expected_run_ts_us_=1755162397732246, ObClockGenerator::getRealClock()=1755162398737641, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891991430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ce10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737650] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891991430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ce10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737666] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] task scheduled too delay(scheduled_delay=995260, expected_run_ts_us_=1755162397742406, ObClockGenerator::getRealClock()=1755162398737666, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aac90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737675] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aac90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737685] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=989979, expected_run_ts_us_=1755162397747705, ObClockGenerator::getRealClock()=1755162398737685, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737693] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737702] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=995167, expected_run_ts_us_=1755162397742535, ObClockGenerator::getRealClock()=1755162398737702, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737710] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737719] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=990413, expected_run_ts_us_=1755162397747306, ObClockGenerator::getRealClock()=1755162398737719, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e62b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737729] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e62b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737738] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=992037, expected_run_ts_us_=1755162397745700, ObClockGenerator::getRealClock()=1755162398737738, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73940b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737746] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73940b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737755] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=980256, expected_run_ts_us_=1755162397757499, ObClockGenerator::getRealClock()=1755162398737755, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eeb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737765] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eeb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737773] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=980128, expected_run_ts_us_=1755162397757645, ObClockGenerator::getRealClock()=1755162398737773, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73936b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eec90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737782] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73936b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eec90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737791] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=977975, expected_run_ts_us_=1755162397759815, ObClockGenerator::getRealClock()=1755162398737791, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737799] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737810] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=971259, expected_run_ts_us_=1755162397766550, ObClockGenerator::getRealClock()=1755162398737810, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737818] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737827] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=977287, expected_run_ts_us_=1755162397760539, ObClockGenerator::getRealClock()=1755162398737827, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72765b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737835] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72765b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737844] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=969145, expected_run_ts_us_=1755162397768698, ObClockGenerator::getRealClock()=1755162398737844, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737854] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737863] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=971494, expected_run_ts_us_=1755162397766368, ObClockGenerator::getRealClock()=1755162398737862, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737871] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737879] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=976187, expected_run_ts_us_=1755162397761692, ObClockGenerator::getRealClock()=1755162398737879, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737888] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737898] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=963345, expected_run_ts_us_=1755162397774552, ObClockGenerator::getRealClock()=1755162398737898, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aab10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.774, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737906] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aab10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.774, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737915] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=965201, expected_run_ts_us_=1755162397772714, ObClockGenerator::getRealClock()=1755162398737915, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739ef90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737923] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739ef90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737932] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=957932, expected_run_ts_us_=1755162397779999, ObClockGenerator::getRealClock()=1755162398737932, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a45b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737942] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a45b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737951] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=962420, expected_run_ts_us_=1755162397775530, ObClockGenerator::getRealClock()=1755162398737951, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eee10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737959] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eee10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737967] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=962259, expected_run_ts_us_=1755162397775708, ObClockGenerator::getRealClock()=1755162398737967, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73931b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737975] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73931b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737986] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=950878, expected_run_ts_us_=1755162397787107, ObClockGenerator::getRealClock()=1755162398737986, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.737994] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.738002] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=951588, expected_run_ts_us_=1755162397786414, ObClockGenerator::getRealClock()=1755162398738002, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.738010] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.738019] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=951270, expected_run_ts_us_=1755162397786748, ObClockGenerator::getRealClock()=1755162398738019, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7273c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.738028] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7273c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.738037] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=954114, expected_run_ts_us_=1755162397783922, ObClockGenerator::getRealClock()=1755162398738037, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d27b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.738045] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d27b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.738053] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=948668, expected_run_ts_us_=1755162397789385, ObClockGenerator::getRealClock()=1755162398738053, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d94b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.738061] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d94b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.738078] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] task scheduled too delay(scheduled_delay=957758, expected_run_ts_us_=1755162397780319, ObClockGenerator::getRealClock()=1755162398738077, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.780, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.738086] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.780, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.738094] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=957605, expected_run_ts_us_=1755162397780489, ObClockGenerator::getRealClock()=1755162398738094, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739bbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.780, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.738103] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739bbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.780, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.738111] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=956342, expected_run_ts_us_=1755162397781769, ObClockGenerator::getRealClock()=1755162398738111, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fa10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.738121] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fa10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.738136] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=954914, expected_run_ts_us_=1755162397783221, ObClockGenerator::getRealClock()=1755162398738136, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f727dc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f729ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.738144] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f727dc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f729ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.738153] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=954457, expected_run_ts_us_=1755162397783695, ObClockGenerator::getRealClock()=1755162398738153, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f727d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f729af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.738161] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f727d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f729af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.738170] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=952326, expected_run_ts_us_=1755162397785844, ObClockGenerator::getRealClock()=1755162398738170, ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.738187] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.738197] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=944454, expected_run_ts_us_=1755162397793742, ObClockGenerator::getRealClock()=1755162398738197, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891814fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.793, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.738205] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891814fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.793, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.738213] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=941766, expected_run_ts_us_=1755162397796447, ObClockGenerator::getRealClock()=1755162398738213, ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.738223] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:37.761793] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=38631][errcode=-4389] timer task use too much time(end=1755162397761792, start=1755162396819604, delta=942188) [2025-08-14 09:06:38.738280] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=975008][errcode=0] task scheduled too delay(scheduled_delay=944345, expected_run_ts_us_=1755162397793934, ObClockGenerator::getRealClock()=1755162398738280, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:37.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:38.738313] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:37.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:37.778859] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12799][errcode=0] task scheduled too delay(scheduled_delay=1023370, expected_run_ts_us_=1755162396755487, ObClockGenerator::getRealClock()=1755162397778858, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:36.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:37.782498] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:38.738660] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=958350][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:36.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:38.738666] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=954721][errcode=0] epoll handler too much time :956176 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:38.738676] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:37.778|207=959.82ms|210=0us|222=1us|total=959.83ms) [2025-08-14 09:06:38.738686] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] timer task use too much time(end=1755162398738686, start=1755162397778857, delta=959829) [2025-08-14 09:06:38.738698] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=0] task scheduled too delay(scheduled_delay=1649381, expected_run_ts_us_=1755162397089316, ObClockGenerator::getRealClock()=1755162398738697, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:37.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:38.738708] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:37.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:38.738719] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1534421, expected_run_ts_us_=1755162397204297, ObClockGenerator::getRealClock()=1755162398738718, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:37.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:38.738727] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:37.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:38.738744] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=979610, expected_run_ts_us_=1755162397759134, ObClockGenerator::getRealClock()=1755162398738744, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:37.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:38.738753] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:37.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:38.738774] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc190, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc470, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:10.00s, expected_run_time:09:06:38.720, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:37.739360] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=1204][errcode=-4015] clock out of order(cur_ts=1755162396759724, cur_ts_=1755162396723476, delta=36248) [2025-08-14 09:06:37.779442] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=15337][errcode=-4389] timer task use too much time(end=1755162397779441, start=1755162396581993, delta=1197448) [2025-08-14 09:06:37.727829] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:38.740063] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1010667][errcode=0] epoll handler too much time :1012243 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:38.740081] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:37.783297] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:38.740095] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=955353][errcode=0] epoll handler too much time :956808 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:38.744366] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=961] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:38.747701] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8030, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8310, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:10.00s, expected_run_time:09:06:38.730, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:38.749603] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:38.754752] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=13] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:38.754803] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:38.754829] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:38.755309] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:38.755351] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:38.756689] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:38.757251] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:38.757625] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=950349, expected_run_ts_us_=1755162397807273, ObClockGenerator::getRealClock()=1755162398757623, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7facb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.757664] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7facb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.757685] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] task scheduled too delay(scheduled_delay=762028, expected_run_ts_us_=1755162397995656, ObClockGenerator::getRealClock()=1755162398757685, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199fbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919acc10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.995, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.757700] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199fbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919acc10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:37.995, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.757725] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728bb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:38.718, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.757739] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180fbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:38.719, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.757753] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:38.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.757767] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:38.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.757781] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:38.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.758013] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:38.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:38.758906] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:38.762266] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:38.765481] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:38.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:38.765511] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:38.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:38.766258] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:38.767122] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=20] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:06:38.767808] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918105b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:38.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.770855] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:38.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:04:15.742878] EDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=110797881][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 142812144, owner: ObLog, click_count: 7, time dist:[FORMAT_END=1002144, ObMalloc#start=2, ObMalloc#end=1008399, ObMalloc#start=140801592, ObMalloc#end=2, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1b53d414 0x97ad9aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:15.742878] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=110797881][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:37.774119] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1009769] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2025-08-14 09:06:38.777868] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:38.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.777899] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:38.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.777918] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891815730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:38.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.780229] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:38.780262] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:38.780279] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:38.780299] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:38.780311] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:38.780323] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:38.780338] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:38.780350] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:38.780366] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:38.780376] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:38.780375] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=21] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:38.780389] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:38.780405] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:38.780416] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:38.780428] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:38.780440] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:38.780451] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:38.780462] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:38.780474] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:38.780486] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:38.780498] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:38.780651] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:38.780664] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:38.781754] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:38.782186] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:38.784625] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=28] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:38.787930] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eca90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:38.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.787954] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e94b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:38.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.787967] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:38.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.787980] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180ea30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:38.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.797997] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:38.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:38.798024] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:38.799254] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=23] gc stale ls task succ [2025-08-14 09:06:38.801106] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:38.802546] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:38.821592] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:38.837809] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:38.837876] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:38.837928] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:38.837976] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:38.838016] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:38.838034] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:38.838051] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:38.838341] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:38.855409] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:06:38.855437] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22] follower pending log not clear, need wait(bool_ret=false, palf_id=1001, last_slide_end_lsn={lsn:7323686009}, pending_end_lsn={lsn:7323687369}) [2025-08-14 09:06:38.862044] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=34] PNIO [ratelimit] time: 1755162398862043, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007576, add_bytes: 0 [2025-08-14 09:06:38.879857] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:38.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:38.902555] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=6] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:06:38.902582] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:38.902680] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc4ff8, timer:0x7f9970f40f80, task:0x564d6d07c170, task_type:"N9oceanbase8observer8ObServer25ObRefreshNetworkSpeedTaskE", scheduled_time:1755161615749643, pushed_time:0, delay:1000000}, running_task_token={this:0x7f9970fc5fb8, timer:0x7f9970f40f80, task:0x564d727182f0, task_type:"N9oceanbase5share16ObAshRefreshTaskE", scheduled_time:1755161611717613, pushed_time:1755161614757742, delay:120000000}, total_delay=783153036, threashold=1000000, tenant_id=500) [2025-08-14 09:06:38.902713] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc49d8, timer:0x7f9970f40f80, task:0x564d6ef827d0, task_type:"N9oceanbase7storage20ObDDLCtrlSpeedHandle22RefreshSpeedHandleTaskE", scheduled_time:1755161615757725, pushed_time:0, delay:1000000}, running_task_token={this:0x7f9970fc5fb8, timer:0x7f9970f40f80, task:0x564d727182f0, task_type:"N9oceanbase5share16ObAshRefreshTaskE", scheduled_time:1755161611717613, pushed_time:1755161614757742, delay:120000000}, total_delay=783144988, threashold=1000000, tenant_id=500) [2025-08-14 09:06:38.974593] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=25] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:38.999591] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=159][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:39.002591] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:39.017968] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:39.079978] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:38.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.102627] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:39.178158] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [280][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=887382][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:27, tid:770}]) [2025-08-14 09:06:39.202695] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:39.219230] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:39.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.221831] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:39.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.238593] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:39.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.241073] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:39.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.251132] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:39.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.269535] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:39.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.269568] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:39.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.274266] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:39.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.280075] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:39.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.280820] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:39.280844] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:39.280862] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:39.280874] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:39.280887] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:39.280900] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:39.280915] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:39.280928] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:39.280943] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:39.280954] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:39.280965] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:39.280980] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:39.280991] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:39.281003] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:39.281022] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:39.281033] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:39.281044] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:39.281056] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:39.281069] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:39.281081] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:39.281270] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:39.281285] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:39.302752] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:39.304470] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:39.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.321897] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:39.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.338627] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:39.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.368092] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=22] gc stale ls task succ [2025-08-14 09:06:39.402803] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:39.421956] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:39.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.438688] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:39.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.480194] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:39.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.492482] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:39.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.502876] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:39.511022] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:39.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.511053] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:39.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.522015] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:39.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.525886] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:39.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.526299] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 18.781678 loop:26707, slowcnt: 184 [2025-08-14 09:06:39.538746] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:39.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.591521] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:39.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.602947] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:39.622053] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:39.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.638788] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:39.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.680301] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:39.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.702984] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:39.712156] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:39.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.722113] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:39.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.722473] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=115] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:38.769694] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EECF-0-0] [lt=8] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:06:39.727895] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:39.727944] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=61] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:39.728212] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=261] temporary file module use no blocks [2025-08-14 09:06:39.728224] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:39.728247] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=21] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:39.728262] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623301167515, flush_io_finished_ret_:0, flush_io_finished_round_:26417, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:3, normal_idle_loop_cnt_:3, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:39.730471] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:38.771078] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=142812144][errcode=0] task scheduled too delay(scheduled_delay=254436501, expected_run_ts_us_=1755162144334575, ObClockGenerator::getRealClock()=1755162398771077, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fa190, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa470, time_wheel:0x7f99341cdbf0, thread_pool:0x7f99341cd3d0, is_running:1, time_interval:2.00s, expected_run_time:09:02:24.334, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.730499] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=25] temporary file module use no blocks [2025-08-14 09:06:39.730510] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:39.730524] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:39.730517] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=957970][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:38.771|187=959.42ms|203=1us|207=14us|210=0us|222=1us|total=959.46ms) [2025-08-14 09:06:39.730541] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4389] timer task use too much time(end=1755162399730540, start=1755162398771076, delta=959464) [2025-08-14 09:06:39.730540] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=16] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623301167408, flush_io_finished_ret_:0, flush_io_finished_round_:26455, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:3, normal_idle_loop_cnt_:3, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:38.755984] INFO [CLOG] operator() (ob_log_replay_service.cpp:1678) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=92] get_replay_process success(id={id:1}, submitted_log_size=194888693, unsubmitted_log_size=0, replayed_log_size=194888693, unreplayed_log_size=0) [2025-08-14 09:06:39.730997] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=22] gc stale ls task succ [2025-08-14 09:06:39.735371] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.718, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:39.735407] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.726, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:39.735421] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a67b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7397290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.728, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:39.738351] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=33] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:39.738840] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:39.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.743316] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:39.744541] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:39.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.745710] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919d9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:39.745756] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7277c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eac90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:39.749462] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=36] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [396]OB_MALLOC COST TOO MUCH TIME, cost_time=1007294, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1007293], seq:[0] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=1007524, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=88, WASH_BLOCK_END=100, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1007330], seq:[0, 1, 2, 3, 4] [2025-08-14 09:06:37.761454] EDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=44143][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1985805, owner: ObLog, click_count: 5, time dist:[FORMAT_END=978226, ObMalloc#start=2, ObMalloc#end=1007534, ALLOC_END=38, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:37.761454] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=44143][errcode=0] epoll handler too much time :44235 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:39.750320] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1985805] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:39.751593] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=26] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:39.752427] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:39.752599] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=973180] get_min_unreplayed_log_info(lsn={lsn:7323685240}, scn={val:1755100946796590002, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873485, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:8946764, ret_code_:0}}, next_to_submit_lsn_:{lsn:7323685240}, next_to_submit_scn_:{val:1755100946796590002, v:0}, base_lsn_:{lsn:7180210176}, base_scn_:{val:1753840803166611000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:7323685240}, end_lsn:{lsn:7323685240}, read_buf:{buf_len_:3747840, buf_:0x7f98f7a05000}, block_size:67104768, log_storage_:0x7f9901131bf0, log_storage_:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:28, read_io_size:98508800, read_disk_cost_ts:645339, start_lsn:{lsn:7180210176}}}}}}) [2025-08-14 09:06:39.752694] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1085) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=94] replay status is not follower(min_unreplayed_lsn={lsn:7323685240}, base_lsn={lsn:7180210176}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161740873485, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:8946764, ret_code_:0}}, next_to_submit_lsn_:{lsn:7323685240}, next_to_submit_scn_:{val:1755100946796590002, v:0}, base_lsn_:{lsn:7180210176}, base_scn_:{val:1753840803166611000, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7a05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:7323685240}, end_lsn:{lsn:7323685240}, read_buf:{buf_len_:3747840, buf_:0x7f98f7a05000}, block_size:67104768, log_storage_:0x7f9901131bf0, log_storage_:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:28, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:28, read_io_size:98508800, read_disk_cost_ts:645339, start_lsn:{lsn:7180210176}}}}}}) [2025-08-14 09:06:39.752756] INFO [CLOG] operator() (ob_log_replay_service.cpp:1678) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=61] get_replay_process success(id={id:1001}, submitted_log_size=143475064, unsubmitted_log_size=0, replayed_log_size=143475064, unreplayed_log_size=0) [2025-08-14 09:06:39.754263] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:39.754302] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:39.755783] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:39.755823] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:39.756919] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:39.757369] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:39.759800] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:39.750334] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:39.751877] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:39.762298] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10404][errcode=0] epoll handler too much time :10429 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:39.769246] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EECF-0-0] [lt=953353][errcode=0] cost too much time:mds_tenant_service.cpp:void oceanbase::storage::mds::ObTenantMdsTimer::dump_special_mds_table_status_task(), (this=|threshold=1.00s|start at 09:06:37.763|total=2.01s) [2025-08-14 09:06:39.769275] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EECF-0-0] [lt=27][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99571630c0, caller:mds_tenant_service.cpp:mtl_start:117, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_145, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571633a0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:0, time_interval:15.00s, expected_run_time:09:06:52.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.769294] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EECF-0-0] [lt=17][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:06:39.769|total=2.16s) [2025-08-14 09:06:39.769312] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EECF-0-0] [lt=11][errcode=-4016] task executed too delay(delay_time=2157173, *p_task_={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:0, time_interval:3.00s, expected_run_time:09:06:40.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.772614] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:39.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.780482] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:39.781439] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:39.781472] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:39.781492] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:39.781509] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:39.781527] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:39.781542] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:39.781562] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:39.781580] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:39.781600] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:39.781617] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:39.781633] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:39.781653] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:39.781669] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:39.781683] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:39.781694] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:39.781708] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:39.781733] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:39.781749] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:39.781763] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:39.781777] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:39.781953] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:39.781967] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:39.782711] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:39.783142] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:39.784707] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=16] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:39.799318] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=14] gc stale ls task succ [2025-08-14 09:06:39.802800] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:39.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.803048] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:39.822126] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:39.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.838833] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:39.838891] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:39.838907] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:39.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.838933] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:39.838973] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:39.839008] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:39.839020] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:39.839032] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:39.869664] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=21] PNIO [ratelimit] time: 1755162399869661, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007618, add_bytes: 0 [2025-08-14 09:06:39.880393] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:39.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.903099] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:39.903331] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc4b98, timer:0x7f9970f40f80, task:0x564d6d07c068, task_type:"N9oceanbase8observer16ObTenantDutyTaskE", scheduled_time:1755161617719061, pushed_time:0, delay:10000000}, running_task_token={this:0x7f9970fc5fb8, timer:0x7f9970f40f80, task:0x564d727182f0, task_type:"N9oceanbase5share16ObAshRefreshTaskE", scheduled_time:1755161611717613, pushed_time:1755161614757742, delay:120000000}, total_delay=782184269, threashold=1000000, tenant_id=500) [2025-08-14 09:06:39.922195] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:39.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.975262] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=17] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:40.003158] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:40.022259] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:39.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:40.024033] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:40.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:40.080517] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:39.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:40.103226] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:40.122323] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:40.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:40.203293] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:40.222388] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:40.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:40.275477] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:40.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:40.280644] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:40.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:40.282151] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:40.282175] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:40.282198] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:40.282220] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:40.282240] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:40.282249] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:40.282269] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:40.282278] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:40.282296] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:40.282310] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:40.282321] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:40.282335] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:40.282345] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:40.282357] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:40.303322] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:40.305675] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:40.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:40.322449] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:40.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:40.368155] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=25] gc stale ls task succ [2025-08-14 09:06:40.403372] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:40.422512] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:40.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:40.431363] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fa190, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa470, time_wheel:0x7f99341cdbf0, thread_pool:0x7f99341cd3d0, is_running:1, time_interval:2.00s, expected_run_time:09:06:40.334, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:40.503434] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:40.522571] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:40.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:40.526903] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:40.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:40.603496] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:40.622634] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:40.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:40.703553] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:40.721658] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:40.723044] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:40.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:40.729365] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:40.729429] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:40.729466] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28] temporary file module use no blocks [2025-08-14 09:06:40.729474] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:40.729486] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:40.729500] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623302168059, flush_io_finished_ret_:0, flush_io_finished_round_:26418, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:40.730631] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=22] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:40.730657] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=22] temporary file module use no blocks [2025-08-14 09:06:40.730662] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=5] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:40.730671] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:40.730689] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=17] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623302170332, flush_io_finished_ret_:0, flush_io_finished_round_:26456, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:40.527693] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 19.783072 loop:26707, slowcnt: 194 [2025-08-14 09:06:39.938968] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:39.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:40.731111] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=790944][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:39.938|207=792.14ms|210=0us|222=2us|total=792.16ms) [2025-08-14 09:06:40.731135] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4389] timer task use too much time(end=1755162400731134, start=1755162399938965, delta=792169) [395]OB_MALLOC COST TOO MUCH TIME, cost_time=966553, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=966546], seq:[0] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=964030, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=17, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=963992], seq:[0, 1, 2, 3, 4] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=964041, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=964040], seq:[0, 1] [2025-08-14 09:06:40.403625] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc50d8, timer:0x7f9970f40f80, task:0x564d6d07c190, task_type:"N9oceanbase8observer8ObServer24ObRefreshCpuFreqTimeTaskE", scheduled_time:1755161617737100, pushed_time:0, delay:10000000}, running_task_token={this:0x7f9970fc5fb8, timer:0x7f9970f40f80, task:0x564d727182f0, task_type:"N9oceanbase5share16ObAshRefreshTaskE", scheduled_time:1755161611717613, pushed_time:1755161614757742, delay:120000000}, total_delay=782666524, threashold=1000000, tenant_id=500) [2025-08-14 09:06:40.282368] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:40.731351] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=448300][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:39.765840] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7277230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731370] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:40.731384] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:40.731379] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=964077][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:39.765|207=965.54ms|210=0us|222=1us|total=965.57ms) [2025-08-14 09:06:40.731413] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4389] timer task use too much time(end=1755162400731413, start=1755162399765838, delta=965575) [2025-08-14 09:06:40.731426] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=973693, expected_run_ts_us_=1755162399757731, ObClockGenerator::getRealClock()=1755162400731424, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7272330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a5090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.480755] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:40.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:40.731444] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7272330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a5090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731459] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=975539, expected_run_ts_us_=1755162399755919, ObClockGenerator::getRealClock()=1755162400731458, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919822b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731448] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=250311][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:40.480|207=250.69ms|210=0us|222=2us|total=250.71ms) [2025-08-14 09:06:40.731473] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4389] timer task use too much time(end=1755162400731473, start=1755162400480753, delta=250720) [2025-08-14 09:06:40.731472] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919822b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731487] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=974875, expected_run_ts_us_=1755162399756611, ObClockGenerator::getRealClock()=1755162400731486, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731499] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731513] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=974604, expected_run_ts_us_=1755162399756908, ObClockGenerator::getRealClock()=1755162400731513, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d71b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731525] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d71b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731538] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=973814, expected_run_ts_us_=1755162399757723, ObClockGenerator::getRealClock()=1755162400731538, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731550] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731563] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=974213, expected_run_ts_us_=1755162399757349, ObClockGenerator::getRealClock()=1755162400731563, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a27b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731575] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a27b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731596] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] task scheduled too delay(scheduled_delay=975781, expected_run_ts_us_=1755162399755814, ObClockGenerator::getRealClock()=1755162400731596, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a40b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731608] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a40b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731621] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=979594, expected_run_ts_us_=1755162399752026, ObClockGenerator::getRealClock()=1755162400731621, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7faf30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731633] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7faf30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731646] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=971048, expected_run_ts_us_=1755162399760597, ObClockGenerator::getRealClock()=1755162400731646, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7352330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196fc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731658] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7352330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196fc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731670] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=959912, expected_run_ts_us_=1755162399771758, ObClockGenerator::getRealClock()=1755162400731670, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aca90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731682] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aca90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731695] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=952063, expected_run_ts_us_=1755162399779631, ObClockGenerator::getRealClock()=1755162400731695, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f734fc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891918610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731715] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f734fc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891918610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731734] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] task scheduled too delay(scheduled_delay=946790, expected_run_ts_us_=1755162399784943, ObClockGenerator::getRealClock()=1755162400731734, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731746] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731758] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=946187, expected_run_ts_us_=1755162399785571, ObClockGenerator::getRealClock()=1755162400731758, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731778] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731797] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=937044, expected_run_ts_us_=1755162399794746, ObClockGenerator::getRealClock()=1755162400731797, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.794, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731809] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.794, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731822] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=927286, expected_run_ts_us_=1755162399804536, ObClockGenerator::getRealClock()=1755162400731822, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731834] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731847] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=924787, expected_run_ts_us_=1755162399807059, ObClockGenerator::getRealClock()=1755162400731847, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731859] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731871] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=924715, expected_run_ts_us_=1755162399807156, ObClockGenerator::getRealClock()=1755162400731871, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731883] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731945] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=60][errcode=0] task scheduled too delay(scheduled_delay=930791, expected_run_ts_us_=1755162399801154, ObClockGenerator::getRealClock()=1755162400731945, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7397b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731966] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7397b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731979] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=930259, expected_run_ts_us_=1755162399801719, ObClockGenerator::getRealClock()=1755162400731978, ret=0, ret="OB_SUCCESS", *this={this:0x7f98918299b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891916210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.731991] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918299b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891916210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.732004] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=906143, expected_run_ts_us_=1755162399825860, ObClockGenerator::getRealClock()=1755162400732004, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.825, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.732016] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.825, task_priority:1, with_handle_protected:0}) [397]OB_MALLOC COST TOO MUCH TIME, cost_time=965014, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=964983, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [864]OB_MALLOC COST TOO MUCH TIME, cost_time=963057, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=16, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=15, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=963010, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5] [2025-08-14 09:06:39.769327] WDIAG [OCCAM] dump_statistics (ob_vtable_event_recycle_buffer.h:144) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EED0-0-0] [lt=14][errcode=-4006] not init(ret=-4006) [2025-08-14 09:06:40.731036] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=17] gc stale ls task succ [2025-08-14 09:06:40.731409] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:40.732813] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=1393][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:40.732971] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:40.732985] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:40.734119] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=905954, expected_run_ts_us_=1755162399826071, ObClockGenerator::getRealClock()=1755162400734118, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.826, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.734225] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=104][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.826, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.735550] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1318][errcode=0] task scheduled too delay(scheduled_delay=914776, expected_run_ts_us_=1755162399820773, ObClockGenerator::getRealClock()=1755162400735550, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.820, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.735568] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.820, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.735582] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=907243, expected_run_ts_us_=1755162399828338, ObClockGenerator::getRealClock()=1755162400735582, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72879b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.828, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.735594] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72879b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.828, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.735605] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=889195, expected_run_ts_us_=1755162399846410, ObClockGenerator::getRealClock()=1755162400735605, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.846, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.735647] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:39.846, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.735691] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72760b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.719, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.738481] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=27] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:39.761109] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:174) [807][T1002_TimerWK3_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=13] dump tenant replay process(tenant_id=1002, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=322, replayed_log_size(MB)=322, last_submitted_log_size(MB)=322, last_replayed_log_size(MB)=322, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [395]OB_MALLOC COST TOO MUCH TIME, cost_time=974489, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=966574], seq:[0, 1] [2025-08-14 09:06:39.763289] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:39.752467] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:213) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-08-14 09:06:40.739344] INFO [COMMON] handle (memory_dump.cpp:544) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=985383] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-08-14 09:06:40.739293] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=974527][errcode=0] epoll handler too much time :976012 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [772]OB_MALLOC COST TOO MUCH TIME, cost_time=975194, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=975193], seq:[0] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=975212, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=975207], seq:[0, 1] [2025-08-14 09:06:39.763689] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:39.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:40.740414] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=975246][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:39.763|207=976.72ms|210=0us|222=1us|total=976.74ms) [2025-08-14 09:06:40.740435] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task use too much time(end=1755162400740434, start=1755162399763687, delta=976747) [2025-08-14 09:06:40.740446] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=981311, expected_run_ts_us_=1755162399759134, ObClockGenerator::getRealClock()=1755162400740446, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:39.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:40.740462] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:39.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:40.740479] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=651163, expected_run_ts_us_=1755162400089316, ObClockGenerator::getRealClock()=1755162400740479, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:40.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:40.740493] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:40.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:40.740507] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=536209, expected_run_ts_us_=1755162400204297, ObClockGenerator::getRealClock()=1755162400740506, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:40.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:40.740518] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:40.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:40.740835] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] [COMPACTION DAG STATUS] (start_time=1755162387760940, end_time=1755162400740833, dag_status={}) [2025-08-14 09:06:40.740889] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=51] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:40.740913] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=23] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:40.740924] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:40.740943] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=18] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:40.740952] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:40.740961] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:40.740972] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:40.740983] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:40.740991] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:40.741347] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] [COMPACTION DAG STATUS] (start_time=1755162386767774, end_time=1755162400741346, dag_status={}) [2025-08-14 09:06:40.741365] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:40.741388] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=23] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:40.741405] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:40.741713] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.741741] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.741761] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739be30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.741774] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.630931] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:1, time_interval:3.00s, expected_run_time:09:06:40.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:40.742934] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=111831][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:40.630|207=112.00ms|210=0us|222=1us|total=112.01ms) [2025-08-14 09:06:40.742948] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task use too much time(end=1755162400742947, start=1755162400630928, delta=112019) [2025-08-14 09:06:39.760217] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:39.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:40.743000] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=981297][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:39.760|207=982.78ms|210=0us|222=1us|total=982.79ms) [2025-08-14 09:06:40.743011] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task use too much time(end=1755162400743011, start=1755162399760215, delta=982796) [2025-08-14 09:06:40.743023] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=759491, expected_run_ts_us_=1755162399983531, ObClockGenerator::getRealClock()=1755162400743023, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:39.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:40.743036] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:39.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:39.760481] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10129][errcode=0] epoll handler too much time :10147 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:40.743255] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:40.750112] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:40.751179] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=981137] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:40.751768] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7edc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.751808] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ebe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.751831] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:40.753086] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:40.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:40.754255] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:40.756019] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:40.756875] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:40.756915] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=18] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:40.756944] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:40.756971] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:40.757404] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:40.757646] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:38.772257] EDIAG [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1984247, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1984237, ObMalloc#start=1, ObMalloc#end=5, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x12377491 0x1236e8e9 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [864]OB_MALLOC COST TOO MUCH TIME, cost_time=989874, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=963069, WASH_BLOCK_END=26805], seq:[0, 1, 2] [2025-08-14 09:06:40.743105] INFO [MDS] dump_map_holding_item (mds_tenant_service.cpp:462) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EED0-0-0] [lt=961385] finish scan map holding items(scan_cnt=0) [2025-08-14 09:06:38.772257] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=8] finish to resize log external storage handler(current_ts=1755162397771947, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-08-14 09:06:40.743115] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=976627][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a49d8, timer:0x7f9922d47110, task:0x7f9922eb44d0, task_type:"N9oceanbase10logservice17ReplayProcessStatE", scheduled_time:1755162387768594, pushed_time:1755162387770884, delay:10000000}, delay_in_thread_pool=4994306, threashold=500000) [2025-08-14 09:06:37.721027] EDIAG [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=14709][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3053798, owner: ObLog, click_count: 7, time dist:[FORMAT_END=3006919, ObMalloc#start=1, ObMalloc#end=46877, ALLOC_END=1, APPEND_END=0, ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=1], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x99721a1 0x997a72e 0x2006ed17 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [397]OB_MALLOC COST TOO MUCH TIME, cost_time=1012392, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=965030, WASH_BLOCK_END=30603, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=16759], seq:[0, 1, 2, 3, 4] [2025-08-14 09:06:39.766115] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1012597, owner: ObLog, click_count: 6, time dist:[FORMAT_END=7, ObMalloc#start=0, ObMalloc#end=1012406, ObMalloc#start=19, ObMalloc#end=44, FREE_END=121], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [374]OB_MALLOC COST TOO MUCH TIME, cost_time=961957, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=17474], seq:[0, 1] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=982076, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=36287], seq:[0, 1] [982]OB_MALLOC COST TOO MUCH TIME, cost_time=962111, tenant_id=500, label=SqlCompile, ctx_id=0, prio=0, size=776, [2025-08-14 09:06:39.875792] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=703390][errcode=-4018] LOGGER COST TOO MUCH TIME, cost: 1861128, owner: ObLog, click_count: 5, time dist:[FORMAT_END=888074, ObMalloc#start=1, ObMalloc#end=14979, ALLOC_END=958074, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3b4b 0x113d34aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:39.868568] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=696505][errcode=-4018] LOGGER COST TOO MUCH TIME, cost: 1868392, owner: ObLog, click_count: 8, time dist:[FORMAT_END=895293, ObMalloc#start=1, ObMalloc#end=15044, ObMalloc#start=956679, ObMalloc#end=53, ObMalloc#start=1168, ObMalloc#end=61, FREE_END=93], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3b4b 0x113d34aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:18.753419] EDIAG [ELECT] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=4020338][errcode=0] LOGGER COST TOO MUCH TIME, cost: 142783765, owner: ObLog, click_count: 5, time dist:[FORMAT_END=35, ObMalloc#start=2, ObMalloc#end=88, ALLOC_END=142783639, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ea7c4 0x9818f9c 0xf9d17c6 0xf9d4150 0xfa09ca2 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [864]OB_MALLOC COST TOO MUCH TIME, cost_time=993166, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=993157], seq:[0] [864]OB_MALLOC COST TOO MUCH TIME, cost_time=993190, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=993180], seq:[0, 1] [2025-08-14 09:06:39.768194] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1983129, owner: ObLog, click_count: 7, time dist:[FORMAT_END=13, ObMalloc#start=1, ObMalloc#end=989886, ObMalloc#start=28, ObMalloc#end=993198, ALLOC_END=3, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:40.760477] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=84] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:40.739244] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:233) [529][T1_RFLWorker][T1][YB42AC4D0007-00063C495B199C16-0-0] [lt=11333] ObRemoteFetchWorker is running(thread_index=0) [2025-08-14 09:06:40.763259] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [677][T1001_Occam][T1001][YB42AC4D0007-00063C4967999C34-0-0] [lt=12] ====== tenant freeze timer task ====== [2025-08-14 09:06:37.721027] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=14709] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:40.779477] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=3053798] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:40.779486] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:40.779492] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:40.779497] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:40.779503] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:40.779508] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=4] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:40.779513] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:40.779517] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=4] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:40.780593] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:40.877277] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=25] PNIO [ratelimit] time: 1755162400877275, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007614, add_bytes: 0 [2025-08-14 09:06:40.745289] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=21534] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:06:40.803613] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:06:40.779525] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=4] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:06:41.724585] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=919577][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:40.763346] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:06:41.724600] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:40.990263] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:40.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:41.724631] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=733252][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:40.990|207=734.36ms|210=0us|222=2us|total=734.39ms) [2025-08-14 09:06:41.724658] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4389] timer task use too much time(end=1755162401724657, start=1755162400990261, delta=734396) [2025-08-14 09:06:40.761843] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.750, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.725306] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=962005][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:40.761|207=963.46ms|210=0us|222=2us|total=963.48ms) [2025-08-14 09:06:41.725323] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] timer task use too much time(end=1755162401725322, start=1755162400761840, delta=963482) [2025-08-14 09:06:41.725332] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] task scheduled too delay(scheduled_delay=970109, expected_run_ts_us_=1755162400755221, ObClockGenerator::getRealClock()=1755162401725331, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.725344] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.726714] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=959803] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:06:41.726752] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=943643] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:06:41.728363] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:41.735910] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=324][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:41.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:41.739462] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=26] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:39.875792] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=703390][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:06:41.743340] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:40.743278] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=14277][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1011756, owner: ObLog, click_count: 3, time dist:[FORMAT_END=980120, ALLOC_END=31636, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:41.753373] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:41.753398] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=14] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:41.753420] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:41.753449] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:41.722318] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:39.768194] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=25] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:06:41.754405] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:41.724675] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=480040, expected_run_ts_us_=1755162401244633, ObClockGenerator::getRealClock()=1755162401724673, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:41.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:41.754424] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=29702][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:41.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:41.753669] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=282][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:41.754453] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=778][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:41.754453] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:41.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:41.754466] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:41.754475] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:41.754485] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:41.754504] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:41.754515] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:41.754674] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:41.754690] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:41.756267] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:41.739442] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [411][sql_nio4][T0][Y0-0000000000000000-0-0] [lt=1935642] [sql nio session](*s={this:0x7f98f72c3030, session_id:3221590336, trace_id:YB42AC4D0007-00063C49634A0941-0-0, sql_handling_stage:3, sql_initiative_shutdown:false, reader:{fd:255}, err:5, last_decode_time:1755162354747599, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1422, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:06:39.752401] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1001, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:8571658, max_lsn:7323687369, max_scn:{val:1755161749741193002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:7278610047}, reuse_lsn:{lsn:7323686131}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:7281744329}}, last_submit_lsn:{lsn:7323686131}, last_submit_end_lsn:{lsn:7323687369}, last_submit_log_id:8571658, last_submit_log_pid:49, max_flushed_lsn:{lsn:7323686009}, max_flushed_end_lsn:{lsn:7323686131}, max_flushed_log_pid:49, committed_end_lsn:{lsn:7323686009}, last_slide_log_id:8571656, last_slide_scn:{val:1755161747742105001, v:0}, last_slide_lsn:{lsn:7323685887}, last_slide_end_lsn:{lsn:7323686009}, last_slide_log_pid:49, last_slide_log_accum_checksum:2505458373, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f990112a530}, [StateMgr]={this:0x7f990112e9b0, self:"172.77.0.7:2882", palf_id:1001, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"PENDING", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:49}, leader:"0.0.0.0:0", leader_epoch:-1, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:7323687369}, scan_disk_log_finished:true, last_check_start_id:8571657, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161767744317, palf_role_change_cb:0x7f9901135470, allow_vote:true}, [ConfigMgr]={palf_id:1001, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:49, prev:{version:1, config:{config_version:{proposal_id:48, config_seq:47}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:49, config_seq:48}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_mode_pid:49}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_end_lsn:{lsn:7323685240}, prev_mode_pid:49}, reconfig_barrier:{prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_end_lsn:{lsn:7323685240}, prev_mode_pid:49}, persistent_config_version:{proposal_id:49, config_seq:48}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755161739755312, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755161739755312, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f990112bc70}, [ModeMgr]={palf_id:1001, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, accepted_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1001, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1753069671378640001, v:0}, base_lsn_for_block_gc:{lsn:7247314944}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:49}, log_config_meta:{version:3, proposal_id:49, prev:{version:1, config:{config_version:{proposal_id:48, config_seq:47}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:49, config_seq:48}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_mode_pid:49}, log_snapshot_meta:{version:2, base_lsn:{lsn:7247314944}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1753069671378640000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}}, log_meta_storage:{log_tail:{lsn:765952}, readable_log_tail:{lsn:765952}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7381524480}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/meta", dir_fd:232, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66338816, block_header_serialize_buf_:0x7f9901130bb8, flashback_version:0}, log_storage:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, palf_epoch:2, last_purge_throttling_ts:1755161769777679, this:0x7f990112f0f0}, [Reconfirm]={palf_id:1001, self:"172.77.0.7:2882", state:"WAITING_LOG_FLUSHED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:1755161769777679, this:0x7f990112eb30}) [2025-08-14 09:06:40.743278] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=14277] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:06:41.531543] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=203076][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 20.786923 loop:26707, slowcnt: 202 [2025-08-14 09:06:41.759212] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:41.762222] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:41.767841] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=360][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:41.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:41.767883] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:41.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:41.772018] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EED1-0-0] [lt=16377] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:06:41.772057] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EED1-0-0] [lt=32][errcode=-4389] run this task use too much time(*p_task_={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:0, time_interval:3.00s, expected_run_time:09:06:43.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:41.772092] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EED1-0-0] [lt=25][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:06:41.772|total=2.00s) [2025-08-14 09:06:41.752513] INFO [STORAGE] need_re_freeze (ob_tx_data_table.cpp:1349) [677][T1001_Occam][T1001][YB42AC4D0007-00063C4967999C34-0-0] [lt=16036] tx data table need refreeze(ls_id={id:1}, max_freeze_interval=300000000, last_request_ts="2025-08-14 08:59:36.758344", last_freeze_ts="2025-08-14 08:59:36.758344", current_time="2025-08-14 09:06:41.725090") [2025-08-14 09:06:39.752401] EDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2001100, owner: ObLog, click_count: 11, time dist:[FORMAT_END=1998125, ObMalloc#start=1, ObMalloc#end=2948, ALLOC_END=23, APPEND_END=1, ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=68, ALLOC_BLOCK#start=11656, ALLOC_BLOCK#end=75, ALLOC_BLOCK#start=10917, ALLOC_BLOCK#end=6], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10], BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0xfc0e73d 0x993f560 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:41.781359] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=35896] blacklist_loop exec finished(cost_time=15, is_enabled=true, send_cnt=0) [2025-08-14 09:06:41.781456] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:41.781609] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=11] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:41.782438] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=202] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:41.782513] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=352][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:41.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:41.754105] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=31737][errcode=0] epoll handler too much time :31805 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:41.786339] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=223] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:41.787472] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=159] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:41.725354] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=967750, expected_run_ts_us_=1755162400757603, ObClockGenerator::getRealClock()=1755162401725353, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728d1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7291110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.787600] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=62149][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728d1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7291110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.787631] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] task scheduled too delay(scheduled_delay=1018737, expected_run_ts_us_=1755162400768893, ObClockGenerator::getRealClock()=1755162401787631, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aa990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.787649] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aa990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.787662] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1023389, expected_run_ts_us_=1755162400764272, ObClockGenerator::getRealClock()=1755162401787661, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.787673] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.787685] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1023103, expected_run_ts_us_=1755162400764582, ObClockGenerator::getRealClock()=1755162401787685, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919afe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.787697] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919afe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.787710] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=1019625, expected_run_ts_us_=1755162400768085, ObClockGenerator::getRealClock()=1755162401787710, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d62b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.787723] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d62b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.787735] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1019572, expected_run_ts_us_=1755162400768162, ObClockGenerator::getRealClock()=1755162401787735, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.787747] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.787762] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=1015646, expected_run_ts_us_=1755162400772115, ObClockGenerator::getRealClock()=1755162401787761, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.787773] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.787785] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1012642, expected_run_ts_us_=1755162400775143, ObClockGenerator::getRealClock()=1755162401787785, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.787796] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.787808] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1017579, expected_run_ts_us_=1755162400770229, ObClockGenerator::getRealClock()=1755162401787808, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.787821] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.787833] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1016086, expected_run_ts_us_=1755162400771747, ObClockGenerator::getRealClock()=1755162401787833, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7397a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.787844] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7397a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.787856] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1003133, expected_run_ts_us_=1755162400784723, ObClockGenerator::getRealClock()=1755162401787856, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.787875] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.787890] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=1006828, expected_run_ts_us_=1755162400781061, ObClockGenerator::getRealClock()=1755162401787889, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.787902] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.787913] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1005255, expected_run_ts_us_=1755162400782658, ObClockGenerator::getRealClock()=1755162401787913, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.787925] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.787936] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=991113, expected_run_ts_us_=1755162400796823, ObClockGenerator::getRealClock()=1755162401787936, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.787950] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.787962] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=985744, expected_run_ts_us_=1755162400802217, ObClockGenerator::getRealClock()=1755162401787962, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.802, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.787979] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.802, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.787992] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=973661, expected_run_ts_us_=1755162400814330, ObClockGenerator::getRealClock()=1755162401787991, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.814, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.788003] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.814, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.788016] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=970196, expected_run_ts_us_=1755162400817820, ObClockGenerator::getRealClock()=1755162401788016, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.817, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.788028] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.817, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.788039] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=969066, expected_run_ts_us_=1755162400818973, ObClockGenerator::getRealClock()=1755162401788039, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.788051] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.788063] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=968944, expected_run_ts_us_=1755162400819118, ObClockGenerator::getRealClock()=1755162401788062, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.819, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.788076] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.819, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.788088] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=974595, expected_run_ts_us_=1755162400813492, ObClockGenerator::getRealClock()=1755162401788087, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.813, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.788099] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.813, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.788111] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=959275, expected_run_ts_us_=1755162400828835, ObClockGenerator::getRealClock()=1755162401788111, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919831b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.828, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.788123] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919831b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.828, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.788136] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=953097, expected_run_ts_us_=1755162400835039, ObClockGenerator::getRealClock()=1755162401788136, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.835, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.788147] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.835, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.788159] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=940472, expected_run_ts_us_=1755162400847687, ObClockGenerator::getRealClock()=1755162401788159, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.847, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.788170] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.847, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.788182] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=943434, expected_run_ts_us_=1755162400844747, ObClockGenerator::getRealClock()=1755162401788182, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.844, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.788195] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.844, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.788220] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] task scheduled too delay(scheduled_delay=943245, expected_run_ts_us_=1755162400844974, ObClockGenerator::getRealClock()=1755162401788220, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.844, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.788239] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:40.844, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.788263] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=172914, expected_run_ts_us_=1755162401615348, ObClockGenerator::getRealClock()=1755162401788262, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ebbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:41.615, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.788274] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ebbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:41.615, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.788302] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919abd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:41.721, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.788321] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919abb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:41.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.788340] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:41.726, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.788352] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fdc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:41.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.788364] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:41.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.788386] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728be10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:41.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.788399] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72847b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:41.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.788411] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:41.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.788432] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891806f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:41.751, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.788444] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:41.750, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.788457] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73539b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7396b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:41.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.789058] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=998107][errcode=-4015] clock out of order(cur_ts=1755162401789056, cur_ts_=1755162401725090, delta=63966) [2025-08-14 09:06:41.789093] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=215] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:06:41.796108] INFO handle (memory_dump.cpp:671) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=16889] statistics: tenant_cnt: 6, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 168 270 0 1 2 110 1 0 1 3 7 0 0 1 4 1 1 0 1 5 85 6 0 1 6 13 0 0 1 7 22 2 0 1 8 53 1 0 1 12 17 3 0 1 13 10 1 0 1 14 1 1 0 1 15 1 1 0 1 16 8 7 0 1 18 4 3 0 1 20 2 1 0 1 22 1 1 0 1 24 1 0 0 1 25 6 0 0 500 0 562 274 0 500 7 190 5 0 500 8 41 1 0 500 9 2 1 0 500 10 4 2 0 500 16 57 4 0 500 17 60 7 0 500 22 38 64 0 500 23 27 10 0 508 0 2 7 0 508 8 5 1 0 509 0 2 7 0 509 8 5 1 0 1001 0 147 252 0 1001 2 26 1 0 1001 3 5 0 0 1001 4 1 1 0 1001 5 85 6 0 1001 6 11 0 0 1001 7 30 2 0 1001 8 36 1 0 1001 12 17 3 0 1001 13 3 1 0 1001 14 1 1 0 1001 16 7 4 0 1001 18 4 3 0 1001 20 4 1 0 1001 25 8 0 0 1002 0 263 278 0 1002 2 198 1 0 1002 3 4 0 0 1002 4 1 1 0 1002 5 101 6 0 1002 6 14 0 0 1002 7 6 2 0 1002 8 73 1 0 1002 12 18 3 0 1002 13 7 1 0 1002 14 1 2 0 1002 16 10 7 0 1002 18 9 3 0 1002 20 2 1 0 1002 21 1 0 0 1002 24 1 0 0 1002 25 7 0 0 cost_time: 1052015 [2025-08-14 09:06:41.796150] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=21] [MEMORY] tenant: 1, limit: 2147483648 hold: 1013174272 cache_hold: 56623104 cache_used: 56623104 cache_item_count: 27 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 444993536 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 230686720 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 1163264 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 483328 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 51617792 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 1458176 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 14204928 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 111149056 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35651584 limit= 429496720 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 8003584 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= PS_CACHE_CTX_ID hold_bytes= 65536 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 14827520 limit= 9223372036854775807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 3481600 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3964928 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 212992 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 884736 limit= 9223372036854775807 [2025-08-14 09:06:41.799448] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=937874] gc stale ls task succ [2025-08-14 09:06:41.802674] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:41.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:41.807760] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:41.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.817823] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d22b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eaf90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:41.808, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.823639] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=485][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:41.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:41.827915] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:41.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.827950] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:41.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.835989] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=79][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:41.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:41.841695] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=743][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:41.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:41.848041] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:41.831, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.848083] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:41.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.852912] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=206] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:41.852981] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=31] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:41.853023] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:41.853078] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:41.853126] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:41.853144] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:41.853160] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:41.858106] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ac10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:41.849, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.868170] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:41.852, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.878213] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:41.861, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.884897] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=14] PNIO [ratelimit] time: 1755162401884894, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007619, add_bytes: 0 [2025-08-14 09:06:41.908361] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919859b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:41.896, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:41.923702] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:41.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:41.936056] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:41.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:41.941750] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:41.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:41.980748] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=233] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:42.014070] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:42.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.014105] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:42.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.023791] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:41.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.024150] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:42.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.036110] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:41.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.041793] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:41.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.094624] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:42.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.123837] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:42.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.136159] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:42.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.141851] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:42.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.215440] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:42.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.223860] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:42.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.236227] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:42.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.241911] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:42.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.254836] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:42.254868] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:42.254887] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:42.254907] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:42.254922] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:42.254935] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:42.254954] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:42.254984] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:42.255004] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:42.255013] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:42.255027] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:42.255043] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:42.255054] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:42.255065] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:42.255095] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:42.255106] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:42.255118] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:42.255131] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:42.255155] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:42.255169] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:42.265737] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:42.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.265769] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:42.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.275752] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:42.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.305964] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:42.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.323894] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:42.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.336283] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:42.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.368252] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=369907] gc stale ls task succ [2025-08-14 09:06:42.423961] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:42.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.436326] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:42.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.436415] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fa190, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa470, time_wheel:0x7f99341cdbf0, thread_pool:0x7f99341cd3d0, is_running:1, time_interval:2.00s, expected_run_time:09:06:42.334, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.517177] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:42.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.517221] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:42.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.524019] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:42.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.527256] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:42.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.536369] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:42.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.597693] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:42.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.624039] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:42.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.636406] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:42.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.708960] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=983213] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25861124096, fast_delete_log_mode=false) [2025-08-14 09:06:42.718535] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:42.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.721778] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28470] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:42.724100] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:42.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.727600] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] [COMPACTION DAG STATUS] (start_time=1755162392726828, end_time=1755162402727598, dag_status={}) [2025-08-14 09:06:42.728346] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:42.729639] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=33] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:42.729673] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=25] temporary file module use no blocks [2025-08-14 09:06:42.729685] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:42.729701] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:42.729718] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=17] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623304169337, flush_io_finished_ret_:0, flush_io_finished_round_:26420, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:18.870707] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:288) [570][T1_ArcFetcher][T1][YB42AC4D0007-00063C495B999C16-0-0] [lt=17792][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2865138, owner: ObLog, click_count: 6, time dist:[FORMAT_END=2864911, ObMalloc#start=1, ObMalloc#end=30, ObMalloc#start=18, ObMalloc#end=22, FREE_END=156], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113b2889 0x979b621 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:20.810476] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [579][T1_MultiVersion][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1954677, owner: ObLog, click_count: 8, time dist:[FORMAT_END=1952862, ObMalloc#start=1, ObMalloc#end=85, ObMalloc#start=21, ObMalloc#end=50, ObMalloc#start=1575, ObMalloc#end=37, FREE_END=46], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0xf9fb357 0x97ae92e 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:42.341981] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:42.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:41.994003] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:41.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.731815] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=389239][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:42.341|207=389.83ms|210=0us|222=1us|total=389.86ms) [2025-08-14 09:06:42.731847] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4389] timer task use too much time(end=1755162402731846, start=1755162402341979, delta=389867) [2025-08-14 09:06:42.557018] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=224712][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 21.812396 loop:26707, slowcnt: 210 [2025-08-14 09:06:42.731848] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=736728][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:41.994|207=737.84ms|210=0us|222=2us|total=737.88ms) [2025-08-14 09:06:42.731891] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=43][errcode=-4389] timer task use too much time(end=1755162402731891, start=1755162401994000, delta=737891) [2025-08-14 09:06:42.731905] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=487270, expected_run_ts_us_=1755162402244633, ObClockGenerator::getRealClock()=1755162402731904, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:42.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.731919] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:42.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.735524] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=909481] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:06:42.735546] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:42.735559] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:42.731169] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=9030] gc stale ls task succ [2025-08-14 09:06:42.736545] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=33] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:42.736582] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=32] temporary file module use no blocks [2025-08-14 09:06:42.736591] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:42.736603] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:42.736615] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623304175642, flush_io_finished_ret_:0, flush_io_finished_round_:26458, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:42.736791] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:42.724, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:42.255369] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:42.737302] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=481188][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:42.739334] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=16] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 444993536 used= 389804064 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 70 washed_blocks= 287 washed_size= 70221824 [MEMORY] request_cached_chunk_cnt= 6 [MEMORY] hold= 389804064 used= 379547427 count= 32900 avg_used= 11536 mod=SUMMARY [2025-08-14 09:06:42.739366] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=15] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 230686720 used= 228884480 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 block_cnt= 110 chunk_cnt= 110 mod=Memstore [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:06:42.739381] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1 ctx_id= EXECUTE_CTX_ID hold= 1163264 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 15 washed_size= 13516800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:42.739396] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:06:42.739411] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 51617792 used= 47452416 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 84 washed_blocks= 334 washed_size= 126640128 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 47452416 used= 43070492 count= 3992 avg_used= 10789 mod=SUMMARY [2025-08-14 09:06:42.739425] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= WORK_AREA hold= 1458176 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 13 washed_blocks= 66 washed_size= 25804800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:42.739439] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 14204928 used= 10970224 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 277 washed_size= 31932416 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10970224 used= 6078924 count= 18573 avg_used= 327 mod=SUMMARY [2025-08-14 09:06:42.739463] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=14] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 111149056 used= 108896256 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 block_cnt= 211 chunk_cnt= 53 mod=CoStack [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 mod=SUMMARY [2025-08-14 09:06:42.739483] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 56623104 used= 56623104 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 56623104 used= 0 count= 27 avg_used= 0 mod=SUMMARY [2025-08-14 09:06:42.739498] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 34365376 limit= 429496720 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 34365376 used= 31086608 count= 475 avg_used= 65445 mod=SUMMARY [2025-08-14 09:06:42.739512] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 8003584 used= 4947392 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 10 washed_blocks= 23 washed_size= 12967936 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4947392 used= 4793344 count= 604 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:42.739526] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:42.739541] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 65536 used= 31024 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 2031616 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 31024 used= 27058 count= 20 avg_used= 1352 mod=SUMMARY [2025-08-14 09:06:42.739556] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= RPC_CTX_ID hold= 14827520 used= 6135808 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 6135808 used= 4051328 count= 253 avg_used= 16013 mod=SUMMARY [2025-08-14 09:06:42.739572] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 3481600 used= 2391232 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 10 washed_size= 4907008 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2391232 used= 2367040 count= 36 avg_used= 65751 mod=SUMMARY [2025-08-14 09:06:42.739589] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 3964928 used= 3080192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 3 washed_size= 229376 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3080192 used= 2983936 count= 376 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:42.739605] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= SCHEMA_SERVICE hold= 212992 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1884160 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:42.739620] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:42.739634] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= MERGE_NORMAL_CTX_ID hold= 884736 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 6 washed_blocks= 9 washed_size= 11698176 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:42.739654] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant: 500, limit: 9223372036854775807 hold: 688734208 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 333418496 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 39239680 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 85983232 limit= 9223372036854775807 [MEMORY] ctx_id= LIBEASY hold_bytes= 3145728 limit= 9223372036854775807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 8388608 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 3063808 limit= 9223372036854775807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 37584896 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 78925824 limit= 9223372036854775807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 98983936 limit= 9223372036854775807 [2025-08-14 09:06:42.739684] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 333418496 used= 301641392 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 500 washed_blocks= 2231 washed_size= 924762112 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 301641392 used= 294211725 count= 12714 avg_used= 23140 mod=SUMMARY [2025-08-14 09:06:42.739709] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 500 ctx_id= GLIBC hold= 39239680 used= 21978864 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 183 washed_blocks= 805 washed_size= 361316352 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 21978864 used= 10372933 count= 131697 avg_used= 78 mod=SUMMARY [2025-08-14 09:06:42.739726] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 500 ctx_id= CO_STACK hold= 85983232 used= 83091456 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 83091456 used= 82936896 count= 161 avg_used= 515136 block_cnt= 161 chunk_cnt= 41 mod=CoStack [MEMORY] hold= 83091456 used= 82936896 count= 161 avg_used= 515136 mod=SUMMARY [2025-08-14 09:06:42.739743] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 500 ctx_id= LIBEASY hold= 3145728 used= 3070816 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1048576 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3070816 used= 2859784 count= 223 avg_used= 12824 mod=SUMMARY [2025-08-14 09:06:42.739758] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 500 ctx_id= LOGGER_CTX_ID hold= 6291456 used= 8323072 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8323072 used= 8318976 count= 4 avg_used= 2079744 mod=SUMMARY [2025-08-14 09:06:42.739782] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 500 ctx_id= RPC_CTX_ID hold= 3063808 used= 1482752 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 57 washed_blocks= 70 washed_size= 116473856 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1482752 used= 979328 count= 61 avg_used= 16054 mod=SUMMARY [2025-08-14 09:06:42.739796] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= PKT_NIO hold= 37584896 used= 36013568 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 50 washed_blocks= 71 washed_size= 99016704 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 36013568 used= 35017984 count= 125 avg_used= 280143 mod=SUMMARY [2025-08-14 09:06:42.739821] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= SCHEMA_SERVICE hold= 78925824 used= 68937344 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 22 washed_size= 5767168 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 68937344 used= 67674815 count= 4860 avg_used= 13924 mod=SUMMARY [2025-08-14 09:06:42.739835] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 500 ctx_id= UNEXPECTED_IN_500 hold= 98983936 used= 97174864 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 97174864 used= 86783232 count= 3014 avg_used= 28793 mod=SUMMARY [2025-08-14 09:06:42.739850] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 508, limit: 1073741824 hold: 15613952 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 5128192 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10485760 limit= 9223372036854775807 [2025-08-14 09:06:42.739860] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 508 ctx_id= DEFAULT_CTX_ID hold= 5128192 used= 3765328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3765328 used= 3723912 count= 84 avg_used= 44332 mod=SUMMARY [2025-08-14 09:06:42.739884] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 508 ctx_id= CO_STACK hold= 10485760 used= 10321920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10321920 used= 10302720 count= 20 avg_used= 515136 mod=SUMMARY [2025-08-14 09:06:42.739924] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 509, limit: 1073741824 hold: 15613952 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 5128192 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10485760 limit= 9223372036854775807 [2025-08-14 09:06:42.739935] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 509 ctx_id= DEFAULT_CTX_ID hold= 5128192 used= 3765328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3765328 used= 3723912 count= 84 avg_used= 44332 mod=SUMMARY [2025-08-14 09:06:42.739957] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 509 ctx_id= CO_STACK hold= 10485760 used= 10321920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10321920 used= 10302720 count= 20 avg_used= 515136 mod=SUMMARY [2025-08-14 09:06:42.740005] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant: 1001, limit: 1073741824 hold: 608747520 cache_hold: 44040192 cache_used: 44040192 cache_item_count: 21 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 302309376 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 54525952 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 3080192 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 483328 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 45514752 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 5521408 limit= 157286400 [MEMORY] ctx_id= GLIBC hold_bytes= 15499264 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 73424896 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35651584 limit= 214748360 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 3252224 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 14680064 limit= 1073741800 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 5570560 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3866624 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 1179648 limit= 9223372036854775807 [2025-08-14 09:06:42.740030] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= DEFAULT_CTX_ID hold= 302309376 used= 268516432 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 81 washed_blocks= 406 washed_size= 113410048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 268516432 used= 262397846 count= 3783 avg_used= 69362 mod=SUMMARY [2025-08-14 09:06:42.740046] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= MEMSTORE_CTX_ID hold= 54525952 used= 54099968 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 54099968 used= 54075216 count= 26 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:06:42.740056] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= EXECUTE_CTX_ID hold= 3080192 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 8 washed_size= 7405568 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:42.740066] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:06:42.740077] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= PLAN_CACHE_CTX_ID hold= 45514752 used= 36552320 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 77 washed_blocks= 201 washed_size= 132743168 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 36552320 used= 33009690 count= 3323 avg_used= 9933 mod=SUMMARY [2025-08-14 09:06:42.740087] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= WORK_AREA hold= 5521408 used= 0 limit= 157286400 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 9 washed_blocks= 40 washed_size= 17547264 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:42.740097] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= GLIBC hold= 15499264 used= 10889728 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 25 washed_blocks= 164 washed_size= 47415296 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10889728 used= 6020141 count= 18313 avg_used= 328 mod=SUMMARY [2025-08-14 09:06:42.740109] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= CO_STACK hold= 73424896 used= 71221248 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 32 washed_blocks= 34 washed_size= 2072576 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 71221248 used= 71088768 count= 138 avg_used= 515136 block_cnt= 138 chunk_cnt= 36 mod=CoStack [MEMORY] hold= 71221248 used= 71088768 count= 138 avg_used= 515136 mod=SUMMARY [2025-08-14 09:06:42.740124] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= KVSTORE_CACHE_ID hold= 44040192 used= 44040192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 44040192 used= 0 count= 21 avg_used= 0 mod=SUMMARY [2025-08-14 09:06:42.740135] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 33906624 limit= 214748360 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33906624 used= 30629648 count= 468 avg_used= 65447 mod=SUMMARY [2025-08-14 09:06:42.740145] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= TX_CALLBACK_CTX_ID hold= 3252224 used= 2342592 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 3 washed_blocks= 8 washed_size= 3039232 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2342592 used= 2269696 count= 286 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:42.740155] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:42.740168] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= RPC_CTX_ID hold= 14680064 used= 4202496 limit= 1073741800 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4202496 used= 2779776 count= 171 avg_used= 16256 mod=SUMMARY [2025-08-14 09:06:42.740181] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= TX_DATA_TABLE hold= 5570560 used= 2342784 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 8 washed_size= 2818048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2342784 used= 2319424 count= 30 avg_used= 77314 mod=SUMMARY [2025-08-14 09:06:42.740193] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= MDS_DATA_ID hold= 3866624 used= 2785280 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 5 washed_size= 4521984 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2785280 used= 2698240 count= 340 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:42.740220] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= MERGE_NORMAL_CTX_ID hold= 1179648 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 8 washed_blocks= 24 washed_size= 15597568 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:42.740249] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 1002, limit: 8589934592 hold: 1945841664 cache_hold: 48234496 cache_used: 48234496 cache_item_count: 23 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 1009610752 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 415236096 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 1114112 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 950272 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 211812352 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 1441792 limit= 429496725 [MEMORY] ctx_id= GLIBC hold_bytes= 12582912 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 149913600 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 37748736 limit= 1717986900 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 7766016 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 5783552 limit= 8589934500 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 5718016 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3047424 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 1032192 limit= 9223372036854775807 [2025-08-14 09:06:42.740278] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= DEFAULT_CTX_ID hold= 1009610752 used= 919141808 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 13 [MEMORY] hold= 185860096 used= 185534336 count= 114 avg_used= 1627494 block_cnt= 114 chunk_cnt= 104 mod=MysqlRequesReco [MEMORY] hold= 167792640 used= 167772192 count= 1 avg_used= 167772192 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 83927040 used= 83886080 count= 2 avg_used= 41943040 block_cnt= 2 chunk_cnt= 2 mod=LogGroupBuffer [MEMORY] hold= 67190784 used= 67109344 count= 4 avg_used= 16777336 block_cnt= 4 chunk_cnt= 4 mod=MrgMemPoolChk [MEMORY] hold= 919141808 used= 901897809 count= 9502 avg_used= 94916 mod=SUMMARY [2025-08-14 09:06:42.740291] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= MEMSTORE_CTX_ID hold= 415236096 used= 411992064 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 411992064 used= 411803568 count= 198 avg_used= 2079816 block_cnt= 198 chunk_cnt= 198 mod=Memstore [MEMORY] hold= 411992064 used= 411803568 count= 198 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:06:42.740300] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1002 ctx_id= EXECUTE_CTX_ID hold= 1114112 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 7 washed_size= 7274496 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:42.740311] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= TRANS_CTX_MGR_ID hold= 950272 used= 933888 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1146880 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 933888 used= 921984 count= 2 avg_used= 460992 mod=SUMMARY [2025-08-14 09:06:42.740322] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= PLAN_CACHE_CTX_ID hold= 211812352 used= 201995008 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 176930880 used= 160228962 count= 13450 avg_used= 11912 block_cnt= 13450 chunk_cnt= 101 mod=SqlPhyPlan [MEMORY] hold= 201995008 used= 183048143 count= 17906 avg_used= 10222 mod=SUMMARY [2025-08-14 09:06:42.740332] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= WORK_AREA hold= 1441792 used= 0 limit= 429496725 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 14 washed_blocks= 54 washed_size= 27918336 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:42.740342] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= GLIBC hold= 12582912 used= 11425280 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 11425280 used= 6322556 count= 19216 avg_used= 329 mod=SUMMARY [2025-08-14 09:06:42.740322] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:42.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.740353] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1002 ctx_id= CO_STACK hold= 149913600 used= 147087360 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 68 washed_blocks= 70 washed_size= 3178496 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 147087360 used= 146813760 count= 285 avg_used= 515136 block_cnt= 285 chunk_cnt= 73 mod=CoStack [MEMORY] hold= 147087360 used= 146813760 count= 285 avg_used= 515136 mod=SUMMARY [2025-08-14 09:06:42.740368] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= KVSTORE_CACHE_ID hold= 48234496 used= 48234496 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 48234496 used= 0 count= 23 avg_used= 0 mod=SUMMARY [2025-08-14 09:06:42.740379] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= META_OBJ_CTX_ID hold= 37748736 used= 35741632 limit= 1717986900 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 35741632 used= 32457488 count= 496 avg_used= 65438 mod=SUMMARY [2025-08-14 09:06:42.740389] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= TX_CALLBACK_CTX_ID hold= 7766016 used= 6934400 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 17 washed_size= 6914048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 6934400 used= 6729728 count= 848 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:42.740399] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= LOB_CTX_ID hold= 147456 used= 16384 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 16384 used= 15872 count= 2 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:42.740412] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= RPC_CTX_ID hold= 5783552 used= 5029888 limit= 8589934500 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 10 washed_blocks= 10 washed_size= 15187968 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 5029888 used= 3315456 count= 210 avg_used= 15787 mod=SUMMARY [2025-08-14 09:06:42.740424] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= TX_DATA_TABLE hold= 5718016 used= 4521856 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 9 washed_blocks= 16 washed_size= 13156352 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4521856 used= 4480128 count= 40 avg_used= 112003 mod=SUMMARY [2025-08-14 09:06:42.740437] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= MDS_DATA_ID hold= 3047424 used= 1998848 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 4 washed_size= 1146880 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1998848 used= 1936384 count= 244 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:42.740447] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= MDS_CTX_ID hold= 147456 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:42.740460] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:42.740469] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= MERGE_NORMAL_CTX_ID hold= 1032192 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 12 washed_size= 13647872 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:42.740642] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=16] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:42.741968] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:42.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.743324] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [412][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=942946] [sql nio session](*s={this:0x7f98f733a030, session_id:3221590255, trace_id:YB42AC4D0007-00063C49651A3232-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:257}, err:0, last_decode_time:1755161952736711, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1716, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:06:42.749267] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2314] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:42.751257] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:42.751334] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=246] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:42.752011] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:42.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.743186] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:42.754225] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11021][errcode=0] epoll handler too much time :11049 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:42.754368] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=7] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:42.754407] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:42.754419] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:42.754436] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:02:21.775189] EDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:801) [368][VTblLocAsyncUp0][T1][YB42AC4D0007-00063C4968299C62-0-0] [lt=11][errcode=-4012] LOGGER COST TOO MUCH TIME, cost: 3979255, owner: ObLog, click_count: 3, time dist:[FORMAT_END=3979254, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x124de942 0x96a686c 0x96a5033 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x9a77e58 0x1fcfa594 0x1f7fa1c3 0x2023397e 0x202381d6 0x13ad9dbd 0x13ad61b5 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:42.757033] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=53] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:42.757079] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=1983129] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:06:42.757375] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:42.734639] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [579][T1_MultiVersion][T1][Y0-0000000000000000-0-0] [lt=1954677][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fb900, caller:ob_multi_version_garbage_collector.cpp:start:115, function_type:ZN9oceanbase19concurrency_control30ObMultiVersionGarbageCollector5startEvE6$_1159, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fbbe0, time_wheel:0x7f994b1d7b50, thread_pool:0x7f994b1d7330, is_running:1, time_interval:60.00s, expected_run_time:09:04:19.702, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.757543] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [579][T1_MultiVersion][T1][Y0-0000000000000000-0-0] [lt=22865][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:04:20.810|207=141.95s|210=0us|222=2us|total=141.95s) [2025-08-14 09:06:42.757560] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [579][T1_MultiVersion][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] timer task use too much time(end=1755162402757560, start=1755162260810474, delta=141947086) [2025-08-14 09:06:42.758260] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:42.761411] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=81] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:42.766270] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=344] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:42.773584] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:42.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.773734] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:42.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.773768] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:42.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.774392] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:42.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:42.774427] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:42.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:42.774442] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:42.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:42.774455] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:42.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:42.774467] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:42.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:42.774482] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d36b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ebb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:42.767, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:42.774494] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:42.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:42.781720] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:42.782461] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:42.784349] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:42.784465] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:42.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:42.784495] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72739b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a5e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:42.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:42.784508] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72779b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eab10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:42.777, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:42.784520] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728bc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:42.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:42.786403] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=25] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:42.794516] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919902b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:42.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:42.794548] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:42.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:42.794559] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:42.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:42.799512] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=19] gc stale ls task succ [2025-08-14 09:06:42.803789] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:42.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.804602] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7277730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:42.794, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:42.804630] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:42.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:42.824164] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:42.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.826863] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:42.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:42.831930] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:42.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.835622] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:42.836926] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:42.820, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:42.840377] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:42.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.846988] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:42.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:42.854188] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:42.854281] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=36] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:42.854323] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:42.854365] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:42.854411] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:42.854423] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:42.854435] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:42.892516] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=22] PNIO [ratelimit] time: 1755162402892515, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007621, add_bytes: 0 [2025-08-14 09:06:42.924217] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:42.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.931991] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:42.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.935681] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:42.940439] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:42.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.967756] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196aa90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:42.952, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:42.967791] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:42.953, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:42.981851] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=20] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:42.987882] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891992fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196de90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:42.970, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:42.993789] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:42.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.015192] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:43.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.015259] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=65][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:43.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.024279] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=55][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:42.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.025051] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:43.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.032018] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:42.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.035701] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:43.040504] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:42.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.095503] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:43.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.124345] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:43.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.132077] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:43.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.135761] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:43.140570] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:43.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.216239] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:43.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.224411] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:43.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.232136] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:43.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.235821] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:43.237479] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:43.237511] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:43.237531] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:43.237541] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:43.237552] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:43.237564] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:43.237578] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:43.237590] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:43.237605] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:43.237615] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:43.237626] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:43.237640] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:43.237650] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:43.237661] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:43.237673] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:43.237683] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:43.237694] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:43.237706] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:43.237719] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:43.237731] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:43.237879] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:43.237892] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:43.240635] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:43.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.245341] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:43.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.255418] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:43.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.266500] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:43.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.266545] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:43.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.276407] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:43.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.306598] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:43.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.324475] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:43.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.332180] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:43.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.335870] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:43.340697] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:43.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.368320] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=22] gc stale ls task succ [2025-08-14 09:06:43.424537] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:43.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.432244] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:43.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.435915] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:43.437461] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=327066][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc43b8, timer:0x7f9970f40f80, task:0x564d6d07a448, task_type:"N9oceanbase5share17ObServerTraceTaskE", scheduled_time:1755161621733806, pushed_time:0, delay:8000000}, running_task_token={this:0x7f9970fc5fb8, timer:0x7f9970f40f80, task:0x564d727182f0, task_type:"N9oceanbase5share16ObAshRefreshTaskE", scheduled_time:1755161611717613, pushed_time:1755161614757742, delay:120000000}, total_delay=781703653, threashold=1000000, tenant_id=500) [2025-08-14 09:06:43.440732] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:43.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.496849] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:43.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.517812] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:43.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.517855] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:43.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.524585] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:43.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.527756] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:43.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.532310] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:43.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.535929] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:43.540809] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:43.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.598279] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:43.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.623589] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=196105][errcode=0] EASY SLOW: thread: 7f995427d300, lastrun: 1755162250.789339 cost: 152.834247 loop:26617, slowcnt: 1194 [2025-08-14 09:06:43.624650] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:43.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.632376] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:43.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.635966] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:43.640896] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:43.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.754450] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=2001100] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [2025-08-14 09:06:43.723091] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=57][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:43.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.719537] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=963618] current log_task status(palf_id=1001, self="172.77.0.7:2882", log_id=8571657, log_task={header:{begin_lsn:{lsn:7323686009}, end_lsn:{lsn:7323686131}, log_id:8571657, min_scn:{val:1755161749741193000, v:0}, max_scn:{val:1755161749741193000, v:0}, data_len:66, proposal_id:49, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:7323686009}, data_checksum:2222604892, accum_checksum:1531236791, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1755161749745909, freeze_ts:1755161749745914, submit_ts:1755161749745940, flushed_ts:1755161785758706, gen_to_freeze cost time:5, gen_to_submit cost time:31, submit_to_flush cost time:36012766}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [2025-08-14 09:06:43.724652] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:43.727448] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:43.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:41.814128] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [561][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=18] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=16, total_destructed_detector=16, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f994b3ee2b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-08-14 09:06:41.814128] EDIAG [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [561][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1913196, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1913149, ObMalloc#start=2, ObMalloc#end=30, ALLOC_END=1, APPEND_END=14], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x973b640 0x973b132 0x973a32d 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:43.730307] WDIAG [DETECT] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [561][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=1913196][errcode=0] cost too much time:ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t), (this=|threshold=100.00ms|start at 09:06:43.730|total=1.92s) [2025-08-14 09:06:43.731224] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=5145] gc stale ls task succ [2025-08-14 09:06:43.731281] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:43.731352] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=39] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:43.731376] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=18] temporary file module use no blocks [2025-08-14 09:06:43.731395] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=18] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:43.731408] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:43.731429] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=21] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623305169526, flush_io_finished_ret_:0, flush_io_finished_round_:26421, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:43.732649] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:43.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.736033] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:43.737324] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=33] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:43.737354] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26] temporary file module use no blocks [2025-08-14 09:06:43.737361] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=5] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:43.737375] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:43.737404] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=29] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623305176417, flush_io_finished_ret_:0, flush_io_finished_round_:26459, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:42.761555] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12087] get wrs ts(ls_id={id:1}, delta=681949911, log_service_ts=1755161719775179, min_tx_service_ts=4611686018427387, timestamp={val:1755161719775179002, v:0}) [2025-08-14 09:06:43.740752] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=21] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:43.740958] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:43.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.741145] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:43.741174] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:43.741192] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:43.741200] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:43.741209] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:43.741350] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:43.741372] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:43.741391] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:43.741407] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:43.741417] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:43.741429] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:43.741444] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:43.741454] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:43.741466] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:43.741477] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:43.741488] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:43.741498] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:43.741510] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:43.741523] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:43.741535] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:43.741684] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:43.741697] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:43.743258] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=51] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:43.743338] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:43.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.743817] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:43.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:43.743852] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:43.737, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:43.743866] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e71b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:43.739, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:43.743880] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918119b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:43.737, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:43.749388] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:43.751250] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=21] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:43.751254] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=30] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:43.751329] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:43.741396] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EED1-0-0] [lt=15][errcode=-4016] task executed too delay(delay_time=87304, *p_task_={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:0, time_interval:3.00s, expected_run_time:09:06:46.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.752254] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:43.754257] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:43.755357] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:43.755493] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72774b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:43.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:43.755520] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fbe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:43.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:43.756618] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:43.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:42.764321] INFO [SQL.PC] need_check_schema_version (ob_plan_cache_value.cpp:2334) [15491][T1002_TimerWK15_ReqMemEvict][T1002][YB42AC4D0007-00063C4982199C19-0-0] [lt=944205] need check schema(new_schema_version=1755156894100160, cached_tenant_schema_version=1755156894100160, is_contain_synonym()=false, contain_sys_name_table_=true, is_contain_tmp_tbl()=false, is_contain_sys_pl_object()=false, need_check=true, constructed_sql_=SELECT VALUE FROM __all_sys_stat WHERE ZONE = ? AND NAME = ? AND TENANT_ID = ? FOR UPDATE) [2025-08-14 09:06:43.757254] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:43.758110] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=17] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:06:43.760324] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EED3-0-0] [lt=11355] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:06:43.762522] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:43.762604] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=66] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:43.765600] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:43.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:43.765640] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:43.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:43.765654] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:43.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:43.766262] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:43.766698] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:43.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.766734] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:43.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.764380] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:43.774857] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10459][errcode=0] epoll handler too much time :10504 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:04:23.767107] INFO dump_info (ob_timer_service.cpp:728) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=8838466] dump info [summary](this=0x7f9969de5330, this={this:0x7f9969de5330, tenant_id_:1, priority_task_queue_.size():23, running_task_set_.size():26, uncanceled_task_set_.size():0, worker_thread_pool_.get_queue_num():10}) [2025-08-14 09:06:43.775792] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919aef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:43.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:43.775821] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7270a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a4190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:43.762, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:43.775833] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:43.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:43.775846] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:43.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:43.775857] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:43.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:43.776698] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:43.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.781827] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=25] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:43.782418] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:43.783374] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:43.785856] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f734de30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891919810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:43.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:43.785896] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:43.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:43.785920] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:43.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:43.785933] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d59b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:43.773, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:43.785953] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efa10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:43.777, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:43.786487] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=23] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:43.795916] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eba10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:43.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:43.795951] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:43.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:43.799578] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=21] gc stale ls task succ [2025-08-14 09:06:43.805956] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:43.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:43.805988] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:43.790, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:43.806804] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:43.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.820907] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] [sql nio session](*s={this:0x7f98f72d0030, session_id:3221589415, trace_id:YB42AC4D0007-00063C4965FA43DC-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:245}, err:0, last_decode_time:1755161751714366, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1662, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:06:43.820940] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=32] [sql nio session](*s={this:0x7f98f72d5030, session_id:3221590200, trace_id:YB42AC4D0007-00063C4965FA43E0-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:248}, err:0, last_decode_time:1755161776777167, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1593, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:06:43.827515] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:43.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.832836] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:43.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.836075] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=40] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:06:43.836109] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:43.836147] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:43.827, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:43.841020] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:43.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.846210] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739af30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196bc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:43.837, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:43.856267] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=49][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:43.841, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:43.856300] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:43.849, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:43.856311] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:43.849, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:43.856421] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:43.856484] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:43.856532] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:43.856576] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:43.856628] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:43.856640] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:43.856651] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:43.882656] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=397][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fc8b0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcb90, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:1, time_interval:3.00s, expected_run_time:09:06:43.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.900127] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=24] PNIO [ratelimit] time: 1755162403900125, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007610, add_bytes: 0 [2025-08-14 09:06:43.906595] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:43.892, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:43.926689] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:43.917, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:43.927593] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:43.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.933038] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:43.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.936136] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:43.936735] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:43.922, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:43.941076] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:43.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.982518] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=19] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:43.998143] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:43.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.018262] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:44.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.018298] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:44.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.027660] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:43.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.028330] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:44.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.033143] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:43.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.036221] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:44.041106] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:43.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.099669] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:44.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.127703] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:44.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.133387] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:44.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.136277] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:44.141171] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:44.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.210267] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:44.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.227767] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:44.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.233451] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:44.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.236297] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:44.241224] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:44.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.241873] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:44.241902] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:44.241918] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:44.241934] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:44.241944] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:44.241953] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:44.241971] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:44.241983] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:44.241997] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:44.242008] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:44.242018] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:44.242032] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=719, timeout ts=09:04:42.406, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:04:12.406], idx=719, next_allow_print_ts="09:05:12.406") [2025-08-14 09:06:44.242043] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:44.242054] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:44.242066] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:44.242076] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:44.242087] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:44.242098] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:44.242111] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:44.242122] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:44.242291] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:44.242304] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:44.249962] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:44.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.260004] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:44.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.260544] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:44.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.260571] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:44.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.279808] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:44.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.309995] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:44.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.327828] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:44.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.333513] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:44.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.336333] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:44.341276] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:44.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.368386] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=22] gc stale ls task succ [2025-08-14 09:06:44.427872] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:44.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.433577] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=51][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:44.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.436393] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:44.437990] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fa190, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa470, time_wheel:0x7f99341cdbf0, thread_pool:0x7f99341cd3d0, is_running:1, time_interval:2.00s, expected_run_time:09:06:44.334, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.441313] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:44.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.491389] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:44.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.511887] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:44.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.511924] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:44.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.521065] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:44.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.527957] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:44.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.533641] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:44.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.536452] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:44.541395] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:44.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.592208] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:44.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.624701] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] EASY SLOW: thread: 7f995427d300, lastrun: 1755162250.789339 cost: 153.835360 loop:26617, slowcnt: 1204 [2025-08-14 09:06:44.628007] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:44.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.633686] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:44.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.636513] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:44.641439] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:44.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.712932] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=64][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:44.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.721827] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:44.728672] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:44.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.728838] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=55] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:44.731286] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=29] gc stale ls task succ [2025-08-14 09:06:44.733729] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:44.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.734663] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd110, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd3f0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:10.00s, expected_run_time:09:06:44.726, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.736575] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:44.737509] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=37] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:44.737539] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26] temporary file module use no blocks [2025-08-14 09:06:44.737550] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:44.737567] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=15] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:44.737595] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=27] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623306177210, flush_io_finished_ret_:0, flush_io_finished_round_:26460, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:04:23.767107] EDIAG dump_info (ob_timer_service.cpp:728) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=8838466][errcode=0] LOGGER COST TOO MUCH TIME, cost: 139796382, owner: ObLog, click_count: 9, time dist:[ALLOC_BLOCK#start=6, ALLOC_BLOCK#end=2, FORMAT_END=995753, ObMalloc#start=0, ObMalloc#end=71, ObMalloc#start=138800545, ObMalloc#end=1, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x2618816f 0x9756b3d 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:20.871822] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1855653, , BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:44.738814] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=1855653][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:04:20.871|207=143.87s|210=0us|222=2us|total=143.87s) [2025-08-14 09:06:44.738853] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=-4389] timer task use too much time(end=1755162404738852, start=1755162260871820, delta=143867032) [2025-08-14 09:06:44.738875] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=139944962, expected_run_ts_us_=1755162264793911, ObClockGenerator::getRealClock()=1755162404738874, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd070, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fd350, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:5.00s, expected_run_time:09:04:24.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.738903] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd070, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fd350, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:5.00s, expected_run_time:09:04:24.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.739054] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:44.728, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:44.739088] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f734c030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891807390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:44.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:04:24.764217] EDIAG [OCCAM] operator() (ob_occam_timer.h:85) [719][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 139764731, owner: ObLog, click_count: 7, time dist:[FORMAT_END=980845, ObMalloc#start=1, ObMalloc#end=63, ObMalloc#start=138783814, ObMalloc#end=3, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ffa3b 0x9835318 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:24.764217] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [719][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:44.740404] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [719][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=139764731][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:06:44.740|total=152.33s) [2025-08-14 09:06:44.737972] INFO dump_info (ob_timer_service.cpp:731) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=139796382] dump queue token(this=0x7f9969de5330, token={this:0x7f994e1a66b8, timer:0x7f9966fddee0, task:0x7f994f1b1e78, task_type:"N9oceanbase5share23ObStorageIOUsageRepoter23TenantIOUsageReportTaskE", scheduled_time:1755161694839037, pushed_time:0, delay:5000000}) [2025-08-14 09:06:44.740659] INFO dump_info (ob_timer_service.cpp:731) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=2682] dump queue token(this=0x7f9969de5330, token={this:0x7f994e1a6258, timer:0x7f9966fddee0, task:0x7f994f1b14e0, task_type:"N9oceanbase5table20ObTableObjectPoolMgr26ObTableSessEliminationTaskE", scheduled_time:1755161695725325, pushed_time:0, delay:5000000}) [2025-08-14 09:06:44.740701] INFO dump_info (ob_timer_service.cpp:731) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=41] dump queue token(this=0x7f9969de5330, token={this:0x7f994e1a64f8, timer:0x7f9966fddee0, task:0x7f994f1b1510, task_type:"N9oceanbase5table20ObTableObjectPoolMgr28ObTableReqResEliminationTaskE", scheduled_time:1755161695734026, pushed_time:0, delay:5000000}) [2025-08-14 09:06:44.740723] INFO dump_info (ob_timer_service.cpp:731) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=22] dump queue token(this=0x7f9969de5330, token={this:0x7f994e1a6418, timer:0x7f9966fddee0, task:0x7f994f1b14f8, task_type:"N9oceanbase5table20ObTableObjectPoolMgr27ObTableSessSysVarUpdateTaskE", scheduled_time:1755161695734641, pushed_time:0, delay:5000000}) [2025-08-14 09:06:44.740734] INFO dump_info (ob_timer_service.cpp:731) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=11] dump queue token(this=0x7f9969de5330, token={this:0x7f994e1a4f18, timer:0x7f9966fddee0, task:0x7f994b3f6b30, task_type:"N9oceanbase3sql3dtl23ObDTLIntermResultGCTaskE", scheduled_time:1755161700757633, pushed_time:0, delay:10000000}) [2025-08-14 09:06:44.740749] INFO dump_info (ob_timer_service.cpp:731) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=14] dump queue token(this=0x7f9969de5330, token={this:0x7f994e1a6178, timer:0x7f9966fddee0, task:0x7f994f1b2fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161720715668, pushed_time:0, delay:900000000}) [2025-08-14 09:06:44.740760] INFO dump_info (ob_timer_service.cpp:731) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=11] dump queue token(this=0x7f9969de5330, token={this:0x7f994e1a58b8, timer:0x7f9966ee3e30, task:0x7f994f1a3a78, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr15MediumCheckTaskE", scheduled_time:1755161764744054, pushed_time:0, delay:20000000}) [2025-08-14 09:06:44.740774] INFO dump_info (ob_timer_service.cpp:731) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=13] dump queue token(this=0x7f9969de5330, token={this:0x7f994e1a4658, timer:0x7f9956bffee0, task:0x7f9969bb3038, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr12TabletGCTaskE", scheduled_time:1755161923366542, pushed_time:0, delay:20000}) [2025-08-14 09:06:44.740781] INFO dump_info (ob_timer_service.cpp:731) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=6] dump queue token(this=0x7f9969de5330, token={this:0x7f994e1a6798, timer:0x7f9966fddee0, task:0x7f99697d29f8, task_type:"N9oceanbase5table21ObTableGroupCommitMgr19ObTableGroupOpsTaskE", scheduled_time:1755161925783083, pushed_time:0, delay:1000000}) [2025-08-14 09:06:44.740823] INFO dump_info (ob_timer_service.cpp:731) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=42] dump queue token(this=0x7f9969de5330, token={this:0x7f994e1a6a38, timer:0x7f9966fddee0, task:0x7f994f1b35a8, task_type:"N9oceanbase3omt11ObTenantSrs27TenantSrsUpdatePeriodicTaskE", scheduled_time:1755161929777851, pushed_time:0, delay:0}) [2025-08-14 09:06:44.740837] INFO dump_info (ob_timer_service.cpp:731) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=14] dump queue token(this=0x7f9969de5330, token={this:0x7f994e1a5a78, timer:0x7f996714bf20, task:0x7f994b3fce90, task_type:"N9oceanbase7storage21ObTenantFreezeInfoMgr24UpdateLSResvSnapshotTaskE", scheduled_time:1755161931783996, pushed_time:0, delay:10000000}) [2025-08-14 09:06:44.740848] INFO dump_info (ob_timer_service.cpp:731) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=11] dump queue token(this=0x7f9969de5330, token={this:0x7f994e1a4578, timer:0x7f9956bffee0, task:0x7f9969bb3028, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr17RefreshConfigTaskE", scheduled_time:1755161932756260, pushed_time:0, delay:10000000}) [2025-08-14 09:06:44.740855] INFO dump_info (ob_timer_service.cpp:731) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=6] dump queue token(this=0x7f9969de5330, token={this:0x7f994e1a6958, timer:0x7f9966fddee0, task:0x7f99697d33a0, task_type:"N9oceanbase5table20ObTableClientInfoMgr27ObTableClientInfoRetireTaskE", scheduled_time:1755161984788358, pushed_time:0, delay:60000000}) [2025-08-14 09:06:44.740860] INFO dump_info (ob_timer_service.cpp:731) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=6] dump queue token(this=0x7f9969de5330, token={this:0x7f994e1a74b8, timer:0x7f9966fddee0, task:0x7f9969106030, task_type:"N9oceanbase8observer20ObTableQueryASyncMgrE", scheduled_time:1755161985758107, pushed_time:0, delay:0}) [2025-08-14 09:06:44.740874] INFO dump_info (ob_timer_service.cpp:731) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=13] dump queue token(this=0x7f9969de5330, token={this:0x7f994e1a5fb8, timer:0x7f994b3feee0, task:0x7f994b3fef80, task_type:"N9oceanbase8observer18ObTableLoadService21ObClientTaskPurgeTaskE", scheduled_time:1755162027715356, pushed_time:0, delay:1000000}) [2025-08-14 09:06:44.740881] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:44.740885] INFO dump_info (ob_timer_service.cpp:731) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=10] dump queue token(this=0x7f9969de5330, token={this:0x7f994e1a5b58, timer:0x7f994b3feee0, task:0x7f994b3fef08, task_type:"N9oceanbase8observer18ObTableLoadService17ObCheckTenantTaskE", scheduled_time:1755162032742702, pushed_time:0, delay:1000000}) [2025-08-14 09:06:44.740901] INFO dump_info (ob_timer_service.cpp:731) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=17] dump queue token(this=0x7f9969de5330, token={this:0x7f994e1a4e38, timer:0x7f9967161f10, task:0x7f994b3f4c40, task_type:"N9oceanbase7obmysql20ObConstructQueueTaskE", scheduled_time:1755162048238296, pushed_time:0, delay:500000}) [2025-08-14 09:06:44.740910] INFO dump_info (ob_timer_service.cpp:731) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=8] dump queue token(this=0x7f9969de5330, token={this:0x7f994e1a5ed8, timer:0x7f994b3feee0, task:0x7f994b3fef68, task_type:"N9oceanbase8observer18ObTableLoadService25ObClientTaskAutoAbortTaskE", scheduled_time:1755162048762959, pushed_time:0, delay:1000000}) [2025-08-14 09:06:44.740918] INFO dump_info (ob_timer_service.cpp:731) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=8] dump queue token(this=0x7f9969de5330, token={this:0x7f994e1a5d18, timer:0x7f994b3feee0, task:0x7f994b3fef38, task_type:"N9oceanbase8observer18ObTableLoadService8ObGCTaskE", scheduled_time:1755162074747216, pushed_time:0, delay:30000000}) [2025-08-14 09:06:44.740926] INFO dump_info (ob_timer_service.cpp:731) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=9] dump queue token(this=0x7f9969de5330, token={this:0x7f994e1a5c38, timer:0x7f994b3feee0, task:0x7f994b3fef20, task_type:"N9oceanbase8observer18ObTableLoadService15ObHeartBeatTaskE", scheduled_time:1755162132756830, pushed_time:0, delay:10000000}) [2025-08-14 09:06:44.740939] INFO dump_info (ob_timer_service.cpp:731) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=13] dump queue token(this=0x7f9969de5330, token={this:0x7f994e1a56f8, timer:0x7f9966ed5f40, task:0x7f994f1a3a68, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr24TabletUpdaterRefreshTaskE", scheduled_time:1755162326715099, pushed_time:0, delay:300000000}) [2025-08-14 09:06:44.740949] INFO dump_info (ob_timer_service.cpp:731) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=9] dump queue token(this=0x7f9969de5330, token={this:0x7f994e1a6338, timer:0x7f996705db70, task:0x7f994b3a0078, task_type:"N9oceanbase8observer32ObTenantTabletMetaTableCheckTaskE", scheduled_time:1755162620219677, pushed_time:0, delay:0}) [2025-08-14 09:06:44.740958] INFO dump_info (ob_timer_service.cpp:731) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=10] dump queue token(this=0x7f9969de5330, token={this:0x7f994e1a6098, timer:0x7f9966fddee0, task:0x7f994f1b2f88, task_type:"N9oceanbase6common28ObOptStatMonitorFlushAllTaskE", scheduled_time:1755222019703160, pushed_time:0, delay:86400000000}) [2025-08-14 09:06:44.740980] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a66b8, timer:0x7f9966fddee0, task:0x7f994f1b1e78, task_type:"N9oceanbase5share23ObStorageIOUsageRepoter23TenantIOUsageReportTaskE", scheduled_time:1755161694839037, pushed_time:0, delay:5000000}, running_task_token={this:0x7f994e1a65d8, timer:0x7f9966fddee0, task:0x7f994f1bbcc0, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161694824965, pushed_time:1755161925772608, delay:5000000}, total_delay=709901942, threashold=1000000, tenant_id=1) [2025-08-14 09:06:44.741012] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=32][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a6258, timer:0x7f9966fddee0, task:0x7f994f1b14e0, task_type:"N9oceanbase5table20ObTableObjectPoolMgr26ObTableSessEliminationTaskE", scheduled_time:1755161695725325, pushed_time:0, delay:5000000}, running_task_token={this:0x7f994e1a65d8, timer:0x7f9966fddee0, task:0x7f994f1bbcc0, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161694824965, pushed_time:1755161925772608, delay:5000000}, total_delay=709015687, threashold=1000000, tenant_id=1) [2025-08-14 09:06:44.741025] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a64f8, timer:0x7f9966fddee0, task:0x7f994f1b1510, task_type:"N9oceanbase5table20ObTableObjectPoolMgr28ObTableReqResEliminationTaskE", scheduled_time:1755161695734026, pushed_time:0, delay:5000000}, running_task_token={this:0x7f994e1a65d8, timer:0x7f9966fddee0, task:0x7f994f1bbcc0, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161694824965, pushed_time:1755161925772608, delay:5000000}, total_delay=709006999, threashold=1000000, tenant_id=1) [2025-08-14 09:06:44.741037] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a6418, timer:0x7f9966fddee0, task:0x7f994f1b14f8, task_type:"N9oceanbase5table20ObTableObjectPoolMgr27ObTableSessSysVarUpdateTaskE", scheduled_time:1755161695734641, pushed_time:0, delay:5000000}, running_task_token={this:0x7f994e1a65d8, timer:0x7f9966fddee0, task:0x7f994f1bbcc0, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161694824965, pushed_time:1755161925772608, delay:5000000}, total_delay=709006396, threashold=1000000, tenant_id=1) [2025-08-14 09:06:44.741063] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a4f18, timer:0x7f9966fddee0, task:0x7f994b3f6b30, task_type:"N9oceanbase3sql3dtl23ObDTLIntermResultGCTaskE", scheduled_time:1755161700757633, pushed_time:0, delay:10000000}, running_task_token={this:0x7f994e1a65d8, timer:0x7f9966fddee0, task:0x7f994f1bbcc0, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161694824965, pushed_time:1755161925772608, delay:5000000}, total_delay=703983430, threashold=1000000, tenant_id=1) [2025-08-14 09:06:44.741075] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a6178, timer:0x7f9966fddee0, task:0x7f994f1b2fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161720715668, pushed_time:0, delay:900000000}, running_task_token={this:0x7f994e1a65d8, timer:0x7f9966fddee0, task:0x7f994f1bbcc0, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161694824965, pushed_time:1755161925772608, delay:5000000}, total_delay=684025407, threashold=1000000, tenant_id=1) [2025-08-14 09:06:44.741087] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a58b8, timer:0x7f9966ee3e30, task:0x7f994f1a3a78, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr15MediumCheckTaskE", scheduled_time:1755161764744054, pushed_time:0, delay:20000000}, running_task_token={this:0x7f994e1a57d8, timer:0x7f9966ee3e30, task:0x7f994f1a3a38, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr14MediumLoopTaskE", scheduled_time:1755161760733696, pushed_time:1755161760733763, delay:120000000}, total_delay=639997033, threashold=1000000, tenant_id=1) [2025-08-14 09:06:44.741100] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a4658, timer:0x7f9956bffee0, task:0x7f9969bb3038, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr12TabletGCTaskE", scheduled_time:1755161923366542, pushed_time:0, delay:20000}, running_task_token={this:0x7f994e1a4498, timer:0x7f9956bffee0, task:0x7f9969bb3010, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr11TableGCTaskE", scheduled_time:1755161923356444, pushed_time:1755161923356659, delay:20000}, total_delay=481374558, threashold=1000000, tenant_id=1) [2025-08-14 09:06:44.741113] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a6798, timer:0x7f9966fddee0, task:0x7f99697d29f8, task_type:"N9oceanbase5table21ObTableGroupCommitMgr19ObTableGroupOpsTaskE", scheduled_time:1755161925783083, pushed_time:0, delay:1000000}, running_task_token={this:0x7f994e1a65d8, timer:0x7f9966fddee0, task:0x7f994f1bbcc0, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161694824965, pushed_time:1755161925772608, delay:5000000}, total_delay=478958030, threashold=1000000, tenant_id=1) [2025-08-14 09:06:44.741124] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a6a38, timer:0x7f9966fddee0, task:0x7f994f1b35a8, task_type:"N9oceanbase3omt11ObTenantSrs27TenantSrsUpdatePeriodicTaskE", scheduled_time:1755161929777851, pushed_time:0, delay:0}, running_task_token={this:0x7f994e1a65d8, timer:0x7f9966fddee0, task:0x7f994f1bbcc0, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161694824965, pushed_time:1755161925772608, delay:5000000}, total_delay=474963273, threashold=1000000, tenant_id=1) [2025-08-14 09:06:44.741154] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a5a78, timer:0x7f996714bf20, task:0x7f994b3fce90, task_type:"N9oceanbase7storage21ObTenantFreezeInfoMgr24UpdateLSResvSnapshotTaskE", scheduled_time:1755161931783996, pushed_time:0, delay:10000000}, running_task_token={this:0x7f994e1a5998, timer:0x7f996714bf20, task:0x7f994b3fce78, task_type:"N9oceanbase7storage21ObTenantFreezeInfoMgr10ReloadTaskE", scheduled_time:1755161924772450, pushed_time:1755161924772520, delay:3000000}, total_delay=472957158, threashold=1000000, tenant_id=1) [2025-08-14 09:06:44.741166] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a4578, timer:0x7f9956bffee0, task:0x7f9969bb3028, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr17RefreshConfigTaskE", scheduled_time:1755161932756260, pushed_time:0, delay:10000000}, running_task_token={this:0x7f994e1a4498, timer:0x7f9956bffee0, task:0x7f9969bb3010, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr11TableGCTaskE", scheduled_time:1755161923356444, pushed_time:1755161923356659, delay:20000}, total_delay=471984906, threashold=1000000, tenant_id=1) [2025-08-14 09:06:44.741187] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a6958, timer:0x7f9966fddee0, task:0x7f99697d33a0, task_type:"N9oceanbase5table20ObTableClientInfoMgr27ObTableClientInfoRetireTaskE", scheduled_time:1755161984788358, pushed_time:0, delay:60000000}, running_task_token={this:0x7f994e1a65d8, timer:0x7f9966fddee0, task:0x7f994f1bbcc0, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161694824965, pushed_time:1755161925772608, delay:5000000}, total_delay=419952828, threashold=1000000, tenant_id=1) [2025-08-14 09:06:44.741206] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a74b8, timer:0x7f9966fddee0, task:0x7f9969106030, task_type:"N9oceanbase8observer20ObTableQueryASyncMgrE", scheduled_time:1755161985758107, pushed_time:0, delay:0}, running_task_token={this:0x7f994e1a65d8, timer:0x7f9966fddee0, task:0x7f994f1bbcc0, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161694824965, pushed_time:1755161925772608, delay:5000000}, total_delay=418983099, threashold=1000000, tenant_id=1) [2025-08-14 09:06:44.741217] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a5fb8, timer:0x7f994b3feee0, task:0x7f994b3fef80, task_type:"N9oceanbase8observer18ObTableLoadService21ObClientTaskPurgeTaskE", scheduled_time:1755162027715356, pushed_time:0, delay:1000000}, running_task_token={this:0x7f994e1a5df8, timer:0x7f994b3feee0, task:0x7f994b3fef50, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162023758648, pushed_time:1755162122757495, delay:1000000}, total_delay=377025860, threashold=1000000, tenant_id=1) [2025-08-14 09:06:44.741310] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=93][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a5b58, timer:0x7f994b3feee0, task:0x7f994b3fef08, task_type:"N9oceanbase8observer18ObTableLoadService17ObCheckTenantTaskE", scheduled_time:1755162032742702, pushed_time:0, delay:1000000}, running_task_token={this:0x7f994e1a5df8, timer:0x7f994b3feee0, task:0x7f994b3fef50, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162023758648, pushed_time:1755162122757495, delay:1000000}, total_delay=371998608, threashold=1000000, tenant_id=1) [2025-08-14 09:06:44.741332] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a4e38, timer:0x7f9967161f10, task:0x7f994b3f4c40, task_type:"N9oceanbase7obmysql20ObConstructQueueTaskE", scheduled_time:1755162048238296, pushed_time:0, delay:500000}, running_task_token={this:0x7f994e1a4d58, timer:0x7f9967161f10, task:0x7f994b3f4bf0, task_type:"N9oceanbase7obmysql15ObEliminateTaskE", scheduled_time:1755162042245325, pushed_time:1755162047738501, delay:500000}, total_delay=356503036, threashold=1000000, tenant_id=1) [2025-08-14 09:06:44.741344] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a5ed8, timer:0x7f994b3feee0, task:0x7f994b3fef68, task_type:"N9oceanbase8observer18ObTableLoadService25ObClientTaskAutoAbortTaskE", scheduled_time:1755162048762959, pushed_time:0, delay:1000000}, running_task_token={this:0x7f994e1a5df8, timer:0x7f994b3feee0, task:0x7f994b3fef50, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162023758648, pushed_time:1755162122757495, delay:1000000}, total_delay=355978384, threashold=1000000, tenant_id=1) [2025-08-14 09:06:44.741361] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a5d18, timer:0x7f994b3feee0, task:0x7f994b3fef38, task_type:"N9oceanbase8observer18ObTableLoadService8ObGCTaskE", scheduled_time:1755162074747216, pushed_time:0, delay:30000000}, running_task_token={this:0x7f994e1a5df8, timer:0x7f994b3feee0, task:0x7f994b3fef50, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162023758648, pushed_time:1755162122757495, delay:1000000}, total_delay=329994145, threashold=1000000, tenant_id=1) [2025-08-14 09:06:44.741383] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a5c38, timer:0x7f994b3feee0, task:0x7f994b3fef20, task_type:"N9oceanbase8observer18ObTableLoadService15ObHeartBeatTaskE", scheduled_time:1755162132756830, pushed_time:0, delay:10000000}, running_task_token={this:0x7f994e1a5df8, timer:0x7f994b3feee0, task:0x7f994b3fef50, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162023758648, pushed_time:1755162122757495, delay:1000000}, total_delay=271984553, threashold=1000000, tenant_id=1) [2025-08-14 09:06:44.741396] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a56f8, timer:0x7f9966ed5f40, task:0x7f994f1a3a68, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr24TabletUpdaterRefreshTaskE", scheduled_time:1755162326715099, pushed_time:0, delay:300000000}, running_task_token={this:0x7f994e1a5618, timer:0x7f9966ed5f40, task:0x7f994f1a3a58, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr18InfoPoolResizeTaskE", scheduled_time:1755162087744944, pushed_time:1755162087745029, delay:30000000}, total_delay=78026297, threashold=1000000, tenant_id=1) [2025-08-14 09:06:44.741503] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:44.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.742436] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=969929] [sql nio session](*s={this:0x7f98f733f030, session_id:3221589417, trace_id:YB42AC4D0007-00063C496359B268-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:252}, err:0, last_decode_time:1755161813752329, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1778, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:06:44.742470] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=34] [sql nio session](*s={this:0x7f98f733e030, session_id:3221589980, trace_id:YB42AC4D0007-00063C496359B261-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:251}, err:0, last_decode_time:1755161748778070, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1681, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:06:44.742483] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=13] [sql nio session](*s={this:0x7f98f7332030, session_id:3221590132, trace_id:YB42AC4D0007-00063C496359B265-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:266}, err:0, last_decode_time:1755161768783802, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:2125, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:06:44.742481] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:44.742492] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=9] [sql nio session](*s={this:0x7f98f733b030, session_id:3221590977, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:10, sql_initiative_shutdown:false, reader:{fd:211}, err:5, last_decode_time:1755162219768794, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:06:44.742501] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:44.742523] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:44.742531] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:44.742540] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:44.742549] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:44.742560] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:44.742577] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:44.742592] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:44.742602] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:44.742613] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:44.742627] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:44.742649] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:44.742661] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:44.742672] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:44.742691] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:44.742703] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:44.742715] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:44.742727] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:44.742884] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:44.742897] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:44.743403] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:44.744967] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:44.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.749996] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:44.750496] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aae10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:44.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:44.750529] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:44.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:44.750543] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e67b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:44.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:44.750555] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:44.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:44.750568] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919bfe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aa810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:44.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:44.750580] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:44.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:44.750593] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:44.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:44.750606] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:44.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:44.750618] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:44.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:44.751261] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:44.751288] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=18] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:44.752414] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1201] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:44.753041] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:44.753255] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:44.754249] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:44.754636] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:2005) [807][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=16392] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1002) [2025-08-14 09:06:44.754699] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1507) [807][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=56] start lib cache evict(tenant_id=1002, mem_hold=211812352, mem_limit=429496725, cache_obj_num=1331, cache_node_num=1330) [2025-08-14 09:06:44.754724] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1524) [807][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=23] end lib cache evict(tenant_id=1002, cache_evict_num=0, mem_hold=211812352, mem_limit=429496725, cache_obj_num=1331, cache_node_num=1330) [2025-08-14 09:06:44.754769] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2792) [807][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=42] schedule next cache evict task(evict_interval=5000000) [2025-08-14 09:06:44.755050] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:44.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.756201] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=47] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:44.757254] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:44.757286] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:44.758247] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:06:44.760549] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:44.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:44.761623] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:44.765424] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:44.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.765459] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:44.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.766003] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:44.775176] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:44.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.780673] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72707b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a4010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:44.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:44.781944] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=26] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:44.782425] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:44.783448] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:44.786533] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=16] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:44.790734] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:44.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:44.799642] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=30] gc stale ls task succ [2025-08-14 09:06:44.800792] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739acb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196bb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:44.798, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:44.805375] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=52][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:44.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.810845] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:44.806, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:44.810875] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:44.809, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:44.820905] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:44.815, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:44.828704] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=96][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:44.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.833788] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:44.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.836676] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:44.838917] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:06:44.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.838956] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd070, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fd350, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:5.00s, expected_run_time:09:06:44.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.841027] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:44.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:44.841055] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:44.832, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:44.841538] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:44.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.850217] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=977394] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:06:44.851090] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:44.841, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:44.857410] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:44.857471] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:44.857521] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:44.857568] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:44.857610] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:44.857627] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:44.857643] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:44.860453] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=978308] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:06:44.861155] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:44.859, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:44.867482] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=1011756] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:06:44.907748] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=23] PNIO [ratelimit] time: 1755162404907745, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007620, add_bytes: 0 [2025-08-14 09:06:44.928749] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=49][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:44.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.933851] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:44.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.936693] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:44.941620] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:44.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:44.983199] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=19] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:44.996533] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:44.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.016944] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:45.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.016982] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:45.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.026687] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:45.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.028828] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:44.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.033913] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:44.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.036722] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:45.041682] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:44.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.097306] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:45.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.128891] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:45.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.133977] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:45.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.136785] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:45.141743] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:45.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.218030] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:45.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.228948] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:45.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.234024] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:45.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.236841] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:45.241803] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:45.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.243027] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:45.243059] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:45.243079] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:45.243095] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:45.243108] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:45.243124] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:45.243140] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:45.243157] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:45.243170] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:45.243183] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:45.243191] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:45.243209] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:45.243229] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:45.243247] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:45.243259] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:45.243275] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:45.243288] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:45.243303] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:45.243315] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:45.243467] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:45.243485] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:45.247976] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:45.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.258034] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:45.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.268272] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:45.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.268305] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:45.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.278287] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:45.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.308477] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:45.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.329008] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:45.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.334058] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:45.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.336909] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:45.341833] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:45.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.368452] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=19] gc stale ls task succ [2025-08-14 09:06:45.429038] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:45.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.434121] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:45.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.436968] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:45.441871] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:45.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.499336] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:45.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.519748] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:45.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.519788] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:45.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.529094] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:45.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.529800] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:45.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.534172] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:45.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.537027] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:45.541932] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:45.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.590195] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:45.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.625194] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [390][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] EASY SLOW: thread: 7f995427d300, lastrun: 1755162250.789339 cost: 154.835851 loop:26617, slowcnt: 1214 [2025-08-14 09:06:45.629130] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:45.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.634204] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:45.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.637088] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:45.641996] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:45.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.710844] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=59][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:45.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.848584] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [878][T1002_ArcSender][T1002][YB42AC4D0007-00063C495EA99C16-0-0] [lt=25] ObArchiveSender is running(thread_index=4) [2025-08-14 09:06:43.848584] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [878][T1002_ArcSender][T1002][YB42AC4D0007-00063C495EA99C16-0-0] [lt=25][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1867021, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1867009, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=1, APPEND_END=7], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3c61 0x113d348b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:45.721818] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:45.721905] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=3663] LogLoopThread round_cost_time(us)(round_cost_time=6866466) [2025-08-14 09:06:45.726783] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73516b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196f510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.716, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:45.730316] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:45.730370] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:45.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.731912] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=55] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:45.731939] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=21] temporary file module use no blocks [2025-08-14 09:06:45.731949] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:45.731970] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=19] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:45.731984] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623307171508, flush_io_finished_ret_:0, flush_io_finished_round_:26423, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:44.762127] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2510) [807][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=13] Dumping All Cache Objs(alloc_obj_list.count()=1331, alloc_obj_list=[{obj_id:1, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:49654}, {obj_id:2, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50591}, {obj_id:3, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49570}, {obj_id:4, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:158277}, {obj_id:5, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207030}, {obj_id:6, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:129654}, {obj_id:7, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:170276}, {obj_id:8, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:92905}, {obj_id:9, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55437}, {obj_id:11, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49896}, {obj_id:12, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54788}, {obj_id:13, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60105}, {obj_id:15, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:108509}, {obj_id:16, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54018}, {obj_id:17, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:107151}, {obj_id:18, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127908}, {obj_id:19, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52851}, {obj_id:20, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46488}, {obj_id:21, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62127}, {obj_id:22, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:112117}, {obj_id:23, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71118}, {obj_id:24, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47416}, {obj_id:25, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64955}, {obj_id:26, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:107240}, {obj_id:27, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203554}, {obj_id:28, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:149171}, {obj_id:29, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:99644}, {obj_id:30, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:109417}, {obj_id:31, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:104852}, {obj_id:32, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:81681}, {obj_id:33, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:86868}, {obj_id:34, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:77320}, {obj_id:35, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73401}, {obj_id:36, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67092}, {obj_id:37, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:90423}, {obj_id:38, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73995}, {obj_id:39, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:115258}, {obj_id:40, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78254}, {obj_id:41, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63461}, {obj_id:42, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60610}, {obj_id:43, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59274}, {obj_id:44, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74545}, {obj_id:45, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:68364}, {obj_id:46, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62191}, {obj_id:47, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59473}, {obj_id:48, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62548}, {obj_id:49, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59505}, {obj_id:50, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62549}, {obj_id:51, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59441}, {obj_id:52, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65989}, {obj_id:53, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:61026}, {obj_id:54, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57028}, {obj_id:55, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85137}, {obj_id:56, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56916}, {obj_id:57, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:61962}, {obj_id:58, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57316}, {obj_id:59, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65864}, {obj_id:60, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56996}, {obj_id:61, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58212}, {obj_id:62, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65512}, {obj_id:63, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60453}, {obj_id:64, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56884}, {obj_id:65, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59085}, {obj_id:66, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235524}, {obj_id:67, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:133191}, {obj_id:68, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:122973}, {obj_id:69, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222820}, {obj_id:70, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:119011}, {obj_id:71, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62425}, {obj_id:72, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63273}, {obj_id:73, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173058}, {obj_id:74, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93811}, {obj_id:75, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:90631}, {obj_id:76, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:81787}, {obj_id:77, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71611}, {obj_id:78, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65300}, {obj_id:79, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51380}, {obj_id:80, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:81, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48067}, {obj_id:82, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:83, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52125}, {obj_id:84, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:85, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120160}, {obj_id:86, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:87, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97242}, {obj_id:88, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:90, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:91, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:46504}, {obj_id:92, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:68970}, {obj_id:93, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59166}, {obj_id:94, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60338}, {obj_id:95, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51478}, {obj_id:96, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:109830}, {obj_id:97, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48968}, {obj_id:98, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:112041}, {obj_id:99, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46440}, {obj_id:100, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69133}, {obj_id:101, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:102, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54372}, {obj_id:103, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:104, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48280}, {obj_id:105, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:116242}, {obj_id:106, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53185}, {obj_id:107, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:48908}, {obj_id:108, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51274}, {obj_id:109, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:108920}, {obj_id:110, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57540}, {obj_id:111, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54911}, {obj_id:112, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66049}, {obj_id:113, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:114, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:107403}, {obj_id:115, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59412}, {obj_id:116, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59924}, {obj_id:117, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64784}, {obj_id:118, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:119, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:120, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:121, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:30464}, {obj_id:122, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:123, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:124, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64774}, {obj_id:125, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63004}, {obj_id:126, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:127, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:30482}, {obj_id:133, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59119}, {obj_id:134, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:109167}, {obj_id:140, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57392}, {obj_id:142, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72205}, {obj_id:143, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60318}, {obj_id:144, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59063}, {obj_id:145, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52621}, {obj_id:146, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60640}, {obj_id:147, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:110708}, {obj_id:148, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:79964}, {obj_id:149, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105305}, {obj_id:150, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64637}, {obj_id:152, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:107581}, {obj_id:153, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56979}, {obj_id:154, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:68485}, {obj_id:155, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51889}, {obj_id:157, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50408}, {obj_id:158, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47936}, {obj_id:160, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:112534}, {obj_id:161, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69948}, {obj_id:162, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60075}, {obj_id:163, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50629}, {obj_id:164, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50612}, {obj_id:165, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:33277}, {obj_id:167, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78418}, {obj_id:168, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50072}, {obj_id:169, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51380}, {obj_id:170, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:172, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55901}, {obj_id:173, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57684}, {obj_id:175, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55665}, {obj_id:177, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49689}, {obj_id:178, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62390}, {obj_id:179, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:111173}, {obj_id:181, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67757}, {obj_id:187, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:30668}, {obj_id:188, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67536}, {obj_id:189, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:113436}, {obj_id:191, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67820}, {obj_id:192, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52042}, {obj_id:193, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46840}, {obj_id:194, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46856}, {obj_id:195, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193886}, {obj_id:197, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:128794}, {obj_id:198, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69301}, {obj_id:199, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173059}, {obj_id:201, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:145729}, {obj_id:202, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:203, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:98424}, {obj_id:205, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:137715}, {obj_id:206, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59503}, {obj_id:207, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192000}, {obj_id:210, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47408}, {obj_id:211, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59513}, {obj_id:213, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59188}, {obj_id:215, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46848}, {obj_id:218, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54467}, {obj_id:219, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62656}, {obj_id:222, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57417}, {obj_id:223, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57076}, {obj_id:225, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:80665}, {obj_id:226, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:68110}, {obj_id:227, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59478}, {obj_id:228, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54200}, {obj_id:229, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71579}, {obj_id:230, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:79425}, {obj_id:231, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53027}, {obj_id:232, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50327}, {obj_id:233, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52701}, {obj_id:234, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53495}, {obj_id:235, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54912}, {obj_id:236, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:115782}, {obj_id:237, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49369}, {obj_id:239, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71297}, {obj_id:240, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:45664}, {obj_id:241, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:242, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:29563}, {obj_id:243, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:29547}, {obj_id:244, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72439}, {obj_id:245, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:82003}, {obj_id:246, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60693}, {obj_id:247, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:248, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:249, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:68970}, {obj_id:253, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:13745}, {obj_id:254, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53850}, {obj_id:255, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1244037}, {obj_id:256, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53353}, {obj_id:257, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:979432}, {obj_id:258, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:259, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:441763}, {obj_id:260, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:131652}, {obj_id:261, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102208}, {obj_id:262, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174688}, {obj_id:263, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:68064}, {obj_id:264, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:124055}, {obj_id:265, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51856}, {obj_id:266, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55892}, {obj_id:267, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53476}, {obj_id:268, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248928}, {obj_id:270, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:111648}, {obj_id:272, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55724}, {obj_id:273, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:139752}, {obj_id:274, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65613}, {obj_id:275, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74271}, {obj_id:276, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189587}, {obj_id:277, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46437}, {obj_id:278, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47893}, {obj_id:279, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:150376}, {obj_id:280, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49545}, {obj_id:281, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175345}, {obj_id:282, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:131048}, {obj_id:283, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72474}, {obj_id:284, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:285, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:286, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240919}, {obj_id:288, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55756}, {obj_id:291, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:81957}, {obj_id:292, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268057}, {obj_id:297, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85376}, {obj_id:298, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85602}, {obj_id:299, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:111929}, {obj_id:300, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:286644}, {obj_id:301, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56303}, {obj_id:302, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182524}, {obj_id:303, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:137836}, {obj_id:304, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:216847}, {obj_id:307, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55868}, {obj_id:308, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72832}, {obj_id:309, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102124}, {obj_id:310, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225211}, {obj_id:313, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:75504}, {obj_id:314, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:147275}, {obj_id:315, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204179}, {obj_id:316, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:588871}, {obj_id:319, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69223}, {obj_id:320, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206652}, {obj_id:321, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:324, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74659}, {obj_id:325, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:232949}, {obj_id:326, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53958}, {obj_id:327, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180179}, {obj_id:328, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:135494}, {obj_id:332, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:80174}, {obj_id:333, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:259278}, {obj_id:334, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55154}, {obj_id:335, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181355}, {obj_id:336, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:136712}, {obj_id:342, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67384}, {obj_id:343, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197887}, {obj_id:345, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53618}, {obj_id:346, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179771}, {obj_id:347, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:135134}, {obj_id:348, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:350, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:146442}, {obj_id:364, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:75480}, {obj_id:368, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50876}, {obj_id:369, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74415}, {obj_id:370, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51322}, {obj_id:371, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51084}, {obj_id:372, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51442}, {obj_id:374, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51202}, {obj_id:375, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52790}, {obj_id:376, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52854}, {obj_id:377, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54387}, {obj_id:378, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52854}, {obj_id:379, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51586}, {obj_id:380, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54872}, {obj_id:381, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51378}, {obj_id:382, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53030}, {obj_id:383, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55599}, {obj_id:384, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51378}, {obj_id:385, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51314}, {obj_id:386, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51250}, {obj_id:387, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53046}, {obj_id:388, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54815}, {obj_id:389, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51218}, {obj_id:390, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51218}, {obj_id:391, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51250}, {obj_id:392, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52870}, {obj_id:393, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51266}, {obj_id:394, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51362}, {obj_id:395, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51250}, {obj_id:396, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59119}, {obj_id:397, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51362}, {obj_id:398, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53062}, {obj_id:399, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51362}, {obj_id:400, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51362}, {obj_id:401, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51250}, {obj_id:402, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51218}, {obj_id:409, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:149775}, {obj_id:410, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66617}, {obj_id:415, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:5265}, {obj_id:416, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:31009}, {obj_id:417, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:418, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48336}, {obj_id:419, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59108}, {obj_id:420, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50938}, {obj_id:421, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48671}, {obj_id:422, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56563}, {obj_id:423, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46872}, {obj_id:424, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:61808}, {obj_id:425, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59824}, {obj_id:427, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:618230}, {obj_id:428, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:153758}, {obj_id:429, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69455}, {obj_id:431, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55724}, {obj_id:433, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55756}, {obj_id:436, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85376}, {obj_id:437, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:108532}, {obj_id:439, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55868}, {obj_id:441, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:75504}, {obj_id:443, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:390100}, {obj_id:444, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:360217}, {obj_id:445, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:457, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55740}, {obj_id:458, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:96462}, {obj_id:459, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212819}, {obj_id:460, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:340001}, {obj_id:461, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:473, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:154884}, {obj_id:476, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:375543}, {obj_id:477, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:116952}, {obj_id:478, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:123944}, {obj_id:479, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105139}, {obj_id:490, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50527}, {obj_id:491, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:151767}, {obj_id:494, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47368}, {obj_id:495, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48232}, {obj_id:496, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62071}, {obj_id:497, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:61352}, {obj_id:500, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52851}, {obj_id:504, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:333278}, {obj_id:505, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:129618}, {obj_id:507, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55724}, {obj_id:509, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55756}, {obj_id:512, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85376}, {obj_id:514, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55868}, {obj_id:516, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:75504}, {obj_id:525, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:549, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53306}, {obj_id:550, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62783}, {obj_id:551, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54109}, {obj_id:552, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:103697}, {obj_id:553, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120708}, {obj_id:554, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67452}, {obj_id:560, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:349340}, {obj_id:561, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127623}, {obj_id:563, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55724}, {obj_id:565, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55756}, {obj_id:568, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85376}, {obj_id:570, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55868}, {obj_id:572, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:75504}, {obj_id:581, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:605, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53171}, {obj_id:606, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55852}, {obj_id:607, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:116904}, {obj_id:608, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:258163}, {obj_id:609, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:132926}, {obj_id:610, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:80606}, {obj_id:611, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127311}, {obj_id:612, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:124808}, {obj_id:613, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54894}, {obj_id:614, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58088}, {obj_id:615, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:88608}, {obj_id:616, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:121604}, {obj_id:617, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63204}, {obj_id:618, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52886}, {obj_id:623, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:465799}, {obj_id:625, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55724}, {obj_id:627, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55756}, {obj_id:630, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85376}, {obj_id:632, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55868}, {obj_id:636, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:75504}, {obj_id:647, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:674, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:281099}, {obj_id:675, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:133643}, {obj_id:677, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55724}, {obj_id:679, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55756}, {obj_id:682, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85376}, {obj_id:684, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55868}, {obj_id:686, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:75504}, {obj_id:692, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196769}, {obj_id:700, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55740}, {obj_id:701, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:728, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:277051}, {obj_id:730, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55724}, {obj_id:732, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55756}, {obj_id:735, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85376}, {obj_id:737, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55868}, {obj_id:739, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:75504}, {obj_id:749, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:773, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:603737}, {obj_id:774, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58158}, {obj_id:775, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:84541}, {obj_id:776, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66315}, {obj_id:777, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200157}, {obj_id:778, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:277379}, {obj_id:780, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279299}, {obj_id:781, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54040}, {obj_id:782, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57275}, {obj_id:783, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71438}, {obj_id:785, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:108323}, {obj_id:786, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:145723}, {obj_id:787, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63771}, {obj_id:789, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:792, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:798, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:473748}, {obj_id:800, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55724}, {obj_id:802, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55756}, {obj_id:805, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85376}, {obj_id:807, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55868}, {obj_id:809, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:75504}, {obj_id:820, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:842, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46504}, {obj_id:843, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53171}, {obj_id:844, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55852}, {obj_id:845, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:116904}, {obj_id:846, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:258163}, {obj_id:847, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:80606}, {obj_id:848, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127311}, {obj_id:849, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54894}, {obj_id:855, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:81073}, {obj_id:856, tenant_id:1002, log_del_time:9223372036854775807, re [2025-08-14 09:06:45.732441] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2800) [807][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=968842] schedule next cache evict task(evict_interval=5000000) [2025-08-14 09:06:45.732454] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a4118, timer:0x7f9922d73f40, task:0x7f9922ff1b70, task_type:"N9oceanbase3sql26ObPlanCacheEliminationTaskE", scheduled_time:1755162387772774, pushed_time:1755162387780972, delay:5000000}, delay_in_thread_pool=12998298, threashold=500000) [2025-08-14 09:06:44.740603] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=974802] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=2, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1755161287778691002, v:0}", last_lsn={lsn:8461098161}, last_gts={val:1755161724715616000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-08-14 09:06:45.733586] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [858][T1002_LSArchive][T1002][YB42AC4D0007-00063C495E199C16-0-0] [lt=17] gc stale ls task succ [2025-08-14 09:06:45.734430] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=51][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:45.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:43.757515] EDIAG [SERVER] query (ob_inner_sql_connection.cpp:946) [368][VTblLocAsyncUp0][T1][YB42AC4D0007-00063C4968299C62-0-0] [lt=3979255][errcode=-4012] LOGGER COST TOO MUCH TIME, cost: 1974742, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1974716, ObMalloc#start=1, ObMalloc#end=4, ALLOC_END=0, APPEND_END=21], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x124dfcec 0x96a5a2f 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x9a77e58 0x1fcfa594 0x1f7fa1c3 0x2023397e 0x202381d6 0x13ad9dbd 0x13ad61b5 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:43.757515] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:946) [368][VTblLocAsyncUp0][T1][YB42AC4D0007-00063C4968299C62-0-0] [lt=3979255][errcode=-4012] execute failed(ret=-4012, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1753073979411448, local_tenant_schema_version=1753073979411448) [2025-08-14 09:06:45.736849] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.720, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:45.736882] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.727, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:45.736898] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.727, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:45.736911] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72851b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.721, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:45.737148] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:06:45.737172] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:45.741007] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:45.741360] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:45.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.742057] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:45.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.743438] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:45.743661] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:45.743684] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:45.743703] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:45.743716] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:45.743729] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:45.743742] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:45.743757] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:45.743770] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:45.743786] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:45.743798] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:45.743809] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=659, timeout ts=09:04:18.815, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:04:09.738], idx=659, next_allow_print_ts="09:04:48.815") [2025-08-14 09:06:45.743825] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:45.743838] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:45.743851] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:45.743862] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:45.743873] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:45.743886] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:45.743900] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:45.743914] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:45.744091] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:45.744105] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:45.747568] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.737, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:45.747612] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:45.749504] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=89] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:41.753212] EDIAG [ELECT] prepare (election_proposer.cpp:380) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=142783765][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3991102, owner: ObLog, click_count: 5, time dist:[FORMAT_END=3991092, ObMalloc#start=2, ObMalloc#end=3, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0xf9d67b1 0xf9d4184 0xfa09ca2 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:18.758931] EDIAG [COMMON] wait (ob_io_define.cpp:880) [701][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4012] LOGGER COST TOO MUCH TIME, cost: 146770180, owner: ObLog, click_count: 7, time dist:[FORMAT_END=6954372, ObMalloc#start=1, ObMalloc#end=5056, ObMalloc#start=139810744, ObMalloc#end=3, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1f9642ff 0x960cf07 0x960bdb0 0x1f9d0af8 0x960b36b 0x960a4e4 0xfa291e5 0xfb97703 0xfa8c926 0xfc0511a 0xfaa843f 0xfab3378 0x9607d79 0xfab2b2c 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:18.758931] WDIAG [COMMON] wait (ob_io_define.cpp:880) [701][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4012] fail to wait result condition(ret=-4012, wait_ms=300000, *this={is_inited_:true, is_finished_:false, is_canceled_:false, has_estimated_:false, complete_size_:0, offset_:659456, size_:4096, timeout_us_:300000000, result_ref_cnt_:2, out_ref_cnt_:1, flag_:{mode:"WRITE", group_id_:0, func_type_:12, wait_event_id_:16, is_sync_:true, is_unlimited_:false, is_detect_:false, is_write_through_:false, is_sealed_:true, is_time_detect_:false, need_close_dev_and_fd_:false, reserved_:0}, ret_code_:{io_ret_:0, fs_errno_:0}, tenant_id_:1001, tenant_io_mgr_:{ptr:0x7f9934554030}, user_data_buf_:null, buf_:0x7f99297ab000, io_callback_:null, time_log:{begin_ts:1755161957768539, enqueue_used:3, dequeue_used:-1, submit_used:-1, return_used:-1, callback_enqueue_used:-1, callback_dequeue_used:-1, callback_finish_used:-1, end_used:-1}}) [2025-08-14 09:04:29.762179] EDIAG [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=26014][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1993702, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1006548, ObMalloc#start=1, ObMalloc#end=987130, ALLOC_END=20, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x1b51d27b 0x986d830 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:45.751438] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:45.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.751999] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:45.753039] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=17] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:45.753072] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:45.753291] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:45.753311] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:45.758328] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:06:45.745258] WDIAG [RPC.OBRPC] in_black (ob_net_keepalive.cpp:257) [677][T1001_Occam][T1001][YB42AC4D0007-00063C4967999C34-0-0] [lt=19953][errcode=-4016] keepalive thread maybe not work, last_write_ts: 1755162235783184 [2025-08-14 09:06:45.761353] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:45.837210] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:45.915385] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=35] PNIO [ratelimit] time: 1755162405915383, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007638, add_bytes: 0 [2025-08-14 09:06:45.937271] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:45.983841] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=19] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:45.994350] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:45.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:46.037331] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:46.137401] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:46.237453] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:46.243667] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a4d58, timer:0x7f9967161f10, task:0x7f994b3f4bf0, task_type:"N9oceanbase7obmysql15ObEliminateTaskE", scheduled_time:1755162405241421, pushed_time:0, delay:500000}, running_task_token={this:0x7f994e1a4e38, timer:0x7f9967161f10, task:0x7f994b3f4c40, task_type:"N9oceanbase7obmysql20ObConstructQueueTaskE", scheduled_time:1755162048238296, pushed_time:1755162404741607, delay:500000}, total_delay=1002244, threashold=1000000, tenant_id=1) [2025-08-14 09:06:46.246060] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:46.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:46.256121] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:46.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:46.337481] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:46.437541] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:46.439092] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fa190, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa470, time_wheel:0x7f99341cdbf0, thread_pool:0x7f99341cd3d0, is_running:1, time_interval:2.00s, expected_run_time:09:06:46.334, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:46.492645] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:46.483, task_priority:1, with_handle_protected:1}) [822]OB_MALLOC COST TOO MUCH TIME, cost_time=969182, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=4557, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=969055, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=969045, ALLOC_CHUNK_END=5], seq:[0, 1] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=968713, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=968696, ALLOC_CHUNK_END=12], seq:[0, 1] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=973129, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=7110, ALLOC_CHUNK_END=965997, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=977880, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=17, ALLOC_CHUNK_END=11757, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=966087, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=974778, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 6, time dist:[ALLOC_CHUNK_END=15, ALLOC_CHUNK_END=8551, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=966185, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=965818, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=965779, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [389]OB_MALLOC COST TOO MUCH TIME, cost_time=969734, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=969725, ALLOC_CHUNK_END=5], seq:[0, 1] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=959215, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=959172, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=951717, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=951682, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=949249, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=949194, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=950390, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=950350, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=949533, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=949452, ALLOC_CHUNK_END=46], seq:[0, 1, 2, 3, 4, 5, 6, 7] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=946385, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=946344, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=933371, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=933328, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=902717, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=902666, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=898750, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=898699, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [656]OB_MALLOC COST TOO MUCH TIME, cost_time=893734, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=893692, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=891166, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=891124, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=874995, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=874945, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [276]OB_MALLOC COST TOO MUCH TIME, cost_time=691337, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=691297, ALLOC_CHUNK_END=6], seq:[0, 1, 2, 3, 4, 5, 6, 7] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=491966, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=491916, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=367985, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=367944, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=110323, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 8, time dist:[ALLOC_CHUNK_END=19, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=110269, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3, 4, 5, 6, 7] [2025-08-14 09:06:46.537612] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:46.637674] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:46.721272] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:46.729641] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=29] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:46.729667] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=26] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:46.729677] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:46.729685] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:46.729692] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:46.729699] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:46.731739] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:46.738169] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=35] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:46.738215] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=42] temporary file module use no blocks [2025-08-14 09:06:46.738223] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:46.738287] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=63] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:46.738300] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623308177467, flush_io_finished_ret_:0, flush_io_finished_round_:26462, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:46.739269] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:46.742249] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:46.742413] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:46.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:46.743263] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:46.714072] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:125) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=1993702] try to adjust log cb pool [397]OB_MALLOC COST TOO MUCH TIME, cost_time=978686, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=965827, WASH_BLOCK_END=12859], seq:[0, 1, 2] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=962022, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=949259, WASH_BLOCK_END=12763], seq:[0, 1, 2] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=964591, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=951728, WASH_BLOCK_END=12863], seq:[0, 1, 2] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=988483, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=974789, WASH_BLOCK_END=13694], seq:[0, 1, 2] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=972836, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=959226, WASH_BLOCK_END=13610], seq:[0, 1, 2] [389]OB_MALLOC COST TOO MUCH TIME, cost_time=983819, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=969741, WASH_BLOCK_END=14078], seq:[0, 1, 2] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=992919, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=977891, WASH_BLOCK_END=15028], seq:[0, 1, 2] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=988600, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=973139, WASH_BLOCK_END=15461], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=984346, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=968721, WASH_BLOCK_END=15624], seq:[0, 1, 2] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=984924, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=969074, WASH_BLOCK_END=15850], seq:[0, 1, 2] [2025-08-14 09:06:45.765272] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:46.745514] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=978759][errcode=0] epoll handler too much time :980251 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:45.782058] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:45.756262] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:46.746287] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=988529][errcode=0] epoll handler too much time :990036 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:45.771980] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:45.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:46.746350] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=972895][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:45.771|207=974.37ms|210=0us|222=1us|total=974.40ms) [2025-08-14 09:06:46.746380] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=-4389] timer task use too much time(end=1755162406746379, start=1755162405771977, delta=974402) [2025-08-14 09:06:46.746394] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=954043, expected_run_ts_us_=1755162405792349, ObClockGenerator::getRealClock()=1755162406746393, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:45.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:46.746414] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:45.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.761391] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:46.746742] WDIAG ev_invoke_pending (ev.c:2136) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=983862][errcode=0] epoll handler too much time :985369 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [822]OB_MALLOC COST TOO MUCH TIME, cost_time=989448, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=969205, WASH_BLOCK_END=20243], seq:[0, 1, 2] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=909581, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=891175, WASH_BLOCK_END=18406], seq:[0, 1, 2] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=893419, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=875005, WASH_BLOCK_END=18414], seq:[0, 1, 2] [656]OB_MALLOC COST TOO MUCH TIME, cost_time=912264, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=893744, WASH_BLOCK_END=18520], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=921355, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=902727, WASH_BLOCK_END=18628], seq:[0, 1, 2] [276]OB_MALLOC COST TOO MUCH TIME, cost_time=709850, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=691347, WASH_BLOCK_END=18503], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=917441, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=898760, WASH_BLOCK_END=18681], seq:[0, 1, 2] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=384230, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=367997, WASH_BLOCK_END=16232], seq:[0, 1, 2] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=965242, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=946397, WASH_BLOCK_END=18845], seq:[0, 1, 2] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=952223, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=933385, WASH_BLOCK_END=18838], seq:[0, 1, 2] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=126583, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=110334, WASH_BLOCK_END=16249], seq:[0, 1, 2] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=969511, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=950408, WASH_BLOCK_END=19103], seq:[0, 1, 2] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=968546, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=949546, WASH_BLOCK_END=19000], seq:[0, 1, 2] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=510805, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=491983, WASH_BLOCK_END=18822], seq:[0, 1, 2] [575]OB_MALLOC COST TOO MUCH TIME, cost_time=993225, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=993223], seq:[0] [575]OB_MALLOC COST TOO MUCH TIME, cost_time=993256, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=993252], seq:[0, 1] [2025-08-14 09:04:31.745907] EDIAG [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:125) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1980686][errcode=0] LOGGER COST TOO MUCH TIME, cost: 135794201, owner: ObLog, click_count: 7, time dist:[FORMAT_END=987751, ObMalloc#start=2, ObMalloc#end=25902, ObMalloc#start=133787278, ObMalloc#end=993266, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x1b51d35d 0x986d8bc 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [374]OB_MALLOC COST TOO MUCH TIME, cost_time=1008114, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1008112], seq:[0] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=1008109, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1008102], seq:[0] [822]OB_MALLOC COST TOO MUCH TIME, cost_time=1014538, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1014529, ALLOC_CHUNK_END=4], seq:[0, 1] [822]OB_MALLOC COST TOO MUCH TIME, cost_time=1014642, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1014555, WASH_BLOCK_END=87], seq:[0, 1, 2] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=1013777, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1013767, ALLOC_CHUNK_END=5], seq:[0, 1] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=1013797, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1013784, WASH_BLOCK_END=13], seq:[0, 1, 2] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=1013178, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1013170, ALLOC_CHUNK_END=4], seq:[0, 1] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=1013196, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1013186, WASH_BLOCK_END=10], seq:[0, 1, 2] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=1013212, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1013202, ALLOC_CHUNK_END=5], seq:[0, 1] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=1013227, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1013219, WASH_BLOCK_END=8], seq:[0, 1, 2] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=1013198, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1013188, ALLOC_CHUNK_END=5], seq:[0, 1] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=1013214, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1013205, WASH_BLOCK_END=9], seq:[0, 1, 2] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=1013229, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1013221, ALLOC_CHUNK_END=4], seq:[0, 1] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=1013245, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1013236, WASH_BLOCK_END=9], seq:[0, 1, 2] [656]OB_MALLOC COST TOO MUCH TIME, cost_time=1013260, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1013251, ALLOC_CHUNK_END=5], seq:[0, 1] [656]OB_MALLOC COST TOO MUCH TIME, cost_time=1013278, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1013268, WASH_BLOCK_END=10], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=1013273, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1013264, ALLOC_CHUNK_END=5], seq:[0, 1] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=1013292, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1013283, WASH_BLOCK_END=9], seq:[0, 1, 2] [276]OB_MALLOC COST TOO MUCH TIME, cost_time=1013332, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1013313, ALLOC_CHUNK_END=15], seq:[0, 1] [276]OB_MALLOC COST TOO MUCH TIME, cost_time=1013351, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1013341, WASH_BLOCK_END=10], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=1013354, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1013345, ALLOC_CHUNK_END=4], seq:[0, 1] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=1013388, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1013361, WASH_BLOCK_END=27], seq:[0, 1, 2] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=1013384, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1013375, ALLOC_CHUNK_END=5], seq:[0, 1] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=1013401, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1013392, WASH_BLOCK_END=9], seq:[0, 1, 2] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=1013406, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1013397, ALLOC_CHUNK_END=4], seq:[0, 1] [390]OB_MALLOC COST TOO MUCH TIME, cost_time=1013422, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1013414, WASH_BLOCK_END=8], seq:[0, 1, 2] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=1013423, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1013414, ALLOC_CHUNK_END=5], seq:[0, 1] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=1013440, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1013431, WASH_BLOCK_END=9], seq:[0, 1, 2] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=1013502, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1013440, ALLOC_CHUNK_END=58], seq:[0, 1] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=1013521, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1013511, WASH_BLOCK_END=10], seq:[0, 1, 2] [2025-08-14 09:06:45.858363] EDIAG [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1907353, owner: ObLog, click_count: 6, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=893427, ObMalloc#start=19, ObMalloc#end=1013251, FREE_END=645], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1f9d7fcf 0x99b8be2 0x1f981833 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:45.786638] EDIAG do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1979150, owner: ObLog, click_count: 6, time dist:[FORMAT_END=13, ObMalloc#start=0, ObMalloc#end=965250, ObMalloc#start=18, ObMalloc#end=1013408, FREE_END=461], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x202c530b 0x202c6888 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:45.782447] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1983363, owner: ObLog, click_count: 6, time dist:[FORMAT_END=9, ObMalloc#start=0, ObMalloc#end=969519, ObMalloc#start=19, ObMalloc#end=1013445, FREE_END=371], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:45.839515] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1926605, owner: ObLog, click_count: 7, time dist:[FORMAT_END=19, ObMalloc#start=1, ObMalloc#end=912272, ObMalloc#start=18, ObMalloc#end=1013284, ALLOC_END=1010, APPEND_END=1], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:45.834450] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1931747, owner: ObLog, click_count: 7, time dist:[FORMAT_END=25, ObMalloc#start=1, ObMalloc#end=917449, ObMalloc#start=18, ObMalloc#end=1013396, ALLOC_END=858, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:46.753000] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1014752, owner: ObLog, click_count: 6, time dist:[FORMAT_END=7, ObMalloc#start=1, ObMalloc#end=1013233, ObMalloc#start=26, ObMalloc#end=44, FREE_END=1440], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:45.842122] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1924380, owner: ObLog, click_count: 8, time dist:[FORMAT_END=20, ObMalloc#start=1, ObMalloc#end=909595, ObMalloc#start=22, ObMalloc#end=1013220, ObMalloc#start=1097, ObMalloc#end=41, FREE_END=384], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:46.752201] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1015809, owner: ObLog, click_count: 6, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=1013803, ObMalloc#start=27, ObMalloc#end=42, FREE_END=1926], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:45.830433] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1936309, owner: ObLog, click_count: 8, time dist:[FORMAT_END=32, ObMalloc#start=1, ObMalloc#end=921362, ObMalloc#start=39, ObMalloc#end=1013298, ObMalloc#start=1069, ObMalloc#end=53, FREE_END=455], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:45.783449] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1983348, owner: ObLog, click_count: 6, time dist:[FORMAT_END=7, ObMalloc#start=1, ObMalloc#end=968554, ObMalloc#start=19, ObMalloc#end=1013527, FREE_END=1239], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:46.752905] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1015479, owner: ObLog, click_count: 8, time dist:[FORMAT_END=21, ObMalloc#start=1, ObMalloc#end=1013202, ObMalloc#start=25, ObMalloc#end=43, ObMalloc#start=1490, ObMalloc#end=58, FREE_END=639], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:45.749058] EDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2018045, owner: ObLog, click_count: 8, time dist:[FORMAT_END=11228, ObMalloc#start=1, ObMalloc#end=989462, ObMalloc#start=23, ObMalloc#end=1014652, ObMalloc#start=2569, ObMalloc#end=41, FREE_END=69], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0xfb622cd 0xfb619cb 0xfb892d3 0x994150a 0x993eecd 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [659]OB_MALLOC COST TOO MUCH TIME, cost_time=1016328, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1016319, ALLOC_CHUNK_END=4], seq:[0, 1] [659]OB_MALLOC COST TOO MUCH TIME, cost_time=1016352, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1016343, WASH_BLOCK_END=9], seq:[0, 1, 2] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=1015256, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1015247, ALLOC_CHUNK_END=4], seq:[0, 1] [858]OB_MALLOC COST TOO MUCH TIME, cost_time=1015272, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1015264, WASH_BLOCK_END=8], seq:[0, 1, 2] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=1014481, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1014466, ALLOC_CHUNK_END=11], seq:[0, 1] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=1014506, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1014492, WASH_BLOCK_END=14], seq:[0, 1, 2] [2025-08-14 09:06:46.755543] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1014715, owner: ObLog, click_count: 6, time dist:[FORMAT_END=8, ObMalloc#start=1, ObMalloc#end=1014514, ObMalloc#start=20, ObMalloc#end=46, FREE_END=126], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:46.753506] EDIAG [OCCAM] operator() (ob_occam_timer.h:85) [659][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 1017563, owner: ObLog, click_count: 8, time dist:[FORMAT_END=19, ObMalloc#start=0, ObMalloc#end=1016360, ObMalloc#start=28, ObMalloc#end=42, ObMalloc#start=986, ObMalloc#end=49, FREE_END=79], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ffa3b 0x9835318 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [372]OB_MALLOC COST TOO MUCH TIME, cost_time=1015853, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1015842, ALLOC_CHUNK_END=5], seq:[0, 1] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=1015882, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1015867, WASH_BLOCK_END=15], seq:[0, 1, 2] [836]OB_MALLOC COST TOO MUCH TIME, cost_time=1014837, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1014824, ALLOC_CHUNK_END=5], seq:[0, 1] [836]OB_MALLOC COST TOO MUCH TIME, cost_time=1014856, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1014845, WASH_BLOCK_END=11], seq:[0, 1, 2] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=1014858, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1014848, ALLOC_CHUNK_END=5], seq:[0, 1] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=1014926, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1014877, WASH_BLOCK_END=49], seq:[0, 1, 2] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=1014852, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1014841, ALLOC_CHUNK_END=6], seq:[0, 1] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=1014881, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1014870, WASH_BLOCK_END=11], seq:[0, 1, 2] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=1014355, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1014344, ALLOC_CHUNK_END=5], seq:[0, 1] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=1014375, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1014363, WASH_BLOCK_END=12], seq:[0, 1, 2] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=991464, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=991454, ALLOC_CHUNK_END=4], seq:[0, 1] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=991485, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=991473, WASH_BLOCK_END=12], seq:[0, 1, 2] [864]OB_MALLOC COST TOO MUCH TIME, cost_time=1013773, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1013762, ALLOC_CHUNK_END=5], seq:[0, 1] [2025-08-14 09:06:46.757023] EDIAG [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1015203, owner: ObLog, click_count: 6, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=1014935, ObMalloc#start=25, ObMalloc#end=40, FREE_END=190], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a1f867 0x27a135c3 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [864]OB_MALLOC COST TOO MUCH TIME, cost_time=1013799, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1013782, WASH_BLOCK_END=17], seq:[0, 1, 2] [800]OB_MALLOC COST TOO MUCH TIME, cost_time=1008395, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1008385, ALLOC_CHUNK_END=5], seq:[0, 1] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=849598, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=849589, ALLOC_CHUNK_END=4], seq:[0, 1] [800]OB_MALLOC COST TOO MUCH TIME, cost_time=1008412, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1008403, WASH_BLOCK_END=9], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=849615, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=849607, WASH_BLOCK_END=7], seq:[0, 1, 2] [404]OB_MALLOC COST TOO MUCH TIME, cost_time=130514, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=130505, ALLOC_CHUNK_END=4], seq:[0, 1] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=933225, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=933214, ALLOC_CHUNK_END=5], seq:[0, 1] [404]OB_MALLOC COST TOO MUCH TIME, cost_time=130529, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=130522, WASH_BLOCK_END=7], seq:[0, 1, 2] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=933245, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=933237, WASH_BLOCK_END=8], seq:[0, 1, 2] [861]OB_MALLOC COST TOO MUCH TIME, cost_time=932750, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=932740, ALLOC_CHUNK_END=5], seq:[0, 1] [861]OB_MALLOC COST TOO MUCH TIME, cost_time=932768, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=932758, WASH_BLOCK_END=10], seq:[0, 1, 2] [733]OB_MALLOC COST TOO MUCH TIME, cost_time=932853, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=932843, ALLOC_CHUNK_END=5], seq:[0, 1] [733]OB_MALLOC COST TOO MUCH TIME, cost_time=932869, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=932861, WASH_BLOCK_END=8], seq:[0, 1, 2] [383]OB_MALLOC COST TOO MUCH TIME, cost_time=1015698, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1015689, ALLOC_CHUNK_END=4], seq:[0, 1] [383]OB_MALLOC COST TOO MUCH TIME, cost_time=1015715, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1015708, WASH_BLOCK_END=7], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=771726, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=771716, ALLOC_CHUNK_END=5], seq:[0, 1] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=771751, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=771740, WASH_BLOCK_END=11], seq:[0, 1, 2] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=887334, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=887324, ALLOC_CHUNK_END=5], seq:[0, 1] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=887355, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=887343, WASH_BLOCK_END=12], seq:[0, 1, 2] [2025-08-14 09:06:46.757095] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1015887, owner: ObLog, click_count: 6, time dist:[FORMAT_END=5, ObMalloc#start=0, ObMalloc#end=1015725, ObMalloc#start=27, ObMalloc#end=43, FREE_END=87], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [393]OB_MALLOC COST TOO MUCH TIME, cost_time=1015941, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=1015933, ALLOC_CHUNK_END=5], seq:[0, 1] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=1015967, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1015952, WASH_BLOCK_END=15], seq:[0, 1, 2] [2025-08-14 09:06:46.757054] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1016164, owner: ObLog, click_count: 6, time dist:[FORMAT_END=6, ObMalloc#start=1, ObMalloc#end=1015974, ObMalloc#start=20, ObMalloc#end=37, FREE_END=126], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:46.757733] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1016162, owner: ObLog, click_count: 6, time dist:[FORMAT_END=8, ObMalloc#start=0, ObMalloc#end=1014382, ObMalloc#start=19, ObMalloc#end=41, FREE_END=1712], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:46.757122] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1016827, owner: ObLog, click_count: 6, time dist:[FORMAT_END=4, ObMalloc#start=0, ObMalloc#end=1014889, ObMalloc#start=20, ObMalloc#end=1735, FREE_END=178], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:46.758438] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1015583, owner: ObLog, click_count: 8, time dist:[FORMAT_END=14, ObMalloc#start=0, ObMalloc#end=1013807, ObMalloc#start=19, ObMalloc#end=37, FREE_END=1706, ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=68], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:46.729709] EDIAG [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1044385, owner: ObLog, click_count: 8, time dist:[ALLOC_BLOCK#start=2, ALLOC_BLOCK#end=1, FORMAT_END=34188, ObMalloc#start=1, ObMalloc#end=1008422, ObMalloc#start=26, ObMalloc#end=44, FREE_END=1701], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x99721a1 0x997a72e 0x2006ed17 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:46.545644] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=958721][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1229595, owner: ObLog, click_count: 8, time dist:[FORMAT_END=211051, ObMalloc#start=1, ObMalloc#end=1014864, ObMalloc#start=26, ObMalloc#end=53, ObMalloc#start=3223, ObMalloc#end=288, FREE_END=89], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:46.244279] EDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] LOGGER COST TOO MUCH TIME, cost: 2467248, owner: ObLog, click_count: 7, time dist:[FORMAT_END=16, ObMalloc#start=0, ObMalloc#end=510820, ObMalloc#start=20, ObMalloc#end=1015894, ALLOC_END=940498, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x13b06ffb 0x958a39e 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [374]OB_MALLOC COST TOO MUCH TIME, cost_time=1964978, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1008129], seq:[0, 1] [2025-08-14 09:06:45.757659] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2953650, owner: ObLog, click_count: 7, time dist:[FORMAT_END=23, ObMalloc#start=1, ObMalloc#end=988610, ObMalloc#start=20, ObMalloc#end=1964994, ALLOC_END=2, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [772]OB_MALLOC COST TOO MUCH TIME, cost_time=1965057, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1008115], seq:[0, 1] [2025-08-14 09:06:46.734917] EDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1076) [368][VTblLocAsyncUp0][T1][YB42AC4D0007-00063C4968299C62-0-0] [lt=1974742][errcode=-4012] LOGGER COST TOO MUCH TIME, cost: 1978236, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1978229, ObMalloc#start=1, ObMalloc#end=6, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x12ead58e 0x981b5d1 0x124ddd52 0x96a5a41 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x9a77e58 0x1fcfa594 0x1f7fa1c3 0x2023397e 0x202381d6 0x13ad9dbd 0x13ad61b5 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [396]OB_MALLOC COST TOO MUCH TIME, cost_time=1992396, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1992390], seq:[0] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=1992418, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1992411], seq:[0, 1] [2025-08-14 09:06:45.761954] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2976448, owner: ObLog, click_count: 7, time dist:[FORMAT_END=24, ObMalloc#start=0, ObMalloc#end=984354, ObMalloc#start=18, ObMalloc#end=1965066, ALLOC_END=26985, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:45.753028] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2985405, owner: ObLog, click_count: 9, time dist:[FORMAT_END=8, ObMalloc#start=1, ObMalloc#end=992930, ObMalloc#start=28, ObMalloc#end=1992433, ALLOC_END=4, APPEND_END=1, ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=2], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:45.761436] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:46.747888] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=984961][errcode=0] epoll handler too much time :986457 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:46.734837] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=995492] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:06:45.799705] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=21] gc stale ls task succ [2025-08-14 09:06:46.368514] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=21] gc stale ls task succ [2025-08-14 09:06:46.662423] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EED5-0-0] [lt=36] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:06:46.757050] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:46.760917] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:46.764645] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:46.799771] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=953394] gc stale ls task succ [2025-08-14 09:06:47.369252] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=386455] gc stale ls task succ [2025-08-14 09:06:47.742336] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:47.743262] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:04:31.745907] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:125) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1980686] try to adjust log cb pool [2025-08-14 09:06:47.746302] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=962065] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:47.758548] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:47.761923] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:47.766537] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:45.839515] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:06:45.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:47.769067] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=1926605][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:45.839|207=1.93s|210=0us|222=2us|total=1.93s) [2025-08-14 09:06:47.769095] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=-4389] timer task use too much time(end=1755162407769094, start=1755162405839513, delta=1929581) [2025-08-14 09:06:45.834450] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:45.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:47.769136] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1931747][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:45.834|207=1.93s|210=1us|222=1us|total=1.93s) [2025-08-14 09:06:47.769145] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] timer task use too much time(end=1755162407769145, start=1755162405834448, delta=1934697) [2025-08-14 09:06:47.643227] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=174604][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 26.898604 loop:26707, slowcnt: 259 [2025-08-14 09:06:48.369314] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=19] gc stale ls task succ [2025-08-14 09:06:46.244279] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:48.715324] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=2467248][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:48.715345] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:45.757659] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.715800] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=2953650][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:45.757|207=2.96s|210=0us|222=2us|total=2.96s) [2025-08-14 09:06:48.715817] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] timer task use too much time(end=1755162408715817, start=1755162405757656, delta=2958161) [2025-08-14 09:06:48.715827] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=2967101, expected_run_ts_us_=1755162405748725, ObClockGenerator::getRealClock()=1755162408715826, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891807210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.715842] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891807210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:46.734917] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1076) [368][VTblLocAsyncUp0][T1][YB42AC4D0007-00063C4968299C62-0-0] [lt=1974742][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4012, err_:"OB_TIMEOUT", retry_type:0, client_ret:-4012}, need_retry=false) [2025-08-14 09:06:47.799848] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=15] gc stale ls task succ [2025-08-14 09:06:47.869173] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:06:47.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:47.969271] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:47.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:48.674553] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=130605][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 27.929932 loop:26707, slowcnt: 268 [2025-08-14 09:06:45.761954] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:45.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:45.753028] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:48.742996] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2985405][errcode=0] epoll handler too much time :2989977 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:48.742993] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=2976448][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:45.761|207=2.98s|210=1us|222=2us|total=2.98s) [2025-08-14 09:06:48.743012] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:48.743023] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:48.743037] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=17080] finish to resize log external storage handler(current_ts=1755162407788339, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-08-14 09:06:48.743058] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=18] snapshot_traffic_info once [2025-08-14 09:06:48.743014] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task use too much time(end=1755162408743014, start=1755162405761952, delta=2981062) [2025-08-14 09:06:48.743120] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=103][errcode=0] task scheduled too delay(scheduled_delay=2983985, expected_run_ts_us_=1755162405759134, ObClockGenerator::getRealClock()=1755162408743120, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:45.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:48.743138] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:45.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:48.743153] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=2653837, expected_run_ts_us_=1755162406089316, ObClockGenerator::getRealClock()=1755162408743153, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:46.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:48.743166] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:46.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:48.743182] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=2538884, expected_run_ts_us_=1755162406204297, ObClockGenerator::getRealClock()=1755162408743181, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:46.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:48.743196] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:46.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:48.743288] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc190, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc470, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:10.00s, expected_run_time:09:06:48.720, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:48.743325] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:48.715853] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=2966265, expected_run_ts_us_=1755162405749587, ObClockGenerator::getRealClock()=1755162408715853, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744111] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28211][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744154] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=37][errcode=0] task scheduled too delay(scheduled_delay=2994579, expected_run_ts_us_=1755162405749573, ObClockGenerator::getRealClock()=1755162408744153, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744170] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744196] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=2984552, expected_run_ts_us_=1755162405759643, ObClockGenerator::getRealClock()=1755162408744195, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d67b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744210] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d67b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744225] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=2961439, expected_run_ts_us_=1755162405782785, ObClockGenerator::getRealClock()=1755162408744225, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891807090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744257] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891807090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744271] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=2957101, expected_run_ts_us_=1755162405787169, ObClockGenerator::getRealClock()=1755162408744271, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739aa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744285] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739aa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744299] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=2952692, expected_run_ts_us_=1755162405791606, ObClockGenerator::getRealClock()=1755162408744299, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.791, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744313] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.791, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744415] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=100][errcode=0] task scheduled too delay(scheduled_delay=2953041, expected_run_ts_us_=1755162405791373, ObClockGenerator::getRealClock()=1755162408744415, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ecf10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.791, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744428] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ecf10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.791, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744441] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=2952416, expected_run_ts_us_=1755162405792025, ObClockGenerator::getRealClock()=1755162408744441, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744453] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744466] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=2945137, expected_run_ts_us_=1755162405799328, ObClockGenerator::getRealClock()=1755162408744465, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.799, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744479] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.799, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744492] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=2943247, expected_run_ts_us_=1755162405801244, ObClockGenerator::getRealClock()=1755162408744492, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744504] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744517] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=2940750, expected_run_ts_us_=1755162405803766, ObClockGenerator::getRealClock()=1755162408744517, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7351e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196f990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.803, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744529] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7351e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196f990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.803, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744551] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] task scheduled too delay(scheduled_delay=2940484, expected_run_ts_us_=1755162405804067, ObClockGenerator::getRealClock()=1755162408744551, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a22b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744564] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a22b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744576] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=2937307, expected_run_ts_us_=1755162405807269, ObClockGenerator::getRealClock()=1755162408744576, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744588] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744637] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] task scheduled too delay(scheduled_delay=2917697, expected_run_ts_us_=1755162405826923, ObClockGenerator::getRealClock()=1755162408744637, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ebe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.826, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744650] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ebe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.826, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744662] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=2919904, expected_run_ts_us_=1755162405824757, ObClockGenerator::getRealClock()=1755162408744662, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7350530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ea90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.824, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744673] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7350530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ea90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.824, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744685] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=2878691, expected_run_ts_us_=1755162405865994, ObClockGenerator::getRealClock()=1755162408744685, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fbbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744700] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fbbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744712] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=2871232, expected_run_ts_us_=1755162405873480, ObClockGenerator::getRealClock()=1755162408744712, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72860b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.873, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744739] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72860b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:45.873, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744758] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=2415556, expected_run_ts_us_=1755162406329201, ObClockGenerator::getRealClock()=1755162408744757, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7faa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744769] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7faa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744782] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=2415041, expected_run_ts_us_=1755162406329741, ObClockGenerator::getRealClock()=1755162408744782, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72842b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744800] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72842b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744823] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=2130491, expected_run_ts_us_=1755162406614331, ObClockGenerator::getRealClock()=1755162408744823, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eef90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.614, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744834] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eef90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.614, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744846] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=2130027, expected_run_ts_us_=1755162406614819, ObClockGenerator::getRealClock()=1755162408744846, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eaf30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.614, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744859] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eaf30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.614, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744873] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1995507, expected_run_ts_us_=1755162406749366, ObClockGenerator::getRealClock()=1755162408744873, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744884] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744896] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=2003645, expected_run_ts_us_=1755162406741251, ObClockGenerator::getRealClock()=1755162408744896, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744908] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744921] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=1994971, expected_run_ts_us_=1755162406749950, ObClockGenerator::getRealClock()=1755162408744921, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919827b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744933] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919827b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744945] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1988583, expected_run_ts_us_=1755162406756361, ObClockGenerator::getRealClock()=1755162408744945, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744956] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744968] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1982999, expected_run_ts_us_=1755162406761969, ObClockGenerator::getRealClock()=1755162408744968, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72742b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744981] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72742b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.744993] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1979731, expected_run_ts_us_=1755162406765261, ObClockGenerator::getRealClock()=1755162408744993, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919840b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.745004] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919840b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.745016] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1983389, expected_run_ts_us_=1755162406761626, ObClockGenerator::getRealClock()=1755162408745016, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180ef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.745033] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180ef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.745054] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] task scheduled too delay(scheduled_delay=1983271, expected_run_ts_us_=1755162406761782, ObClockGenerator::getRealClock()=1755162408745054, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180ecb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.745065] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180ecb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.745077] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1977932, expected_run_ts_us_=1755162406767145, ObClockGenerator::getRealClock()=1755162408745077, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.767, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.745089] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.767, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.745101] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1983139, expected_run_ts_us_=1755162406761961, ObClockGenerator::getRealClock()=1755162408745101, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.745114] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.745127] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1975007, expected_run_ts_us_=1755162406770119, ObClockGenerator::getRealClock()=1755162408745127, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919925b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196d890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.745138] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919925b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196d890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:47.757513] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=60834] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:06:48.745151] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1966736, expected_run_ts_us_=1755162406778414, ObClockGenerator::getRealClock()=1755162408745151, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.745163] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.745176] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=1968114, expected_run_ts_us_=1755162406777062, ObClockGenerator::getRealClock()=1755162408745176, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.777, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.745187] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.777, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.745200] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1957111, expected_run_ts_us_=1755162406788089, ObClockGenerator::getRealClock()=1755162408745200, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728ca30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7291590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.745211] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728ca30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7291590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.742554] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=772112][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:47.969|207=773.28ms|210=1us|222=1us|total=773.29ms) [2025-08-14 09:06:48.745623] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=3063][errcode=-4389] timer task use too much time(end=1755162408745622, start=1755162407969268, delta=776354) [2025-08-14 09:06:48.742412] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=33] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:04:35.771940] EDIAG [ARCHIVE] produce_log_fetch_task_ (ob_archive_sequencer.cpp:166) [731][T1001_ArcSeq][T1001][YB42AC4D0007-00063C495C999C16-0-0] [lt=50439][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1969932, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1969931, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x113df1a7 0x9ac5c05 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:48.745667] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=3238] temporary file module use no blocks [2025-08-14 09:06:48.745682] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=15] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:48.745697] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:48.745710] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623310178309, flush_io_finished_ret_:0, flush_io_finished_round_:26464, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:48.742527] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=872030][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:47.869|207=873.35ms|210=1us|222=1us|total=873.37ms) [2025-08-14 09:06:48.745758] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=3225][errcode=-4389] timer task use too much time(end=1755162408745757, start=1755162407869171, delta=876586) [2025-08-14 09:06:48.742675] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=315][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8030, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8310, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:10.00s, expected_run_time:09:06:48.730, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:48.745906] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=3222][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:48.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:48.745970] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:785) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=2018045] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:1604090090}, pending_end_lsn={lsn:1604090331}) [2025-08-14 09:06:48.747084] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [807][T1002_TimerWK3][T1002][YB42AC4D0007-00063C495F8A1983-0-0] [lt=15][errcode=-4389] timer task too much delay in priority queue(token={this:0x7f991b7a4c78, timer:0x7f99329e1b10, task:0x7f98e4e091f8, task_type:"N9oceanbase5table21ObTTLAllTaskSchedulerE", scheduled_time:1755162384727446, pushed_time:1755162388738461, delay:15000000}, delay_in_priority_queue=4011015, threashold=1000000) [2025-08-14 09:06:48.747111] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][YB42AC4D0007-00063C495F8A1983-0-0] [lt=26][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a4c78, timer:0x7f99329e1b10, task:0x7f98e4e091f8, task_type:"N9oceanbase5table21ObTTLAllTaskSchedulerE", scheduled_time:1755162384727446, pushed_time:1755162388738461, delay:15000000}, delay_in_thread_pool=16994019, threashold=500000) [2025-08-14 09:06:48.747143] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [807][T1002_TimerWK3_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=11] ====== check clog disk timer task ====== [2025-08-14 09:06:47.747274] INFO [STORAGE.TRANS] refresh_elr_tenant_config_ (ob_tx_elr_util.cpp:42) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=30830] refresh tenant config success(tenant_id=1001, *this={last_refresh_ts:1755162407746447, can_tenant_elr:true}) [2025-08-14 09:06:48.715201] WDIAG [SHARE.PT] batch_get (ob_persistent_ls_table.cpp:864) [360][UserLocAsyncUp0][T0][YB42AC4D0007-00063C495D499F18-0-0] [lt=1006295][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", exec_tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 AND ls_id IN (1001) ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-08-14 09:06:48.748790] WDIAG [SHARE.PT] batch_get (ob_ls_table_operator.cpp:399) [360][UserLocAsyncUp0][T0][YB42AC4D0007-00063C495D499F18-0-0] [lt=33533][errcode=-4012] get all ls info by persistent_ls_ failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1002) [2025-08-14 09:04:33.742986] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [869][T1002_MultiVers][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:48.749425] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1769] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:48.750453] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=14] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:48.745224] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1949711, expected_run_ts_us_=1755162406795512, ObClockGenerator::getRealClock()=1755162408745224, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739ee10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.750530] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=5296][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739ee10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.750562] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] task scheduled too delay(scheduled_delay=1945897, expected_run_ts_us_=1755162406804664, ObClockGenerator::getRealClock()=1755162408750561, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ebd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.750576] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ebd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.750600] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] task scheduled too delay(scheduled_delay=1945887, expected_run_ts_us_=1755162406804712, ObClockGenerator::getRealClock()=1755162408750599, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739ec90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.750612] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739ec90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.750632] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] task scheduled too delay(scheduled_delay=1933865, expected_run_ts_us_=1755162406816767, ObClockGenerator::getRealClock()=1755162408750632, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72874b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.750645] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72874b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.750658] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1928073, expected_run_ts_us_=1755162406822584, ObClockGenerator::getRealClock()=1755162408750658, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.750670] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.750682] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1927873, expected_run_ts_us_=1755162406822809, ObClockGenerator::getRealClock()=1755162408750682, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739dc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.750694] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739dc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.750708] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1855533, expected_run_ts_us_=1755162406895174, ObClockGenerator::getRealClock()=1755162408750707, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a59b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.895, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.750719] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a59b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:46.895, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.750743] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1024314, expected_run_ts_us_=1755162407726429, ObClockGenerator::getRealClock()=1755162408750743, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:47.726, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.750755] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:47.726, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.750768] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1029617, expected_run_ts_us_=1755162407721150, ObClockGenerator::getRealClock()=1755162408750767, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739eb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:47.721, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.750780] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739eb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:47.721, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.750792] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1013799, expected_run_ts_us_=1755162407736992, ObClockGenerator::getRealClock()=1755162408750792, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:47.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.750804] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:47.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.750817] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1000374, expected_run_ts_us_=1755162407750442, ObClockGenerator::getRealClock()=1755162408750817, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:47.750, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.750829] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:47.750, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.750841] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=995472, expected_run_ts_us_=1755162407755368, ObClockGenerator::getRealClock()=1755162408750841, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:47.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.750853] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:47.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.750865] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=996131, expected_run_ts_us_=1755162407754733, ObClockGenerator::getRealClock()=1755162408750864, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d80b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:47.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.750876] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d80b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:47.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.750889] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=996010, expected_run_ts_us_=1755162407754878, ObClockGenerator::getRealClock()=1755162408750889, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:47.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.750900] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:47.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.750912] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=992815, expected_run_ts_us_=1755162407758096, ObClockGenerator::getRealClock()=1755162408750912, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891992330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196d710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:47.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.750923] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891992330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196d710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:47.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.750935] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=998270, expected_run_ts_us_=1755162407752664, ObClockGenerator::getRealClock()=1755162408750935, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:47.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.750946] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:47.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.750989] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=989900, expected_run_ts_us_=1755162407761089, ObClockGenerator::getRealClock()=1755162408750989, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:47.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.751001] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:47.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.751014] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=990992, expected_run_ts_us_=1755162407760021, ObClockGenerator::getRealClock()=1755162408751014, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:47.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.751025] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:47.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.751037] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=979169, expected_run_ts_us_=1755162407771868, ObClockGenerator::getRealClock()=1755162408751037, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:47.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.751049] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:47.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.751068] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] task scheduled too delay(scheduled_delay=975031, expected_run_ts_us_=1755162407776036, ObClockGenerator::getRealClock()=1755162408751068, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72751b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:47.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.751079] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72751b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:47.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.751092] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=974956, expected_run_ts_us_=1755162407776136, ObClockGenerator::getRealClock()=1755162408751092, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d31b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:47.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.751104] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d31b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:47.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.751116] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=977166, expected_run_ts_us_=1755162407773949, ObClockGenerator::getRealClock()=1755162408751116, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73959b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:47.773, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.751127] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73959b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:47.773, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.751139] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:48.752740] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=22][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:48.754358] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=2389] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:48.754434] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1433] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:48.754487] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=529] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:48.754512] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=244] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:48.754566] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=152] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:48.755628] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=20] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:48.755720] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=189] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:48.715355] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:48.759585] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=226] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:48.759919] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:48.760828] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:48.755757] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=414] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:06:48.755895] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [561][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=30] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=16, total_destructed_detector=16, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f994b3ee2b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-08-14 09:06:48.755917] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=40494][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:48.762385] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6451][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:48.762409] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:48.762427] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:48.762446] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:48.762461] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:48.762481] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:48.762499] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:48.762513] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:48.762528] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:48.762543] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:48.762556] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:48.762573] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:48.762589] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:48.762752] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:48.762779] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:48.748429] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=999627] get_max_applied_scn(scn={val:1755161719775179002, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:45, palf_committed_end_lsn_:{lsn:8461098908}, palf_committed_end_scn_:{val:1755161719775179002, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161719775179002, v:0}}) [2025-08-14 09:06:46.780830] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=14904][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162406734826], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:06:46.780830] EDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=14904][errcode=-4038] LOGGER COST TOO MUCH TIME, cost: 1979941, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1979930, ObMalloc#start=0, ObMalloc#end=4, ALLOC_END=0, APPEND_END=7], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1b4fd0db 0x1b4fc327 0x1b5014e0 0x1b5009d1 0x1b50387c 0x1b8905de 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:48.763799] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1979941][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1001, srr:[mts=1755162406734826], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:06:48.743210] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=135794201] get_max_applied_scn(scn={val:1755161547738516001, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:50, palf_committed_end_lsn_:{lsn:5282737655}, palf_committed_end_scn_:{val:1755161547738516001, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161547738516001, v:0}}) [2025-08-14 09:06:48.764683] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:48.764784] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [807][T1002_TimerWK3_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=10] get_disk_usage(ret=0, capacity(MB):=24883, used(MB):=8642) [2025-08-14 09:06:46.042320] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc56f8, timer:0x7f9970f40f80, task:0x564d6ef58bc8, task_type:"N9oceanbase7storage24ObPrintTenantMemoryUsageE", scheduled_time:1755161623782823, pushed_time:0, delay:10000000}, running_task_token={this:0x7f9970fc5fb8, timer:0x7f9970f40f80, task:0x564d727182f0, task_type:"N9oceanbase5share16ObAshRefreshTaskE", scheduled_time:1755161611717613, pushed_time:1755161614757742, delay:120000000}, total_delay=782259495, threashold=1000000, tenant_id=500) [2025-08-14 09:06:46.042320] EDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 2718630, owner: ObLog, click_count: 7, time dist:[FORMAT_END=25, ObMalloc#start=1, ObMalloc#end=709857, ObMalloc#start=18, ObMalloc#end=1013356, ALLOC_END=995367, APPEND_END=6], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x9757ced 0x97569df 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:48.765088] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=2718630][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc4818, timer:0x7f9970f40f80, task:0x564d6eb457d0, task_type:"N9oceanbase3sql23ObDASTaskResultGCRunnerE", scheduled_time:1755161623784040, pushed_time:0, delay:10000000}, running_task_token={this:0x7f9970fc5fb8, timer:0x7f9970f40f80, task:0x564d727182f0, task_type:"N9oceanbase5share16ObAshRefreshTaskE", scheduled_time:1755161611717613, pushed_time:1755161614757742, delay:120000000}, total_delay=784981047, threashold=1000000, tenant_id=500) [2025-08-14 09:06:48.766697] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:48.769769] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=1979150] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:48.770170] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=335][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:48.772972] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=310][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:48.773223] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=327][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:48.775734] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=1015583] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:06:48.776957] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=368] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:06:48.776980] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:48.777415] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=495] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:48.777435] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16] temporary file module use no blocks [2025-08-14 09:06:48.777444] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:48.777456] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:48.777469] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623310217156, flush_io_finished_ret_:0, flush_io_finished_round_:26426, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:48.782610] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=279] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:48.782779] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=187] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:48.799910] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=941207] gc stale ls task succ [2025-08-14 09:06:48.845700] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:48.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:48.845816] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:48.876481] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=201] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:48.876541] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:48.876592] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:48.876641] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:48.876693] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:48.876707] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:48.876720] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:48.877017] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:48.938215] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=195] PNIO [ratelimit] time: 1755162408938213, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007625, add_bytes: 0 [2025-08-14 09:06:48.945763] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:48.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:48.965258] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc5378, timer:0x7f9970ecbf90, task:0x564d6cd25100, task_type:"N9oceanbase10rootserver23ObIngressBWAllocServiceE", scheduled_time:1755162167753034, pushed_time:0, delay:3000000}, running_task_token={this:0x7f9970fc5458, timer:0x7f9970ecbf90, task:0x564d6cd25880, task_type:"N9oceanbase10rootserver18ObSSNTAllocServiceE", scheduled_time:1755162050765815, pushed_time:1755162164753047, delay:3000000}, total_delay=241212222, threashold=1000000, tenant_id=500) [2025-08-14 09:06:48.977066] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:48.995047] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:48.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.016150] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:49.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.016189] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:49.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.021854] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:49.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.045827] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:48.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.073175] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=2][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:48.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.076410] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=210] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:49.077125] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:49.096559] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:49.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.145890] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:49.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.173212] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:49.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.173440] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:49.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.177164] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:49.217184] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:49.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.245955] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:49.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.246617] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:49.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.256679] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:49.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.259747] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.248, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:49.262949] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:49.262985] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:49.263005] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:49.263020] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:49.263043] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:49.263056] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:49.263072] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:49.263094] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:49.263111] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:49.263122] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:49.263140] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:49.263162] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:49.263175] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:49.263187] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:49.263199] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:49.263212] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:49.263225] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:49.263269] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:49.263455] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:49.263470] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:49.267434] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:49.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.267470] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:49.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.273262] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=56][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:49.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.273641] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:49.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.277225] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:49.303833] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:49.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.346018] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:49.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.369347] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=15] gc stale ls task succ [2025-08-14 09:06:49.373326] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:49.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.373528] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:49.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.377245] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:49.446082] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:49.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.473388] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:49.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.477307] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:49.498012] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:49.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.518693] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:49.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.518725] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:49.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.525247] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:49.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.546145] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:49.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.573450] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:49.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.573652] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:49.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.577353] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:49.599084] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:49.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.646209] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:49.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.673514] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:49.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.676784] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EED7-0-0] [lt=93289] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:06:49.677417] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:49.719751] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:49.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.721261] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=448] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:49.740943] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:49.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.728316] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=215] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:49.741296] WDIAG [SQL] do_close_plan (ob_result_set.cpp:844) [368][VTblLocAsyncUp0][T1][YB42AC4D0007-00063C4968299C62-0-0] [lt=1978236][errcode=-4006] exec result is null(ret=-4006) [2025-08-14 09:06:49.741321] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12984][errcode=0] epoll handler too much time :13024 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:49.717951] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6062] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:06:49.741373] INFO [COMMON] loop_running_dag_net_list (ob_tenant_dag_scheduler.cpp:3845) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=23380] loop running dag net list(ret=0, running_dag_net_list_size=0, slow_dag_net_cnt=0, finished_dag_net_list_size=0) [2025-08-14 09:06:49.741394] INFO [COMMON] loop_finished_dag_net_list (ob_tenant_dag_scheduler.cpp:3868) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=21] loop finsihed dag net list(ret=0, finished_dag_net_list_size=0) [2025-08-14 09:06:47.747327] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:140) [14963][T1_TimerWK9_CompactionRefre][T1][YB42AC4D0007-00063C4978A99C32-0-0] [lt=0] InfoPoolResizeTask(cost_ts=1000323) [2025-08-14 09:06:49.742262] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7270030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a3b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.735, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:49.742297] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ecb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.739, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:49.742319] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7353730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7396990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:49.742333] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:49.743105] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=67911][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 28.998484 loop:26707, slowcnt: 278 [2025-08-14 09:06:49.743274] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:49.743998] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:49.745806] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=23] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:49.745833] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=22] temporary file module use no blocks [2025-08-14 09:06:49.745843] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:49.745856] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:49.745880] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=23] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623311185501, flush_io_finished_ret_:0, flush_io_finished_round_:26465, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:49.746243] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:49.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.750556] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:49.750991] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:49.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.751400] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=16] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:48.764098] WDIAG [SHARE.LOCATION] batch_renew_ls_locations (ob_ls_location_service.cpp:1293) [360][UserLocAsyncUp0][T0][YB42AC4D0007-00063C495D499F18-0-0] [lt=16][errcode=-4664] the sql used to get ls locations error, treat as timeout(ret=-4664, ret="OB_GET_LOCATION_TIME_OUT", previous_ret=-4012, ls_ids=[{id:1001}]) [2025-08-14 09:06:49.752644] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a5ed8, timer:0x7f99200ff180, task:0x7f99200ff1d8, task_type:"N9oceanbase8observer18ObTableLoadService8ObGCTaskE", scheduled_time:1755162408748371, pushed_time:0, delay:30000000}, running_task_token={this:0x7f991b7a5d18, timer:0x7f99200ff180, task:0x7f99200ff1a8, task_type:"N9oceanbase8observer18ObTableLoadService17ObCheckTenantTaskE", scheduled_time:1755162377761774, pushed_time:1755162392751930, delay:1000000}, total_delay=1004271, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:49.752702] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:49.753332] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:49.753689] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:49.753740] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:49.753854] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ea30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:48.745944] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:582) [677][T1001_Occam][T1001][YB42AC4D0007-00063C4967999C34-0-0] [lt=16051] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=1024, Tenant Frozen TxData Memory(MB)=0, Tenant Active TxData Memory(MB)=0, Freeze TxData Trigger Memory(MB)=51, Total TxDataTable Hold Memory(MB)=5, Total TxDataTable Memory Limit(MB)=204) [2025-08-14 09:06:49.741329] WDIAG [SQL] do_close (ob_result_set.cpp:935) [368][VTblLocAsyncUp0][T1][YB42AC4D0007-00063C4968299C62-0-0] [lt=32][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-08-14 09:06:49.754401] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:49.741846] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:213) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-08-14 09:06:49.754963] INFO [COMMON] handle (memory_dump.cpp:544) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13095] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-08-14 09:06:49.756028] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:49.756066] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=21] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:49.757204] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:49.757849] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:48.763829] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162406734826]}) [2025-08-14 09:06:49.761851] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:49.763359] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] update_check_range(min_check_version=2540, max_check_version=2540, global_version=2650) [772]OB_MALLOC COST TOO MUCH TIME, cost_time=949899, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=3528], seq:[0, 1] [2025-08-14 09:06:48.716182] EDIAG LIBalloc (ob_malloc_allocator.cpp:924) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4013] LOGGER COST TOO MUCH TIME, cost: 1049735, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1047352, ALLOC_END=2382, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0xf9b6d19 0xf9b6367 0xf9b6058 0xf989dfb 0x973c6f6 0x973c18d 0x7f9973af5dfa 0x7f9973af5efd 0x260b8434 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [807]OB_MALLOC COST TOO MUCH TIME, cost_time=956916, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=956908], seq:[0] [2025-08-14 09:06:50.077737] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=31] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:50.451525] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=396][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fa190, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa470, time_wheel:0x7f99341cdbf0, thread_pool:0x7f99341cd3d0, is_running:1, time_interval:2.00s, expected_run_time:09:06:50.334, task_priority:1, with_handle_protected:1}) [830]OB_MALLOC COST TOO MUCH TIME, cost_time=954021, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=954019], seq:[0] [807]OB_MALLOC COST TOO MUCH TIME, cost_time=958649, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=956931], seq:[0, 1] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=962557, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=962555], seq:[0] [830]OB_MALLOC COST TOO MUCH TIME, cost_time=965457, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=954037], seq:[0, 1] [2025-08-14 09:06:49.765397] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=55][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:49.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:50.716789] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=949942][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:49.765|207=951.38ms|210=0us|222=1us|total=951.42ms) [2025-08-14 09:06:50.716816] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4389] timer task use too much time(end=1755162410716816, start=1755162409765396, delta=951420) [2025-08-14 09:06:50.716828] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=957692, expected_run_ts_us_=1755162409759134, ObClockGenerator::getRealClock()=1755162410716827, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:49.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:50.716842] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:49.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:50.716857] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=627541, expected_run_ts_us_=1755162410089316, ObClockGenerator::getRealClock()=1755162410716857, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:50.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:50.721861] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=4994][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:50.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:50.721893] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] task scheduled too delay(scheduled_delay=517595, expected_run_ts_us_=1755162410204297, ObClockGenerator::getRealClock()=1755162410721892, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:50.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:50.721905] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:50.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:50.253839] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=295][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:50.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:49.763566] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a5298, timer:0x7f9922ffd5d0, task:0x7f9922ffd618, task_type:"N9oceanbase7storage10checkpoint19ObCheckPointService24ObCheckClogDiskUsageTaskE", scheduled_time:1755162389769894, pushed_time:1755162389770207, delay:2000000}, delay_in_thread_pool=18976925, threashold=500000) [2025-08-14 09:06:50.728322] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [374]OB_MALLOC COST TOO MUCH TIME, cost_time=981510, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=3503, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=962581], seq:[0, 1, 2, 3, 4] [2025-08-14 09:06:50.716969] WARN [LIB] alloc (ob_malloc_allocator.cpp:924) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=1049735][errcode=-4013] [OOPS]: alloc failed reason is that server memory has reached the upper limit(server_hold: 4294295552, server_limit: 4294967296, alloc_size: 2097152) [2025-08-14 09:06:49.752212] WDIAG [SHARE] get_ls_status_ (ob_ls_status_operator.cpp:1062) [830][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4012] fail to inner get ls status info(ret=-4012, ret="OB_TIMEOUT", sql=SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1002, tenant_id=1002, exec_tenant_id=1001, need_member_list=false) [2025-08-14 09:06:50.744486] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:49.765511] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919afbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:50.748569] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=981555][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:49.765|207=983.04ms|210=1us|222=3us|total=983.08ms) [2025-08-14 09:06:50.748607] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4389] timer task use too much time(end=1755162410748605, start=1755162409765510, delta=983095) [2025-08-14 09:06:50.748619] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=990086, expected_run_ts_us_=1755162409758532, ObClockGenerator::getRealClock()=1755162410748618, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919854b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:50.748636] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919854b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:50.735153] WDIAG [SHARE] get_ls_status_info (ob_ls_status_operator.cpp:626) [830][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=981433][errcode=-4012] failed to get ls status(ret=-4012, ret="OB_TIMEOUT", id={id:1}, tenant_id=1002) [2025-08-14 09:06:50.722101] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1014) [360][UserLocAsyncUp0][T0][YB42AC4D0007-00063C495D499F18-0-0] [lt=986096][errcode=-4664] batch renew ls locations failed(ret=-4664, ret="OB_GET_LOCATION_TIME_OUT", cluster_id=1753069619, tenant_id=1002, ls_ids=[{id:1001}]) [2025-08-14 09:06:50.750060] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:50.750730] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:04:39.759929] EDIAG [SERVER] process_final (ob_inner_sql_connection.cpp:763) [801][T1002_TntTransf][T1002][YB42AC4D0007-00063C496259A60F-0-0] [lt=987914][errcode=0] LOGGER COST TOO MUCH TIME, cost: 130799662, owner: ObLog, click_count: 5, time dist:[FORMAT_END=959309, ObMalloc#start=1, ObMalloc#end=122, ALLOC_END=129840229, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1275158c 0x96a79f6 0x96a5754 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x9a64aaf 0x10502edb 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:49.752637] EDIAG [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=990819][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1004558, owner: ObLog, click_count: 5, time dist:[FORMAT_END=968787, ObMalloc#start=0, ObMalloc#end=35769, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1c02313f 0x97a45d8 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:46.777925] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [876][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E999C16-0-0] [lt=869711][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2003385, owner: ObLog, click_count: 8, time dist:[FORMAT_END=1980357, ObMalloc#start=1, ObMalloc#end=22616, ObMalloc#start=20, ObMalloc#end=50, FREE_END=341, ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=965107], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3c61 0x113d348b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:04:40.742444] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:288) [860][T1002_ArcFetche][T1002][YB42AC4D0007-00063C495E399C16-0-0] [lt=1820][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1019609, owner: ObLog, click_count: 5, time dist:[FORMAT_END=992884, ObMalloc#start=1, ObMalloc#end=26719, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113b2889 0x979b621 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:50.748491] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:288) [732][T1001_ArcFetche][T1001][YB42AC4D0007-00063C495CA99C16-0-0] [lt=19] ObArchiveFetcher is running(thread_index=0) [2025-08-14 09:06:50.745979] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=31] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:50.752563] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=6564] temporary file module use no blocks [2025-08-14 09:06:50.752572] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:50.752587] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:50.752601] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623312185679, flush_io_finished_ret_:0, flush_io_finished_round_:26466, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:50.748649] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=989915, expected_run_ts_us_=1755162409758733, ObClockGenerator::getRealClock()=1755162410748649, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:50.753116] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=4459][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:50.753638] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=48] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:50.755347] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:50.756108] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:50.756170] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=13] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:50.756528] INFO [CLOG] get_ls_status_from_table (ob_garbage_collector.cpp:1481) [830][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=13504] failed to get ls status info from table(ret=-4012, tenant_id=1002, ls_id={id:1}) [2025-08-14 09:06:50.753137] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] task scheduled too delay(scheduled_delay=994166, expected_run_ts_us_=1755162409758970, ObClockGenerator::getRealClock()=1755162410753137, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:50.756568] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=3425][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:50.756594] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] task scheduled too delay(scheduled_delay=1003685, expected_run_ts_us_=1755162409752908, ObClockGenerator::getRealClock()=1755162410756594, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d45b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:50.756605] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d45b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:50.756614] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=990022, expected_run_ts_us_=1755162409766592, ObClockGenerator::getRealClock()=1755162410756614, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72747b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:50.735255] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=23] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:06:50.735786] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=45] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:04:39.759929] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:763) [801][T1002_TntTransf][T1002][YB42AC4D0007-00063C496259A60F-0-0] [lt=987914] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT task_id, status FROM __all_transfer_task"}, process_time=250018210) [2025-08-14 09:06:50.756858] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=243][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72747b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:50.758004] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1141][errcode=0] task scheduled too delay(scheduled_delay=991595, expected_run_ts_us_=1755162409766408, ObClockGenerator::getRealClock()=1755162410758004, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:50.758021] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:50.758032] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=982411, expected_run_ts_us_=1755162409775620, ObClockGenerator::getRealClock()=1755162410758032, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:50.758048] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:49.752637] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=990819] [WRS] update tenant weak read server version(tenant_id=1001, server_version={version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1755162392765637}, version_delta=689977457) [2025-08-14 09:06:50.744518] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 29.999896 loop:26707, slowcnt: 288 [2025-08-14 09:06:49.762323] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=20876] get_min_unreplayed_log_info(lsn={lsn:5282737655}, scn={val:1755100947825033001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161624848121, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:13063726, ret_code_:0}}, next_to_submit_lsn_:{lsn:5282737655}, next_to_submit_scn_:{val:1755100947825033001, v:0}, base_lsn_:{lsn:4831543296}, base_scn_:{val:1755063445733337001, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7605000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:5282737655}, end_lsn:{lsn:5282737655}, read_buf:{buf_len_:3747840, buf_:0x7f98f7605000}, block_size:67104768, log_storage_:0x7f993e529bf0, log_storage_:{log_tail:{lsn:5282737655}, readable_log_tail:{lsn:5282737655}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:5234171904}, min_scn:{val:1755152244802535004, v:0}, curr_block_id:78, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:206, min_block_id:42, max_block_id:79, curr_writable_block_id:78}, logical_block_size_:67104768, curr_block_writable_size_:18539017, block_header_serialize_buf_:0x7f993e52a8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:85, read_io_size:301367296, read_disk_cost_ts:2364604, start_lsn:{lsn:4831543296}}}}}}) [2025-08-14 09:06:50.759964] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=996122] finish update min start scn(max_decided_scn={val:1755161547738516001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1755161493768447002, v:0}, keep_alive_scn_:{val:1755161493768447002, v:0}, update_ts_:1755162410757277}) [2025-08-14 09:06:49.764437] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=15189] get_min_unreplayed_log_info(lsn={lsn:8461098908}, scn={val:1755100947600130001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161920778323, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:23084275, ret_code_:0}}, next_to_submit_lsn_:{lsn:8461098908}, next_to_submit_scn_:{val:1755100947600130001, v:0}, base_lsn_:{lsn:7515734016}, base_scn_:{val:1753840867171901045, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6e05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:8461098908}, end_lsn:{lsn:8461098908}, read_buf:{buf_len_:3747840, buf_:0x7f98f6e05000}, block_size:67104768, log_storage_:0x7f99297a3bf0, log_storage_:{log_tail:{lsn:8461098908}, readable_log_tail:{lsn:8461098908}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8455200768}, min_scn:{val:1755160384851871002, v:0}, curr_block_id:126, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:215, min_block_id:93, max_block_id:127, curr_writable_block_id:126}, logical_block_size_:67104768, curr_block_writable_size_:61206628, block_header_serialize_buf_:0x7f99297a48b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:198, read_io_size:699822080, read_disk_cost_ts:4178035, start_lsn:{lsn:7515734016}}}}}}) [2025-08-14 09:06:50.760660] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=994714] finish update min start scn(max_decided_scn={val:1755161719775179002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1755161287778691002, v:0}, keep_alive_scn_:{val:1755161287778691002, v:0}, update_ts_:1755162410757277}) [2025-08-14 09:06:50.761383] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=19] [COMPACTION DAG STATUS] (start_time=1755162400740833, end_time=1755162410761382, dag_status={}) [2025-08-14 09:06:50.761407] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=22] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:50.761419] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:50.761429] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:50.761439] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:50.761457] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=18] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:50.761467] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:50.761480] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:50.761491] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:50.761500] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:50.761513] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=10] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:06:50.761521] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:06:50.761560] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3484] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:50.761569] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:50.761578] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:50.763049] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:50.756908] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [368][VTblLocAsyncUp0][T1][YB42AC4D0007-00063C4968299C62-0-0] [lt=12955][errcode=-4012] result set close failed(ret=-4012) [2025-08-14 09:06:50.758058] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=982402, expected_run_ts_us_=1755162409775655, ObClockGenerator::getRealClock()=1755162410758058, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:50.764123] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=6053][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:50.764152] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] task scheduled too delay(scheduled_delay=992250, expected_run_ts_us_=1755162409771901, ObClockGenerator::getRealClock()=1755162410764152, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:50.764167] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:50.764179] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=991208, expected_run_ts_us_=1755162409772971, ObClockGenerator::getRealClock()=1755162410764179, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:50.764668] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=958677][errcode=-4389] timer task too much delay in priority queue(token={this:0x7f991b7a5d18, timer:0x7f99200ff180, task:0x7f99200ff1a8, task_type:"N9oceanbase8observer18ObTableLoadService17ObCheckTenantTaskE", scheduled_time:1755162377761774, pushed_time:1755162392751930, delay:1000000}, delay_in_priority_queue=14990156, threashold=1000000) [2025-08-14 09:06:50.764691] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a5d18, timer:0x7f99200ff180, task:0x7f99200ff1a8, task_type:"N9oceanbase8observer18ObTableLoadService17ObCheckTenantTaskE", scheduled_time:1755162377761774, pushed_time:1755162392751930, delay:1000000}, delay_in_thread_pool=17971776, threashold=500000) [2025-08-14 09:06:48.764666] EDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1997300, owner: ObLog, click_count: 7, time dist:[FORMAT_END=1997290, ObMalloc#start=0, ObMalloc#end=2, ALLOC_END=2, APPEND_END=4, ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0xfc0e73d 0x993f560 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:50.761586] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:50.764773] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3182] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:50.764810] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=36] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:06:50.764820] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:06:50.764832] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:06:48.764666] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:8607757, max_lsn:1604090331, max_scn:{val:1755161756730847001, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:1556737504}, reuse_lsn:{lsn:1604090331}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:1562147291}}, last_submit_lsn:{lsn:1604090203}, last_submit_end_lsn:{lsn:1604090331}, last_submit_log_id:8607757, last_submit_log_pid:50, max_flushed_lsn:{lsn:1604090203}, max_flushed_end_lsn:{lsn:1604090331}, max_flushed_log_pid:50, committed_end_lsn:{lsn:1604090090}, last_slide_log_id:8607755, last_slide_scn:{val:1755161749741193000, v:0}, last_slide_lsn:{lsn:1604089968}, last_slide_end_lsn:{lsn:1604090090}, last_slide_log_pid:50, last_slide_log_accum_checksum:1446576759, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f990bfe4530}, [StateMgr]={this:0x7f990bfe89b0, self:"172.77.0.7:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"PENDING", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:51}, leader:"0.0.0.0:0", leader_epoch:-1, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:1604090331}, scan_disk_log_finished:true, last_check_start_id:8607756, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161901828039, palf_role_change_cb:0x7f990bfef470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:50, prev:{version:1, config:{config_version:{proposal_id:49, config_seq:45}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:50, config_seq:46}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_mode_pid:50}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_end_lsn:{lsn:1604088821}, prev_mode_pid:50}, reconfig_barrier:{prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_end_lsn:{lsn:1604088821}, prev_mode_pid:50}, persistent_config_version:{proposal_id:50, config_seq:46}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755161738854289, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755161738854289, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f990bfe5c70}, [ModeMgr]={palf_id:1, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:50, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:1543409664}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:51}, log_config_meta:{version:3, proposal_id:50, prev:{version:1, config:{config_version:{proposal_id:49, config_seq:45}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:50, config_seq:46}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:46, prev_lsn:{lsn:1604088699}, prev_mode_pid:50}, log_snapshot_meta:{version:2, base_lsn:{lsn:1543409664}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:51, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:712704}, readable_log_tail:{lsn:712704}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:1610514432}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/meta", dir_fd:225, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66392064, block_header_serialize_buf_:0x7f990bfeabb8, flashback_version:0}, log_storage:{log_tail:{lsn:1604090331}, readable_log_tail:{lsn:1604090331}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:23, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:229, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:6424101, block_header_serialize_buf_:0x7f990bfec8b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1755161901828095, this:0x7f990bfe90f0}, [Reconfirm]={palf_id:1, self:"172.77.0.7:2882", state:"RECONFIRM_MODE_META", new_proposal_id:51, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"172.77.0.7:2882", majority_max_accept_pid:50, majority_max_lsn:{lsn:1604090331}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:1755161901828373, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f990bfe8b30}) [2025-08-14 09:06:50.721998] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=467448][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:50.253|207=468.16ms|210=0us|222=2us|total=511.33ms) [2025-08-14 09:06:50.765172] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=43107][errcode=-4389] timer task use too much time(end=1755162410765172, start=1755162410253837, delta=511335) [2025-08-14 09:06:50.765186] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=281655, expected_run_ts_us_=1755162410483531, ObClockGenerator::getRealClock()=1755162410765186, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:50.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:50.765201] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:50.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:50.765218] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:50.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:50.744435] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:50.765278] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20810][errcode=0] epoll handler too much time :20854 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:50.765732] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:50.733692] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=996493] gts statistics(tenant_id=1001, gts_rpc_cnt=0, get_gts_cache_cnt=7, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:06:50.765968] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=32224] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1755161724720398], gts:1755161724715616000, latest_srr:[mts=1755162406734826]}) [2025-08-14 09:06:50.765958] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:50.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:50.765987] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1001}) [2025-08-14 09:06:50.766001] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1001}) [2025-08-14 09:06:50.766004] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:50.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:50.753421] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=310][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:50.766049] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12604][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:50.766092] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4016] thread maybe hung!(point=[thread id=835, timeout ts=09:02:56.572, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:26.572], idx=835, next_allow_print_ts="09:03:26.572") [2025-08-14 09:06:50.766106] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:50.766395] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3502] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:50.764191] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:50.766823] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=2623][errcode=0] task scheduled too delay(scheduled_delay=980962, expected_run_ts_us_=1755162409785859, ObClockGenerator::getRealClock()=1755162410766822, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:50.766841] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:47.771409] EDIAG [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2990951, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2990944, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x989c577 0x989bc96 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:47.771409] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=16] [thread pool stat](pool_stat={pool:0x564d6e8922c0, last_check_time:1755162407771406, last_idle_time:102477764956, last_thread_count:12}, *pool={name:TimerWK, this:0x564d6e8922c0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:10, threads_idle_time:102477764956, tenant_id:500}, qsize=10, current_idle_time=102477764956) [2025-08-14 09:06:50.766906] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=2990951] [thread pool stat](pool_stat={pool:0x7f99667cbcf0, last_check_time:1755162410766905, last_idle_time:35019929178, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x7f99667cbcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:35019929178, tenant_id:500}, qsize=-1, current_idle_time=35019929178) [2025-08-14 09:06:50.766922] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=16] [thread pool stat](pool_stat={pool:0x7f99667ccaf0, last_check_time:1755162410766922, last_idle_time:26794418912, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x7f99667ccaf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:26794418912, tenant_id:500}, qsize=-1, current_idle_time=26794418912) [2025-08-14 09:06:50.766933] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] [thread pool stat](pool_stat={pool:0x7f9969de56b0, last_check_time:1755162410766933, last_idle_time:102938681543, last_thread_count:17}, *pool={name:TimerWK, this:0x7f9969de56b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:16, threads_idle_time:102938681543, tenant_id:1}, qsize=10, current_idle_time=102938681543) [2025-08-14 09:06:50.766944] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] [thread pool stat](pool_stat={pool:0x7f99356b0e70, last_check_time:1755162410766944, last_idle_time:102366343177, last_thread_count:15}, *pool={name:TimerWK, this:0x7f99356b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:12, threads_idle_time:102366343177, tenant_id:1001}, qsize=12, current_idle_time=102366343177) [2025-08-14 09:06:50.766955] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] [thread pool stat](pool_stat={pool:0x7f99230b0e70, last_check_time:1755162410766955, last_idle_time:102534969223, last_thread_count:17}, *pool={name:TimerWK, this:0x7f99230b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:16, threads_idle_time:102534969223, tenant_id:1002}, qsize=11, current_idle_time=102534969223) [2025-08-14 09:02:27.786103] EDIAG [OCCAM] operator() (ob_occam_timer.h:85) [835][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=50503][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 262584641, owner: ObLog, click_count: 7, time dist:[FORMAT_END=948163, ObMalloc#start=0, ObMalloc#end=7161, ObMalloc#start=261629312, ObMalloc#end=2, ALLOC_END=0, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ffa3b 0x9835318 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:50.767843] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=16045] log compressor cycles once. (ret=0, cost_time=3000887, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25861672960, fast_delete_log_mode=false) [2025-08-14 09:02:27.786103] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [835][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=50503][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99319fe030, caller:ob_table_lock_service.cpp:start:276, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE6$_1273, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe310, time_wheel:0x7f99535f3990, thread_pool:0x7f99535f3170, is_running:0, time_interval:10.00s, expected_run_time:09:02:36.426, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:50.767952] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [835][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=262584641][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:06:50.767|total=264.20s) [2025-08-14 09:06:50.766853] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=981693, expected_run_ts_us_=1755162409785160, ObClockGenerator::getRealClock()=1755162410766853, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:50.768144] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1288][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:50.768162] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] task scheduled too delay(scheduled_delay=964082, expected_run_ts_us_=1755162409804079, ObClockGenerator::getRealClock()=1755162410768162, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919836b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:50.768175] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919836b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:50.768188] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=937787, expected_run_ts_us_=1755162409830400, ObClockGenerator::getRealClock()=1755162410768188, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:50.764221] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1018) [801][T1002_TntTransf][T1002][YB42AC4D0007-00063C496259A60F-0-0] [lt=130799662][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT task_id, status FROM __all_transfer_task"}, aret=-4012, ret=-4012) [2025-08-14 09:06:50.770507] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=202] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:50.773315] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=287][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:50.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:50.775428] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=257][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:50.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:50.775577] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:50.775602] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:50.780399] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=210][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:50.780485] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=370] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:06:50.780556] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=149] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:50.780570] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] temporary file module use no blocks [2025-08-14 09:06:50.780579] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:50.780591] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:50.780604] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623312219541, flush_io_finished_ret_:0, flush_io_finished_round_:26428, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:50.800058] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=200] gc stale ls task succ [2025-08-14 09:06:50.803519] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:50.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:50.846872] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=2116][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:50.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:50.847952] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=255][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:06:50.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:50.880734] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=187] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:50.880796] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:50.880846] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:50.880895] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:50.880937] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:50.880951] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:50.880964] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:50.946965] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:50.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:50.953439] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=177] PNIO [ratelimit] time: 1755162410953437, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007618, add_bytes: 0 [2025-08-14 09:06:50.975538] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:50.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:50.996580] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:50.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.024835] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:51.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.047026] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:50.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.147068] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:51.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.175635] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:51.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.247137] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:51.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.247958] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:51.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.258021] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:51.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.275781] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:51.275810] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:51.275840] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:51.275856] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:51.275868] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:51.275881] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:51.275893] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:51.275908] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:51.275923] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:51.275934] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:51.276317] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:51.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.306481] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:51.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.347183] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:51.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.369461] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=200] gc stale ls task succ [2025-08-14 09:06:51.375754] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:51.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.447244] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:51.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.499338] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:51.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.527682] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:51.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.547308] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:51.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.575839] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:51.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.647340] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:51.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:50.782876] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=193] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:51.721182] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=936884][errcode=0] epoll handler too much time :938314 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:50.768200] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:50.782900] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=207] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:51.721226] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=936908][errcode=0] epoll handler too much time :938334 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:51.012155] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:51.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.721226] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=951580][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:50.768|207=953.03ms|210=0us|222=2us|total=953.07ms) [2025-08-14 09:06:51.721267] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4389] timer task use too much time(end=1755162411721266, start=1755162410768187, delta=953079) [2025-08-14 09:06:51.721277] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=708037][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:51.121|207=709.11ms|210=0us|222=2us|total=709.14ms) [2025-08-14 09:06:51.721293] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] task scheduled too delay(scheduled_delay=1890267, expected_run_ts_us_=1755162409831025, ObClockGenerator::getRealClock()=1755162411721293, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.831, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.721303] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4389] timer task use too much time(end=1755162411721302, start=1755162411012153, delta=709149) [2025-08-14 09:06:51.721315] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.831, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.721330] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=1859196, expected_run_ts_us_=1755162409862133, ObClockGenerator::getRealClock()=1755162411721330, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.862, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.721325] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=0] task scheduled too delay(scheduled_delay=712189, expected_run_ts_us_=1755162411009134, ObClockGenerator::getRealClock()=1755162411721324, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:51.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.721342] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:49.862, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.721347] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:51.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.721363] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=632046, expected_run_ts_us_=1755162411089316, ObClockGenerator::getRealClock()=1755162411721363, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:51.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.721366] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=983308, expected_run_ts_us_=1755162410738058, ObClockGenerator::getRealClock()=1755162411721366, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.721378] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.721376] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:51.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.721391] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=989955, expected_run_ts_us_=1755162410731436, ObClockGenerator::getRealClock()=1755162411721391, ret=0, ret="OB_SUCCESS", *this={this:0x7f98918114b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.721399] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=0] task scheduled too delay(scheduled_delay=517102, expected_run_ts_us_=1755162411204297, ObClockGenerator::getRealClock()=1755162411721399, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:51.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.721403] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918114b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.721413] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:51.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.721416] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=986963, expected_run_ts_us_=1755162410734452, ObClockGenerator::getRealClock()=1755162411721415, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728aa90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.721428] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728aa90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.721440] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=990622, expected_run_ts_us_=1755162410730817, ObClockGenerator::getRealClock()=1755162411721439, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.080620] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=11] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:51.721451] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.721462] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=989061, expected_run_ts_us_=1755162410732401, ObClockGenerator::getRealClock()=1755162411721462, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.721475] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.721486] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=988049, expected_run_ts_us_=1755162410733437, ObClockGenerator::getRealClock()=1755162411721486, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.721497] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.721509] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=979882, expected_run_ts_us_=1755162410741627, ObClockGenerator::getRealClock()=1755162411721509, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.721521] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.721534] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=977981, expected_run_ts_us_=1755162410743552, ObClockGenerator::getRealClock()=1755162411721533, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.721545] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.721556] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=978401, expected_run_ts_us_=1755162410743155, ObClockGenerator::getRealClock()=1755162411721556, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.721567] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.721579] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=964679, expected_run_ts_us_=1755162410756899, ObClockGenerator::getRealClock()=1755162411721579, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fc830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.721591] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fc830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.721603] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=965414, expected_run_ts_us_=1755162410756188, ObClockGenerator::getRealClock()=1755162411721603, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:50.789620] INFO handle (memory_dump.cpp:671) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=20] statistics: tenant_cnt: 6, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 168 270 0 1 2 110 1 0 1 3 7 0 0 1 4 1 1 0 1 5 85 6 0 1 6 13 0 0 1 7 22 2 0 1 8 53 1 0 1 12 17 3 0 1 13 10 1 0 1 14 1 1 0 1 15 1 1 0 1 16 8 7 0 1 18 4 3 0 1 20 2 1 0 1 22 1 1 0 1 24 1 0 0 1 25 6 0 0 500 0 561 274 0 500 7 194 5 0 500 8 41 1 0 500 9 2 1 0 500 10 7 2 0 500 16 57 4 0 500 17 60 7 0 500 22 38 64 0 500 23 27 10 0 508 0 2 7 0 508 8 5 1 0 509 0 2 7 0 509 8 5 1 0 1001 0 147 252 0 1001 2 26 1 0 1001 3 5 0 0 1001 4 1 1 0 1001 5 85 6 0 1001 6 11 0 0 1001 7 30 2 0 1001 8 36 1 0 1001 12 17 3 0 1001 13 3 1 0 1001 14 1 1 0 1001 16 7 4 0 1001 18 4 3 0 1001 20 4 1 0 1001 25 8 0 0 1002 0 263 277 0 1002 2 198 1 0 1002 3 4 0 0 1002 4 1 1 0 1002 5 101 6 0 1002 6 14 0 0 1002 7 6 2 0 1002 8 73 1 0 1002 12 18 3 0 1002 13 7 1 0 1002 14 1 2 0 1002 16 10 7 0 1002 18 9 3 0 1002 20 2 1 0 1002 21 1 0 0 1002 24 1 0 0 1002 25 7 0 0 cost_time: 1026294 [2025-08-14 09:06:51.721614] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.721625] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=965306, expected_run_ts_us_=1755162410756319, ObClockGenerator::getRealClock()=1755162411721625, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a54b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:50.758698] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:551) [360][UserLocAsyncUp0][T0][YB42AC4D0007-00063C495D499F18-0-0] [lt=26819][errcode=-4664] fail to renew location(ret=-4664, ret="OB_GET_LOCATION_TIME_OUT", task={cluster_id:1753069619, tenant_id:1002, ls_id:{id:1001}, renew_for_tenant:false, add_timestamp:1755161937800377}) [2025-08-14 09:06:50.774223] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=303][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:50.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:50.765019] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1997300] [PALF_DUMP](palf_id=1, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161974728707, last_flushed_end_lsn:{lsn:1604090331}}]) [2025-08-14 09:06:51.275948] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:51.723435] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=446806][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:51.723460] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:51.723486] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:51.723502] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:51.723519] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:51.723539] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:51.723707] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:51.723730] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:51.721636] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a54b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.721589] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=729] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:51.727254] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=5604][errcode=0] task scheduled too delay(scheduled_delay=966706, expected_run_ts_us_=1755162410760547, ObClockGenerator::getRealClock()=1755162411727253, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72edc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.727293] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72edc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.727308] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] task scheduled too delay(scheduled_delay=962089, expected_run_ts_us_=1755162410765218, ObClockGenerator::getRealClock()=1755162411727308, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72edb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.727321] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72edb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.727336] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=961213, expected_run_ts_us_=1755162410766122, ObClockGenerator::getRealClock()=1755162411727335, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.727348] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.727360] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=958726, expected_run_ts_us_=1755162410768633, ObClockGenerator::getRealClock()=1755162411727360, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.727372] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.727385] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=966386, expected_run_ts_us_=1755162410760998, ObClockGenerator::getRealClock()=1755162411727384, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.727399] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.727411] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=956576, expected_run_ts_us_=1755162410770834, ObClockGenerator::getRealClock()=1755162411727411, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.727423] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.721644] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=930591] [MEMORY] tenant: 1, limit: 2147483648 hold: 1013174272 cache_hold: 56623104 cache_used: 56623104 cache_item_count: 27 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 444993536 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 230686720 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 1163264 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 483328 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 51617792 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 1458176 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 14204928 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 111149056 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35651584 limit= 429496720 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 8003584 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= PS_CACHE_CTX_ID hold_bytes= 65536 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 14827520 limit= 9223372036854775807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 3481600 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3964928 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 212992 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 884736 limit= 9223372036854775807 [2025-08-14 09:06:51.727435] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=948095, expected_run_ts_us_=1755162410779339, ObClockGenerator::getRealClock()=1755162411727434, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728ac10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.727446] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728ac10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.727461] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=924555, expected_run_ts_us_=1755162410802905, ObClockGenerator::getRealClock()=1755162411727461, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.802, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.727473] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.802, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.727485] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=926010, expected_run_ts_us_=1755162410801474, ObClockGenerator::getRealClock()=1755162411727485, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.727497] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.727509] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=923198, expected_run_ts_us_=1755162410804310, ObClockGenerator::getRealClock()=1755162411727508, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728f9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.727523] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728f9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.727536] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=908995, expected_run_ts_us_=1755162410818540, ObClockGenerator::getRealClock()=1755162411727536, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eae10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.727547] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eae10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.727560] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=904652, expected_run_ts_us_=1755162410822907, ObClockGenerator::getRealClock()=1755162411727559, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.727571] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.727585] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=862227, expected_run_ts_us_=1755162410865358, ObClockGenerator::getRealClock()=1755162411727585, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.727597] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.727609] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=860836, expected_run_ts_us_=1755162410866773, ObClockGenerator::getRealClock()=1755162411727609, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.866, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.727621] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:50.866, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.727639] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=398539, expected_run_ts_us_=1755162411329099, ObClockGenerator::getRealClock()=1755162411727639, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.727652] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.728177] WDIAG [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1766) [830][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4012] failed to get ls status from table(ret=-4012, tenant_id=1002, ls_id={id:1}) [2025-08-14 09:06:51.728198] WDIAG [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1666) [830][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=0] get_ls_status_ failed(tmp_ret=-4012, ls->get_ls_id()={id:1}) [2025-08-14 09:06:51.728258] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:51.729241] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:603) [360][UserLocAsyncUp0][T0][YB42AC4D0007-00063C495D499F18-0-0] [lt=961876][errcode=-4664] fail to batch process task(ret=-4664) [2025-08-14 09:06:51.729628] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=955922] current log_task status(palf_id=1, self="172.77.0.7:2882", log_id=8607756, log_task={header:{begin_lsn:{lsn:1604090090}, end_lsn:{lsn:1604090203}, log_id:8607756, min_scn:{val:1755161749741193001, v:0}, max_scn:{val:1755161749741193001, v:0}, data_len:57, proposal_id:50, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:1604090090}, data_checksum:2031283194, accum_checksum:759186326, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1755161750716165, freeze_ts:1755161750716173, submit_ts:1755161750716183, flushed_ts:1755161853765770, gen_to_freeze cost time:8, gen_to_submit cost time:18, submit_to_flush cost time:103049587}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161974728707, last_flushed_end_lsn:{lsn:1604090331}}]) [2025-08-14 09:06:51.722040] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=946382][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:50.774|207=947.82ms|210=0us|222=1us|total=947.83ms) [2025-08-14 09:06:51.731224] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=9169][errcode=-4389] timer task use too much time(end=1755162411731223, start=1755162410774221, delta=957002) [2025-08-14 09:06:51.734054] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72756b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.734082] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.724, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.734095] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a36b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.739884] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1808) [801][T1002_TntTransf][T1002][YB42AC4D0007-00063C496259A60F-0-0] [lt=4234][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1002, sql=SELECT task_id, status FROM __all_transfer_task) [2025-08-14 09:06:51.740223] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=858478][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:51.740245] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:51.740730] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:51.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.743416] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:51.744119] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919bfbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aa690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.744154] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919911b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196cc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.744165] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d40b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.744179] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.744194] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.744207] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.744218] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.744276] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:51.747516] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=60][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:51.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.749331] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:51.750791] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:51.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.751086] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=11] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:51.751458] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=287] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:51.751895] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=215] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:51.752494] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=212] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:51.754978] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.744, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.755010] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aaf90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.755025] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aba10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.755037] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.755050] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196cb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.755062] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.756333] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=9] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:51.756476] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:51.756568] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:51.756594] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:51.756664] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:51.757442] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=190] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:51.729296] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:502) [360][UserLocAsyncUp0][T0][YB42AC4D0007-00063C495D499F18-0-0] [lt=53][errcode=-4664] fail to batch execute task(ret=-4664, tasks.count()=1) [2025-08-14 09:06:51.758028] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task too much delay in priority queue(token={this:0x7f991b7a4578, timer:0x7f9966a63f10, task:0x7f9922e05028, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr17RefreshConfigTaskE", scheduled_time:1755162386750680, pushed_time:1755162392765831, delay:10000000}, delay_in_priority_queue=6015151, threashold=1000000) [2025-08-14 09:06:51.758048] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a4578, timer:0x7f9966a63f10, task:0x7f9922e05028, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr17RefreshConfigTaskE", scheduled_time:1755162386750680, pushed_time:1755162392765831, delay:10000000}, delay_in_thread_pool=17998877, threashold=500000) [2025-08-14 09:06:51.758073] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a41f8, timer:0x7f9922df93b0, task:0x7f9922ffab60, task_type:"N9oceanbase3sql27ObSqlPlanMonitorRecycleTaskE", scheduled_time:1755162392771052, pushed_time:1755162392775885, delay:1000000}, delay_in_thread_pool=18982183, threashold=500000) [2025-08-14 09:06:51.761030] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=15261][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 31.016409 loop:26707, slowcnt: 298 [2025-08-14 09:06:51.761070] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:51.761588] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:51.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.761627] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:51.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.762158] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5783] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 444993536 used= 389812256 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 70 washed_blocks= 287 washed_size= 70221824 [MEMORY] request_cached_chunk_cnt= 6 [MEMORY] hold= 389812256 used= 379555363 count= 32901 avg_used= 11536 mod=SUMMARY [2025-08-14 09:06:51.762190] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=18] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 230686720 used= 228884480 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 block_cnt= 110 chunk_cnt= 110 mod=Memstore [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:06:51.762212] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=16] [MEMORY] tenant_id= 1 ctx_id= EXECUTE_CTX_ID hold= 1163264 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 15 washed_size= 13516800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:51.762227] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:06:51.762244] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1079) [801][T1002_TntTransf][T1002][YB42AC4D0007-00063C496259A60F-0-0] [lt=25][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1002) [2025-08-14 09:06:51.762255] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=21] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 51617792 used= 47452416 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 84 washed_blocks= 334 washed_size= 126640128 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 47452416 used= 43070492 count= 3992 avg_used= 10789 mod=SUMMARY [2025-08-14 09:06:51.762287] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=26] [MEMORY] tenant_id= 1 ctx_id= WORK_AREA hold= 1458176 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 13 washed_blocks= 66 washed_size= 25804800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:51.762305] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 14204928 used= 10970224 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 277 washed_size= 31932416 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10970224 used= 6078924 count= 18573 avg_used= 327 mod=SUMMARY [2025-08-14 09:06:51.762328] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 111149056 used= 108896256 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 block_cnt= 211 chunk_cnt= 53 mod=CoStack [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 mod=SUMMARY [2025-08-14 09:06:51.762346] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 56623104 used= 56623104 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 56623104 used= 0 count= 27 avg_used= 0 mod=SUMMARY [2025-08-14 09:06:51.762362] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 34365376 limit= 429496720 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 34365376 used= 31086608 count= 475 avg_used= 65445 mod=SUMMARY [2025-08-14 09:06:51.762386] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 8003584 used= 4947392 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 10 washed_blocks= 23 washed_size= 12967936 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4947392 used= 4793344 count= 604 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:51.762401] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:51.762414] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 65536 used= 31024 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 2031616 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 31024 used= 27058 count= 20 avg_used= 1352 mod=SUMMARY [2025-08-14 09:06:51.762429] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= RPC_CTX_ID hold= 14827520 used= 6135808 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 6135808 used= 4051328 count= 253 avg_used= 16013 mod=SUMMARY [2025-08-14 09:06:51.762445] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 3481600 used= 2391232 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 10 washed_size= 4907008 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2391232 used= 2367040 count= 36 avg_used= 65751 mod=SUMMARY [2025-08-14 09:06:51.762462] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 3964928 used= 3080192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 3 washed_size= 229376 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3080192 used= 2983936 count= 376 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:51.762480] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= SCHEMA_SERVICE hold= 212992 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1884160 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:51.762495] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:51.762507] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= MERGE_NORMAL_CTX_ID hold= 884736 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 6 washed_blocks= 9 washed_size= 11698176 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:51.762524] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 500, limit: 9223372036854775807 hold: 688619520 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 333369344 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 39174144 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 85983232 limit= 9223372036854775807 [MEMORY] ctx_id= LIBEASY hold_bytes= 3145728 limit= 9223372036854775807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 8388608 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 3063808 limit= 9223372036854775807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 37584896 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 78925824 limit= 9223372036854775807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 98983936 limit= 9223372036854775807 [2025-08-14 09:06:51.762555] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 333369344 used= 301557360 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 499 washed_blocks= 2229 washed_size= 922714112 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 301557360 used= 294139997 count= 12707 avg_used= 23147 mod=SUMMARY [2025-08-14 09:06:51.762587] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 500 ctx_id= GLIBC hold= 39174144 used= 21978864 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 183 washed_blocks= 806 washed_size= 361381888 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 21978864 used= 10372933 count= 131697 avg_used= 78 mod=SUMMARY [2025-08-14 09:06:51.730362] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:199) [368][VTblLocAsyncUp0][T1][YB42AC4D0007-00063C4968299C62-0-0] [lt=7180][errcode=-4012] result set close failed(ret=-4012) [2025-08-14 09:06:51.762602] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= CO_STACK hold= 85983232 used= 83091456 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 83091456 used= 82936896 count= 161 avg_used= 515136 block_cnt= 161 chunk_cnt= 41 mod=CoStack [MEMORY] hold= 83091456 used= 82936896 count= 161 avg_used= 515136 mod=SUMMARY [2025-08-14 09:06:51.762604] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:951) [368][VTblLocAsyncUp0][T1][YB42AC4D0007-00063C4968299C62-0-0] [lt=32192][errcode=-4012] failed to close result(close_ret=-4012, ret=-4012) [2025-08-14 09:06:51.762616] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 500 ctx_id= LIBEASY hold= 3145728 used= 3070816 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1048576 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3070816 used= 2859784 count= 223 avg_used= 12824 mod=SUMMARY [2025-08-14 09:06:51.762629] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 500 ctx_id= LOGGER_CTX_ID hold= 10485760 used= 12484608 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 12484608 used= 12478464 count= 6 avg_used= 2079744 mod=SUMMARY [2025-08-14 09:06:51.762653] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 500 ctx_id= RPC_CTX_ID hold= 3063808 used= 1482752 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 57 washed_blocks= 70 washed_size= 116473856 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1482752 used= 979328 count= 61 avg_used= 16054 mod=SUMMARY [2025-08-14 09:06:51.762667] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= PKT_NIO hold= 37584896 used= 36013568 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 50 washed_blocks= 71 washed_size= 99016704 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 36013568 used= 35017984 count= 125 avg_used= 280143 mod=SUMMARY [2025-08-14 09:06:51.762690] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 500 ctx_id= SCHEMA_SERVICE hold= 78925824 used= 68937344 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 22 washed_size= 5767168 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 68937344 used= 67674815 count= 4860 avg_used= 13924 mod=SUMMARY [2025-08-14 09:06:51.762704] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= UNEXPECTED_IN_500 hold= 98983936 used= 97174864 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 97174864 used= 86783232 count= 3014 avg_used= 28793 mod=SUMMARY [2025-08-14 09:06:51.762719] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 508, limit: 1073741824 hold: 15613952 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 5128192 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10485760 limit= 9223372036854775807 [2025-08-14 09:06:51.762729] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 508 ctx_id= DEFAULT_CTX_ID hold= 5128192 used= 3765328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3765328 used= 3723912 count= 84 avg_used= 44332 mod=SUMMARY [2025-08-14 09:06:51.762753] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 508 ctx_id= CO_STACK hold= 10485760 used= 10321920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10321920 used= 10302720 count= 20 avg_used= 515136 mod=SUMMARY [2025-08-14 09:06:51.762796] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 509, limit: 1073741824 hold: 15613952 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 5128192 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10485760 limit= 9223372036854775807 [2025-08-14 09:06:51.762808] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 509 ctx_id= DEFAULT_CTX_ID hold= 5128192 used= 3765328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3765328 used= 3723912 count= 84 avg_used= 44332 mod=SUMMARY [2025-08-14 09:06:51.762832] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 509 ctx_id= CO_STACK hold= 10485760 used= 10321920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10321920 used= 10302720 count= 20 avg_used= 515136 mod=SUMMARY [2025-08-14 09:06:51.762889] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant: 1001, limit: 1073741824 hold: 608747520 cache_hold: 44040192 cache_used: 44040192 cache_item_count: 21 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 302309376 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 54525952 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 3080192 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 483328 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 45514752 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 5521408 limit= 157286400 [MEMORY] ctx_id= GLIBC hold_bytes= 15499264 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 73424896 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35651584 limit= 214748360 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 3252224 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 14680064 limit= 1073741800 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 5570560 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3866624 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 1179648 limit= 9223372036854775807 [2025-08-14 09:06:51.762913] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= DEFAULT_CTX_ID hold= 302309376 used= 268516432 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 81 washed_blocks= 406 washed_size= 113410048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 268516432 used= 262397846 count= 3783 avg_used= 69362 mod=SUMMARY [2025-08-14 09:06:51.762926] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= MEMSTORE_CTX_ID hold= 54525952 used= 54099968 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 54099968 used= 54075216 count= 26 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:06:51.762935] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= EXECUTE_CTX_ID hold= 3080192 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 8 washed_size= 7405568 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:51.762953] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=14] [MEMORY] tenant_id= 1001 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:06:51.762964] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= PLAN_CACHE_CTX_ID hold= 45514752 used= 36552320 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 77 washed_blocks= 201 washed_size= 132743168 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 36552320 used= 33009690 count= 3323 avg_used= 9933 mod=SUMMARY [2025-08-14 09:06:51.762973] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= WORK_AREA hold= 5521408 used= 0 limit= 157286400 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 9 washed_blocks= 40 washed_size= 17547264 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:51.762983] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= GLIBC hold= 15499264 used= 10889728 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 25 washed_blocks= 164 washed_size= 47415296 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10889728 used= 6020141 count= 18313 avg_used= 328 mod=SUMMARY [2025-08-14 09:06:51.762995] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= CO_STACK hold= 73424896 used= 71221248 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 32 washed_blocks= 34 washed_size= 2072576 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 71221248 used= 71088768 count= 138 avg_used= 515136 block_cnt= 138 chunk_cnt= 36 mod=CoStack [MEMORY] hold= 71221248 used= 71088768 count= 138 avg_used= 515136 mod=SUMMARY [2025-08-14 09:06:51.763009] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= KVSTORE_CACHE_ID hold= 44040192 used= 44040192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 44040192 used= 0 count= 21 avg_used= 0 mod=SUMMARY [2025-08-14 09:06:51.763020] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 33906624 limit= 214748360 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33906624 used= 30629648 count= 468 avg_used= 65447 mod=SUMMARY [2025-08-14 09:06:51.763030] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= TX_CALLBACK_CTX_ID hold= 3252224 used= 2342592 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 3 washed_blocks= 8 washed_size= 3039232 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2342592 used= 2269696 count= 286 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:51.763039] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:51.763056] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= RPC_CTX_ID hold= 14680064 used= 4202496 limit= 1073741800 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4202496 used= 2779776 count= 171 avg_used= 16256 mod=SUMMARY [2025-08-14 09:06:51.763069] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= TX_DATA_TABLE hold= 5570560 used= 2342784 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 8 washed_size= 2818048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2342784 used= 2319424 count= 30 avg_used= 77314 mod=SUMMARY [2025-08-14 09:06:51.763081] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= MDS_DATA_ID hold= 3866624 used= 2785280 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 5 washed_size= 4521984 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2785280 used= 2698240 count= 340 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:51.763098] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= MERGE_NORMAL_CTX_ID hold= 1179648 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 8 washed_blocks= 24 washed_size= 15597568 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:51.763116] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant: 1002, limit: 8589934592 hold: 1945841664 cache_hold: 48234496 cache_used: 48234496 cache_item_count: 23 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 1009610752 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 415236096 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 1114112 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 950272 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 211812352 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 1441792 limit= 429496725 [MEMORY] ctx_id= GLIBC hold_bytes= 12582912 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 149913600 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 37748736 limit= 1717986900 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 7766016 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 5783552 limit= 8589934500 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 5718016 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3047424 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 1032192 limit= 9223372036854775807 [2025-08-14 09:06:51.763145] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= DEFAULT_CTX_ID hold= 1009610752 used= 919125424 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 13 [MEMORY] hold= 185860096 used= 185534336 count= 114 avg_used= 1627494 block_cnt= 114 chunk_cnt= 104 mod=MysqlRequesReco [MEMORY] hold= 167792640 used= 167772192 count= 1 avg_used= 167772192 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 83927040 used= 83886080 count= 2 avg_used= 41943040 block_cnt= 2 chunk_cnt= 2 mod=LogGroupBuffer [MEMORY] hold= 67190784 used= 67109344 count= 4 avg_used= 16777336 block_cnt= 4 chunk_cnt= 4 mod=MrgMemPoolChk [MEMORY] hold= 919125424 used= 901881937 count= 9500 avg_used= 94934 mod=SUMMARY [2025-08-14 09:06:51.763159] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1002 ctx_id= MEMSTORE_CTX_ID hold= 415236096 used= 411992064 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 411992064 used= 411803568 count= 198 avg_used= 2079816 block_cnt= 198 chunk_cnt= 198 mod=Memstore [MEMORY] hold= 411992064 used= 411803568 count= 198 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:06:51.763171] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1002 ctx_id= EXECUTE_CTX_ID hold= 1114112 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 7 washed_size= 7274496 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:51.763179] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= TRANS_CTX_MGR_ID hold= 950272 used= 933888 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1146880 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 933888 used= 921984 count= 2 avg_used= 460992 mod=SUMMARY [2025-08-14 09:06:51.763188] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= PLAN_CACHE_CTX_ID hold= 211812352 used= 201995008 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 176930880 used= 160228962 count= 13450 avg_used= 11912 block_cnt= 13450 chunk_cnt= 101 mod=SqlPhyPlan [MEMORY] hold= 201995008 used= 183048143 count= 17906 avg_used= 10222 mod=SUMMARY [2025-08-14 09:06:51.763207] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= WORK_AREA hold= 1441792 used= 0 limit= 429496725 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 14 washed_blocks= 54 washed_size= 27918336 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:51.763216] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= GLIBC hold= 12582912 used= 11425280 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 11425280 used= 6322556 count= 19216 avg_used= 329 mod=SUMMARY [2025-08-14 09:06:51.763225] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1002 ctx_id= CO_STACK hold= 149913600 used= 147087360 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 68 washed_blocks= 70 washed_size= 3178496 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 147087360 used= 146813760 count= 285 avg_used= 515136 block_cnt= 285 chunk_cnt= 73 mod=CoStack [MEMORY] hold= 147087360 used= 146813760 count= 285 avg_used= 515136 mod=SUMMARY [2025-08-14 09:06:51.763250] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= KVSTORE_CACHE_ID hold= 48234496 used= 48234496 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 48234496 used= 0 count= 23 avg_used= 0 mod=SUMMARY [2025-08-14 09:06:51.763258] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1002 ctx_id= META_OBJ_CTX_ID hold= 37748736 used= 35741632 limit= 1717986900 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 35741632 used= 32457488 count= 496 avg_used= 65438 mod=SUMMARY [2025-08-14 09:06:51.763265] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= TX_CALLBACK_CTX_ID hold= 7766016 used= 6934400 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 17 washed_size= 6914048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 6934400 used= 6729728 count= 848 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:51.763281] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1002 ctx_id= LOB_CTX_ID hold= 147456 used= 16384 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 16384 used= 15872 count= 2 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:51.763309] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1002 ctx_id= RPC_CTX_ID hold= 5783552 used= 5029888 limit= 8589934500 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 10 washed_blocks= 10 washed_size= 15187968 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 5029888 used= 3315456 count= 210 avg_used= 15787 mod=SUMMARY [2025-08-14 09:06:51.763323] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= TX_DATA_TABLE hold= 5718016 used= 4521856 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 9 washed_blocks= 16 washed_size= 13156352 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4521856 used= 4480128 count= 40 avg_used= 112003 mod=SUMMARY [2025-08-14 09:06:51.763335] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= MDS_DATA_ID hold= 3047424 used= 1998848 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 4 washed_size= 1146880 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1998848 used= 1936384 count= 244 avg_used= 7936 mod=SUMMARY [2025-08-14 09:06:51.763344] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1002 ctx_id= MDS_CTX_ID hold= 147456 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:51.763357] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:51.763367] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= MERGE_NORMAL_CTX_ID hold= 1032192 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 12 washed_size= 13647872 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:06:51.766402] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:51.766692] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.751, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.770165] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:51.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.772482] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=20] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:51.776422] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:51.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.776753] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.780581] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=13] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:06:51.780713] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=22] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:51.780731] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=15] temporary file module use no blocks [2025-08-14 09:06:51.780741] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:51.780756] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:51.780768] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623313220394, flush_io_finished_ret_:0, flush_io_finished_round_:26429, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:51.782333] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:51.782371] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:51.786832] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919acf10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.786863] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.796887] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919abe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.796915] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.796925] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199fe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919acd90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.796938] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e80b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.796955] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a31b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ede10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.800097] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=26] gc stale ls task succ [2025-08-14 09:06:51.802518] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:51.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.817015] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.806, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.817050] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.817062] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.817072] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.800, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.817088] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728e330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7290690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.805, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.817113] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.803, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.833391] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=2094][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:51.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.837127] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.824, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.840309] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=8] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:06:51.840335] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:51.847188] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.832, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.847568] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:51.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.848868] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:06:51.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.877400] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ade10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.864, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.877434] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.877446] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919adc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.877460] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919adb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.882394] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:51.882462] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:51.882505] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:51.882548] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:51.882588] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:51.882601] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:51.882613] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:51.887457] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72865b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.878, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.907572] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728de30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7290b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.899, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.917633] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:51.902, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:51.933454] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:51.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.940374] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:51.947603] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:51.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.961056] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=24] PNIO [ratelimit] time: 1755162411961054, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007617, add_bytes: 0 [2025-08-14 09:06:51.976552] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:51.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:51.992113] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:51.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.012946] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:52.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.012981] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:52.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.023664] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:52.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.033515] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:51.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.040405] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:52.047635] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:51.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.093386] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:52.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.121740] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=639864] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:52.129013] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.111, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:52.133563] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:52.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.140464] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:52.147701] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:52.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.176669] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:52.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.214122] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:52.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.223907] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:52.223946] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:52.223964] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:52.223975] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:52.223988] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:52.223999] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:52.233608] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:52.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.240524] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:52.264424] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:52.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.264461] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:52.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.275037] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:52.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.305244] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:52.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.333658] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:52.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.340583] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:52.369532] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=24] gc stale ls task succ [2025-08-14 09:06:52.376797] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:52.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.433703] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:52.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.440646] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:52.452704] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fa190, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa470, time_wheel:0x7f99341cdbf0, thread_pool:0x7f99341cd3d0, is_running:1, time_interval:2.00s, expected_run_time:09:06:52.334, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.516021] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:52.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.516057] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:52.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.526461] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:52.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.533765] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:52.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.540706] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:52.576923] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:52.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.596413] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:52.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.633841] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:52.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.640768] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:52.717258] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:52.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.247715] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:52.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.722160] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [510][T1_TimerWK1][T1][YB42AC4D0007-00063C495C4A02C9-0-0] [lt=0][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f994e1a4ab8, timer:0x7f996703bb70, task:0x7f994b398518, task_type:"N9oceanbase7storage29ObTenantCheckpointSlogHandler21ObWriteCheckpointTaskE", scheduled_time:1755162071765090, pushed_time:1755162071771615, delay:60000000}, delay_in_thread_pool=332969818, threashold=500000) [2025-08-14 09:06:52.224014] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:52.722163] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=473725][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:52.247|207=474.44ms|210=1us|222=1us|total=474.51ms) [2025-08-14 09:06:52.722245] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=81][errcode=-4389] timer task use too much time(end=1755162412722245, start=1755162412247713, delta=474532) [2025-08-14 09:06:52.722229] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=497456][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:52.722269] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:52.722290] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:52.722313] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:52.722326] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:52.721305] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5670] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:52.722346] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:52.722358] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:52.722370] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:52.722389] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:52.722402] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [830]OB_MALLOC COST TOO MUCH TIME, cost_time=959309, tenant_id=500, label=HeapVar, ctx_id=0, prio=0, size=24600, owner: ObMalloc, click_count: 1, time dist:[ALLOC_CHUNK_END=959306], seq:[0] [2025-08-14 09:06:51.758097] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [807][T1002_TimerWK3_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=9] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):258, sequence:1}) [2025-08-14 09:06:52.722571] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:52.722597] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:52.722576] WDIAG [PALF] runTimerTask (block_gc_timer_task.cpp:94) [807][T1002_TimerWK3_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=963007][errcode=-4389] try_recycle_blocks cost too much time(ret=-4389, cost_time_us=964489, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.77.0.7:2882", log_dir:"/root/ob/observer/store/clog/tenant_1002", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):258, sequence:1}, log_alloc_mgr_:{flying_log_task:1, flying_meta_task:2}}) [2025-08-14 09:06:52.722623] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:97) [807][T1002_TimerWK3_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=47] BlockGCTimerTask success(ret=0, cost_time_us=964489, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.77.0.7:2882", log_dir:"/root/ob/observer/store/clog/tenant_1002", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):24883, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):258, sequence:1}, log_alloc_mgr_:{flying_log_task:1, flying_meta_task:2}}) [2025-08-14 09:06:52.722644] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a4818, timer:0x7f99335d9ee0, task:0x7f991c5e9398, task_type:"N9oceanbase4palf16BlockGCTimerTaskE", scheduled_time:1755162392782020, pushed_time:1755162392785953, delay:30000}, delay_in_thread_pool=18972130, threashold=500000) [2025-08-14 09:06:52.728611] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:52.735024] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:52.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.742514] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:52.742568] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] [COMPACTION DAG STATUS] (start_time=1755162402727598, end_time=1755162412742567, dag_status={}) [2025-08-14 09:06:52.743353] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:52.743397] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:52.743411] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891991430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ce10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:52.743414] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:50.761806] EDIAG [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1979105, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1979099, ObMalloc#start=0, ObMalloc#end=2, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b51df0f 0x986bcde 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:50.760653] EDIAG [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1981641, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1981636, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b51df0f 0x986bcde 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:52.243531] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:52.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.743741] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=499451][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:52.243|207=500.21ms|210=0us|222=1us|total=500.25ms) [2025-08-14 09:06:52.743785] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=43][errcode=-4389] timer task use too much time(end=1755162412743785, start=1755162412243529, delta=500256) [2025-08-14 09:06:52.743798] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=499163, expected_run_ts_us_=1755162412244633, ObClockGenerator::getRealClock()=1755162412743797, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:52.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.743813] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:52.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:50.761806] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=18] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136829134139001, v:0}, clog_base_lsn:{lsn:8186781696}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1753840867372367034, v:0}, all_id_meta:{id_meta:[{limited_id:1755161729775179000, latest_log_ts:{val:1755161719775179001, v:0}}, {limited_id:18000001, latest_log_ts:{val:1755157440846556001, v:0}}, {limited_id:60000001, latest_log_ts:{val:1755135657266496001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:46, palf_env_:0x7f99321fa030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-08-14 09:06:52.744622] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:50.760653] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:291) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=28] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1755136826952869001, v:0}, clog_base_lsn:{lsn:5099962368}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1755071646732636000, v:0}, all_id_meta:{id_meta:[{limited_id:1755161555795565000, latest_log_ts:{val:1755161545796642001, v:0}}, {limited_id:22000001, latest_log_ts:{val:1755135645787806003, v:0}}, {limited_id:60000001, latest_log_ts:{val:1755135646400653001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:50, palf_env_:0x7f9950bf4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-08-14 09:04:44.726507] EDIAG [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:630) [14862][T1001_TimerWK5_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=20422][errcode=0] LOGGER COST TOO MUCH TIME, cost: 10004938, owner: ObLog, click_count: 5, time dist:[FORMAT_END=10004929, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1bf31f83 0x1bf31d6b 0x9aaab88 0x1c3fa49f 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:52.751202] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:52.751483] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=13] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:52.752199] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:52.752476] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:52.752554] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:52.753845] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:52.753934] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:52.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.754392] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aac90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:52.754431] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:52.754442] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:52.754459] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e62b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:52.754476] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73940b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:52.754570] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:52.756718] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:52.756846] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:52.757905] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:52.758259] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=31] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:52.761646] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 32.017025 loop:26707, slowcnt: 308 [2025-08-14 09:06:52.762594] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=53][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:52.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.762641] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:52.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.764476] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eeb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:52.764509] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73936b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eec90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:52.764521] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:52.765706] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:52.756892] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:52.767819] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=30] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25861640192, fast_delete_log_mode=false) [395]OB_MALLOC COST TOO MUCH TIME, cost_time=977947, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=977939], seq:[0] [2025-08-14 09:06:52.722654] WDIAG alarm_if_necessary (ob_timer_service.cpp:143) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task's elapsed_time is greater than period(token={this:0x7f991b7a4818, timer:0x7f99335d9ee0, task:0x7f991c5e9398, task_type:"N9oceanbase4palf16BlockGCTimerTaskE", scheduled_time:1755162392782020, pushed_time:1755162392785953, delay:30000}, elapsed_time=964561, period=30000) [2025-08-14 09:06:52.968669] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=33] PNIO [ratelimit] time: 1755162412968667, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007613, add_bytes: 0 [660]OB_MALLOC COST TOO MUCH TIME, cost_time=975439, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=975434], seq:[0] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=975449, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=975447], seq:[0, 1] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=973639, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=973629, ALLOC_CHUNK_END=5], seq:[0, 1] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=973722, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=973648, WASH_BLOCK_END=74], seq:[0, 1, 2] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=971889, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=971879, ALLOC_CHUNK_END=5], seq:[0, 1] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=971910, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=971898, WASH_BLOCK_END=11], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=969474, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=969464, ALLOC_CHUNK_END=5], seq:[0, 1] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=969493, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=969482, WASH_BLOCK_END=10], seq:[0, 1, 2] [864]OB_MALLOC COST TOO MUCH TIME, cost_time=965968, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=965957, ALLOC_CHUNK_END=5], seq:[0, 1] [864]OB_MALLOC COST TOO MUCH TIME, cost_time=965987, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=965977, WASH_BLOCK_END=10], seq:[0, 1, 2] [832]OB_MALLOC COST TOO MUCH TIME, cost_time=969285, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=969276, ALLOC_CHUNK_END=4], seq:[0, 1] [832]OB_MALLOC COST TOO MUCH TIME, cost_time=969312, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=969302, WASH_BLOCK_END=10], seq:[0, 1, 2] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=967860, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=967833, ALLOC_CHUNK_END=4], seq:[0, 1] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=967897, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=967887, WASH_BLOCK_END=10], seq:[0, 1, 2] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=966917, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=966907, ALLOC_CHUNK_END=4], seq:[0, 1] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=966935, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=966926, WASH_BLOCK_END=9], seq:[0, 1, 2] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=950314, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=950304, ALLOC_CHUNK_END=5], seq:[0, 1] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=950332, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=950322, WASH_BLOCK_END=10], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=927956, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=927947, ALLOC_CHUNK_END=4], seq:[0, 1] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=927972, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=927963, WASH_BLOCK_END=9], seq:[0, 1, 2] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=915569, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=915559, ALLOC_CHUNK_END=4], seq:[0, 1] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=869057, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=869048, ALLOC_CHUNK_END=4], seq:[0, 1] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=915585, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=915576, WASH_BLOCK_END=9], seq:[0, 1, 2] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=869079, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=869070, WASH_BLOCK_END=8], seq:[0, 1, 2] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=908226, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=908217, ALLOC_CHUNK_END=5], seq:[0, 1] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=908243, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=908234, WASH_BLOCK_END=9], seq:[0, 1, 2] [656]OB_MALLOC COST TOO MUCH TIME, cost_time=887546, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=887528, ALLOC_CHUNK_END=12], seq:[0, 1] [656]OB_MALLOC COST TOO MUCH TIME, cost_time=887572, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=887554, WASH_BLOCK_END=18], seq:[0, 1, 2] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=867732, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=867723, ALLOC_CHUNK_END=4], seq:[0, 1] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=867747, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=867739, WASH_BLOCK_END=8], seq:[0, 1, 2] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=755331, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=755314, ALLOC_CHUNK_END=4], seq:[0, 1] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=755363, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=755342, WASH_BLOCK_END=21], seq:[0, 1, 2] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=737222, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=737213, ALLOC_CHUNK_END=5], seq:[0, 1] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=737240, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=737230, WASH_BLOCK_END=9], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=629635, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=629626, ALLOC_CHUNK_END=4], seq:[0, 1] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=629654, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=629644, WASH_BLOCK_END=9], seq:[0, 1, 2] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=529561, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=529552, ALLOC_CHUNK_END=4], seq:[0, 1] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=529578, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=529569, WASH_BLOCK_END=8], seq:[0, 1, 2] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=383050, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=383032, ALLOC_CHUNK_END=5], seq:[0, 1] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=383066, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=383057, WASH_BLOCK_END=9], seq:[0, 1, 2] [764]OB_MALLOC COST TOO MUCH TIME, cost_time=999787, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=999784], seq:[0] [763]OB_MALLOC COST TOO MUCH TIME, cost_time=999781, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=999773], seq:[0] [552]OB_MALLOC COST TOO MUCH TIME, cost_time=999732, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=999731], seq:[0] [2025-08-14 09:06:52.770601] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:52.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:53.747577] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=975493][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:52.770|207=976.97ms|210=1us|222=2us|total=977.01ms) [2025-08-14 09:06:53.747614] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=-4389] timer task use too much time(end=1755162413747613, start=1755162412770599, delta=977014) [2025-08-14 09:06:53.747628] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=955277, expected_run_ts_us_=1755162412792349, ObClockGenerator::getRealClock()=1755162413747627, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:52.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:53.747649] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:52.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.777052] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:52.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:53.748074] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=969551][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:52.777|207=971.02ms|210=0us|222=2us|total=971.04ms) [2025-08-14 09:06:53.748096] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task use too much time(end=1755162413748095, start=1755162412777050, delta=971045) [2025-08-14 09:06:53.749401] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:52.881671] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=416][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fc8b0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcb90, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:1, time_interval:3.00s, expected_run_time:09:06:52.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:53.752131] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=869141][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:52.881|207=870.46ms|210=0us|222=2us|total=870.48ms) [2025-08-14 09:06:53.752149] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4389] timer task use too much time(end=1755162413752149, start=1755162412881668, delta=870481) [2025-08-14 09:06:52.883355] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:53.752499] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=867784] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:53.752564] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:53.752606] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:53.752644] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:53.752654] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:53.752664] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:53.760583] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:53.769650] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=191][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739ef90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:53.769664] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=989664, expected_run_ts_us_=1755162412779999, ObClockGenerator::getRealClock()=1755162413769664, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a45b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:53.769673] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a45b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:53.769682] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=994151, expected_run_ts_us_=1755162412775530, ObClockGenerator::getRealClock()=1755162413769681, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eee10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:53.769690] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eee10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:53.769699] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=993990, expected_run_ts_us_=1755162412775708, ObClockGenerator::getRealClock()=1755162413769699, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73931b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:53.769707] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73931b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:53.769716] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=982609, expected_run_ts_us_=1755162412787107, ObClockGenerator::getRealClock()=1755162413769716, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:53.769731] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:53.769743] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=983328, expected_run_ts_us_=1755162412786414, ObClockGenerator::getRealClock()=1755162413769743, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:53.769754] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:53.769765] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=983017, expected_run_ts_us_=1755162412786748, ObClockGenerator::getRealClock()=1755162413769765, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7273c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:53.769776] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7273c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:53.769788] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=985865, expected_run_ts_us_=1755162412783922, ObClockGenerator::getRealClock()=1755162413769788, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d27b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:53.769799] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d27b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:53.769811] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=980425, expected_run_ts_us_=1755162412789385, ObClockGenerator::getRealClock()=1755162413769810, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d94b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:53.769821] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d94b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:53.769839] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=989519, expected_run_ts_us_=1755162412780319, ObClockGenerator::getRealClock()=1755162413769839, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.780, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:53.769850] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.780, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:53.769862] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=989372, expected_run_ts_us_=1755162412780489, ObClockGenerator::getRealClock()=1755162413769861, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739bbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.780, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:53.769873] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739bbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.780, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:53.762620] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=918648] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:06:53.770214] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7578][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:53.768102] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][YB42AC4D0007-00063C495F8A1984-0-0] [lt=45388][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a5c38, timer:0x7f9935f93f20, task:0x7f99200fe2d0, task_type:"N9oceanbase8datadict17ObDataDictServiceE", scheduled_time:1755162393734105, pushed_time:1755162393734208, delay:1000000}, delay_in_thread_pool=19033913, threashold=500000) [2025-08-14 09:06:53.769884] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=988115, expected_run_ts_us_=1755162412781769, ObClockGenerator::getRealClock()=1755162413769884, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fa10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:53.771710] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1821][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fa10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:53.771736] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] task scheduled too delay(scheduled_delay=988515, expected_run_ts_us_=1755162412783221, ObClockGenerator::getRealClock()=1755162413771736, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f727dc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f729ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:53.771750] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f727dc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f729ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:53.771762] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=988066, expected_run_ts_us_=1755162412783695, ObClockGenerator::getRealClock()=1755162413771762, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f727d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f729af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:53.751583] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=986157] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:06:53.773382] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=21009] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:53.773416] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=32] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:53.773434] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=17] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623314220559, flush_io_finished_ret_:0, flush_io_finished_round_:26430, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:53.773519] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:53.760, task_priority:1, with_handle_protected:1}) [395]OB_MALLOC COST TOO MUCH TIME, cost_time=1004939, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=59, WASH_BLOCK_END=132, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=977969], seq:[0, 1, 2, 3, 4] [2025-08-14 09:06:52.761089] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1011735, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=7, ALLOC_BLOCK#end=3, FORMAT_END=6760, ObMalloc#start=0, ObMalloc#end=1004958, ALLOC_END=1, APPEND_END=6], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:52.761089] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:53.774384] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1011735][errcode=0] epoll handler too much time :1013315 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:53.774405] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:52.875782] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=934303][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:06:52.875735] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=934283][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:06:53.801293] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=952517] gc stale ls task succ [2025-08-14 09:06:53.804277] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:53.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:53.848166] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:53.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:53.862690] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:53.863734] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=418][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:06:53.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:53.948228] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:53.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:53.952967] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:53.953036] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:53.953084] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:53.953138] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:53.953182] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:53.953198] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:53.953211] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:53.962781] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:53.968319] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=533][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:53.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:53.976295] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=15] PNIO [ratelimit] time: 1755162413976293, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007626, add_bytes: 0 [2025-08-14 09:06:53.998112] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=350][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:53.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:54.029395] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:54.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:54.048322] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=51][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:53.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:54.062800] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:54.148365] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:54.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:54.154277] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=630586] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:54.162835] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:54.168443] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:54.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:54.248426] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:54.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:54.249545] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:54.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:54.259600] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:54.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:54.262895] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:54.270719] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:54.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:54.300867] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:54.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:54.348495] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:54.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:54.362968] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:54.368567] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:54.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:54.369655] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=383964] gc stale ls task succ [2025-08-14 09:06:54.448529] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:54.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:54.463028] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:54.522153] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:54.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:54.548589] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:54.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:54.563074] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:54.568690] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:54.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:54.648651] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:54.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:54.663136] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:54.721030] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=978] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:54.727275] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8039] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [808]OB_MALLOC COST TOO MUCH TIME, cost_time=1958660, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1958645], seq:[0] [763]OB_MALLOC COST TOO MUCH TIME, cost_time=1959065, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=999789], seq:[0, 1] [2025-08-14 09:06:52.768312] EDIAG [OCCAM] operator() (ob_occam_timer.h:81) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EED7-0-0] [lt=25][errcode=-4016] LOGGER COST TOO MUCH TIME, cost: 1959101, owner: ObLog, click_count: 5, time dist:[FORMAT_END=15, ObMalloc#start=1, ObMalloc#end=1959079, ALLOC_END=2, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0xf9ff91f 0x9835280 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:53.771773] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f727d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f729af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.729691] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=956461][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:53.771|207=957.92ms|210=0us|222=2us|total=957.96ms) [2025-08-14 09:06:54.729729] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4389] timer task use too much time(end=1755162414729728, start=1755162413771761, delta=957967) [2025-08-14 09:06:54.729753] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=1943900, expected_run_ts_us_=1755162412785844, ObClockGenerator::getRealClock()=1755162414729752, ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.729779] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.729793] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=1936051, expected_run_ts_us_=1755162412793742, ObClockGenerator::getRealClock()=1755162414729793, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891814fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.793, task_priority:1, with_handle_protected:0}) [764]OB_MALLOC COST TOO MUCH TIME, cost_time=1959458, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=999805], seq:[0, 1] [2025-08-14 09:06:54.729805] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891814fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.793, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.729822] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] task scheduled too delay(scheduled_delay=1933375, expected_run_ts_us_=1755162412796447, ObClockGenerator::getRealClock()=1755162414729822, ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.729833] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.729847] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=1922573, expected_run_ts_us_=1755162412807273, ObClockGenerator::getRealClock()=1755162414729847, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7facb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.729858] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7facb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.729872] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1734215, expected_run_ts_us_=1755162412995656, ObClockGenerator::getRealClock()=1755162414729872, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199fbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919acc10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.995, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.729883] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199fbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919acc10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:52.995, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.729910] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1010975, expected_run_ts_us_=1755162413718935, ObClockGenerator::getRealClock()=1755162414729910, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728bb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.718, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.729924] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728bb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.718, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.729936] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1010129, expected_run_ts_us_=1755162413719806, ObClockGenerator::getRealClock()=1755162414729936, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180fbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.719, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.729954] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180fbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.719, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.729966] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=982224, expected_run_ts_us_=1755162413747742, ObClockGenerator::getRealClock()=1755162414729966, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.729977] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.729989] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=981554, expected_run_ts_us_=1755162413748434, ObClockGenerator::getRealClock()=1755162414729989, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730000] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730012] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=980936, expected_run_ts_us_=1755162413749075, ObClockGenerator::getRealClock()=1755162414730012, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730023] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730034] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=974308, expected_run_ts_us_=1755162413755726, ObClockGenerator::getRealClock()=1755162414730034, ret=0, ret="OB_SUCCESS", *this={this:0x7f98918105b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730047] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918105b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730059] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=964663, expected_run_ts_us_=1755162413765395, ObClockGenerator::getRealClock()=1755162414730059, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730070] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730081] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=966447, expected_run_ts_us_=1755162413763633, ObClockGenerator::getRealClock()=1755162414730081, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730092] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730105] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=961831, expected_run_ts_us_=1755162413768273, ObClockGenerator::getRealClock()=1755162414730104, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891815730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730116] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891815730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730127] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=959732, expected_run_ts_us_=1755162413770395, ObClockGenerator::getRealClock()=1755162414730127, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eca90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730138] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eca90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.770, task_priority:1, with_handle_protected:0}) [552]OB_MALLOC COST TOO MUCH TIME, cost_time=1959387, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=999739], seq:[0, 1] [2025-08-14 09:06:54.730150] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=959532, expected_run_ts_us_=1755162413770617, ObClockGenerator::getRealClock()=1755162414730149, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e94b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730162] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e94b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730173] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=959387, expected_run_ts_us_=1755162413770786, ObClockGenerator::getRealClock()=1755162414730173, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730184] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730195] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=955074, expected_run_ts_us_=1755162413775121, ObClockGenerator::getRealClock()=1755162414730195, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180ea30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730207] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180ea30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730221] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=947351, expected_run_ts_us_=1755162413782869, ObClockGenerator::getRealClock()=1755162414730221, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730240] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730252] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=946146, expected_run_ts_us_=1755162413784105, ObClockGenerator::getRealClock()=1755162414730252, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730263] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730274] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=941187, expected_run_ts_us_=1755162413789087, ObClockGenerator::getRealClock()=1755162414730274, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196be10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730287] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196be10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730307] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] task scheduled too delay(scheduled_delay=944969, expected_run_ts_us_=1755162413785337, ObClockGenerator::getRealClock()=1755162414730307, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728d430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7290f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730318] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728d430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7290f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730329] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=934565, expected_run_ts_us_=1755162413795764, ObClockGenerator::getRealClock()=1755162414730329, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:52.767575] EDIAG [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10661][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1960223, owner: ObLog, click_count: 5, time dist:[FORMAT_END=822, ObMalloc#start=0, ObMalloc#end=1959399, ALLOC_END=2, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x1d0f5b0a 0x1d0f57c6 0x1d17ad7c 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:54.730340] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730353] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=934447, expected_run_ts_us_=1755162413795906, ObClockGenerator::getRealClock()=1755162414730353, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73922b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730364] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73922b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:52.768312] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EED7-0-0] [lt=25][errcode=-4016] task executed too delay(delay_time=91721, *p_task_={this:0x7f99571630c0, caller:mds_tenant_service.cpp:mtl_start:117, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_145, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571633a0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:0, time_interval:15.00s, expected_run_time:09:07:07.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:54.730376] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=934339, expected_run_ts_us_=1755162413796036, ObClockGenerator::getRealClock()=1755162414730376, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730387] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730401] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] task scheduled too delay(scheduled_delay=931265, expected_run_ts_us_=1755162413799136, ObClockGenerator::getRealClock()=1755162414730401, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.799, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730412] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.799, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730424] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=937489, expected_run_ts_us_=1755162413792934, ObClockGenerator::getRealClock()=1755162414730424, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730435] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730446] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=921236, expected_run_ts_us_=1755162413809210, ObClockGenerator::getRealClock()=1755162414730446, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.809, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730459] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.809, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730470] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=925610, expected_run_ts_us_=1755162413804860, ObClockGenerator::getRealClock()=1755162414730470, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eaa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730481] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eaa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730492] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=892953, expected_run_ts_us_=1755162413837539, ObClockGenerator::getRealClock()=1755162414730492, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73945b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.837, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730503] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73945b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.837, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730517] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=818749, expected_run_ts_us_=1755162413911768, ObClockGenerator::getRealClock()=1755162414730517, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.911, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730529] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:53.911, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730551] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:54.718, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730562] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:54.726, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.730574] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a67b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7397290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:54.728, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:53.864245] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=223][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:53.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:54.730612] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=865054][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:53.864|207=866.36ms|210=1us|222=1us|total=866.38ms) [2025-08-14 09:06:54.730624] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task use too much time(end=1755162414730624, start=1755162413864244, delta=866380) [2025-08-14 09:06:52.768255] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [764][T1002_MdsT][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=213643, expected_run_ts_us_=1755162412554611, ObClockGenerator::getRealClock()=1755162412768254, ret=0, ret="OB_SUCCESS", *this={this:0x7f9957162ce0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f9957162fc0, time_wheel:0x7f99335f7990, thread_pool:0x7f99335f7170, is_running:1, time_interval:3.00s, expected_run_time:09:06:52.554, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:52.767575] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10661] temporary file module use no blocks [2025-08-14 09:06:54.730787] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=1960223] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:54.730800] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:52.780566] EDIAG [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=992452][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1947514, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1947511, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1fce4e91 0x1fce455d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:54.730822] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=21] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623314196281, flush_io_finished_ret_:0, flush_io_finished_round_:26468, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:52.780566] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=992452] blacklist_loop exec finished(cost_time=15, is_enabled=true, send_cnt=0) [2025-08-14 09:06:54.017732] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=319][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:54.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:54.734353] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=715531][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:54.177|207=716.62ms|210=0us|222=2us|total=716.65ms) [2025-08-14 09:06:54.734380] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4389] timer task use too much time(end=1755162414734380, start=1755162414017729, delta=716651) [2025-08-14 09:06:54.734400] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:54.734906] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [510][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=31][errcode=-4389] timer task too much delay in priority queue(token={this:0x7f994e1a5df8, timer:0x7f994b3feee0, task:0x7f994b3fef50, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162023758648, pushed_time:1755162122757495, delay:1000000}, delay_in_priority_queue=98998847, threashold=1000000) [2025-08-14 09:06:54.734922] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [510][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f994e1a5df8, timer:0x7f994b3feee0, task:0x7f994b3fef50, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162023758648, pushed_time:1755162122757495, delay:1000000}, delay_in_thread_pool=289964706, threashold=500000) [2025-08-14 09:06:54.734930] WDIAG alarm_if_necessary (ob_timer_service.cpp:143) [510][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4389] timer task's elapsed_time is greater than period(token={this:0x7f994e1a5df8, timer:0x7f994b3feee0, task:0x7f994b3fef50, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162023758648, pushed_time:1755162122757495, delay:1000000}, elapsed_time=2012704, period=1000000) [2025-08-14 09:06:54.734962] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [510][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f994e1a41f8, timer:0x7f996714bcc0, task:0x7f994b3fab60, task_type:"N9oceanbase3sql27ObSqlPlanMonitorRecycleTaskE", scheduled_time:1755162124751107, pushed_time:1755162124757550, delay:1000000}, delay_in_thread_pool=289977400, threashold=500000) [2025-08-14 09:06:53.782245] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=562] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:54.735940] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=952252][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:54.461213] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fa190, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa470, time_wheel:0x7f99341cdbf0, thread_pool:0x7f99341cd3d0, is_running:1, time_interval:2.00s, expected_run_time:09:06:54.334, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:54.737942] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=276306][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:54.737963] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task use too much time(end=1755162414737962, start=1755162414461211, delta=276751) [2025-08-14 09:06:53.781762] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8466] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:54.738252] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=955043][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:54.740125] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:54.748795] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=52][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:54.749381] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:54.751314] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=435] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:54.751343] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=2108] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:54.752137] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=739] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:54.753096] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=619] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:54.753258] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1556] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:54.754264] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=2289] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:54.755466] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:54.756270] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1453] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:54.758436] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1322] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:54.761204] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=978949] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:06:54.761433] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:54.762196] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=1472] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:54.763201] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:54.764659] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=241][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:54.765256] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=201] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:54.770258] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:54.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:54.490818] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:54.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:54.772222] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:54.775360] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=999491] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:54.776285] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:54.776325] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=36] temporary file module use no blocks [2025-08-14 09:06:54.776333] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:54.782032] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:54.787486] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:54.800810] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:54.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:54.801347] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=35] gc stale ls task succ [2025-08-14 09:06:54.848853] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=67][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:54.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:54.854671] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:54.846, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.863263] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:06:54.863298] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:54.864319] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd070, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fd350, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:5.00s, expected_run_time:09:06:54.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:54.864355] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:06:54.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:54.948916] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:54.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:54.953982] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:06:54.954042] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:06:54.954090] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:06:54.954138] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:06:54.954189] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:06:54.954206] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:54.954220] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:06:54.963323] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:54.972307] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:54.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:54.983901] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=21] PNIO [ratelimit] time: 1755162414983900, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007607, add_bytes: 0 [2025-08-14 09:06:55.016694] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:55.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:55.016737] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:55.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:55.022092] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:55.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:55.048972] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:54.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:55.063395] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:55.097181] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:55.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:55.149026] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:55.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:55.154849] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=34] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:55.163444] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:55.172464] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:55.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:55.217862] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:55.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:55.249069] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:55.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:55.263511] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:55.268161] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:55.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:55.268196] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:55.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:55.273622] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:55.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:55.303790] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:55.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:55.349099] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:55.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:55.363567] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:55.369716] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=21] gc stale ls task succ [2025-08-14 09:06:55.372571] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:55.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:55.449162] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:55.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:55.463605] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:55.519472] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:55.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:55.519507] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:55.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:55.524957] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:55.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:55.549225] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:55.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:55.563634] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:55.572705] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:55.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:55.599863] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:55.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:55.649277] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:55.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:55.663695] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:55.710540] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:55.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:55.721076] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=47] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:54.742534] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=626] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:55.727060] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=983036][errcode=0] epoll handler too much time :984535 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:54.745398] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2961] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:55.727148] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=980266][errcode=0] epoll handler too much time :981761 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:54.771871] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=280624][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:55.727198] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=953883][errcode=-4389] timer task use too much time(end=1755162415727198, start=1755162414490816, delta=1236382) [2025-08-14 09:06:55.727216] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=982581, expected_run_ts_us_=1755162414744633, ObClockGenerator::getRealClock()=1755162415727215, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:54.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:55.727245] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:54.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:55.727263] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=743732, expected_run_ts_us_=1755162414983531, ObClockGenerator::getRealClock()=1755162415727263, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:54.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:55.727275] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:54.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:54.776350] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:55.727814] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=950024] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623316213232, flush_io_finished_ret_:0, flush_io_finished_round_:26431, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:54.830712] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:55.727892] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=895821][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:54.830|207=897.18ms|210=0us|222=2us|total=897.19ms) [2025-08-14 09:06:55.727912] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4389] timer task use too much time(end=1755162415727911, start=1755162414830710, delta=897201) [2025-08-14 09:06:55.728663] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=42] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [360]OB_MALLOC COST TOO MUCH TIME, cost_time=957174, tenant_id=500, label=SqlSessionSbloc, ctx_id=0, prio=0, size=39040, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=2], seq:[0, 1] [2025-08-14 09:06:53.763562] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=733][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:53.763562] EDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=733][errcode=-4016] LOGGER COST TOO MUCH TIME, cost: 1964866, owner: ObLog, click_count: 9, time dist:[ALLOC_BLOCK#start=2, ALLOC_BLOCK#end=65, ALLOC_BLOCK#start=10057, ALLOC_BLOCK#end=3, FORMAT_END=1954725, ObMalloc#start=1, ObMalloc#end=4, ALLOC_END=0, APPEND_END=8], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x13b06ffb 0x958a39e 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:55.731435] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=1964866][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:55.731456] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:55.731476] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:55.731494] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:55.731515] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:55.731531] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:55.731552] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:55.731570] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:55.731589] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:55.731605] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:55.731622] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:55.731634] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:55.731647] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:52.768132] EDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 2959886, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=1958682, FORMAT_END=1001192, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x9757ced 0x97569df 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:55.731805] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:55.731824] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:52.768132] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a5d18, timer:0x7f99200ff180, task:0x7f99200ff1a8, task_type:"N9oceanbase8observer18ObTableLoadService17ObCheckTenantTaskE", scheduled_time:1755162411764702, pushed_time:0, delay:1000000}, running_task_token={this:0x7f991b7a5fb8, timer:0x7f99200ff180, task:0x7f99200ff1f0, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162378750218, pushed_time:1755162410766348, delay:1000000}, total_delay=1003429, threashold=1000000, tenant_id=1002) [2025-08-14 09:01:04.756643] WDIAG [SQL] do_close_plan (ob_result_set.cpp:844) [14786][T1002_TimerWK5_FreInfoReload][T1001][YB42AC4D0007-00063C4974599C72-0-0] [lt=22000][errcode=-4006] exec result is null(ret=-4006) [2025-08-14 09:06:55.738696] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:06:55.738731] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=30] temporary file module use no blocks [2025-08-14 09:06:55.738745] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:06:55.738764] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=17] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:06:55.738784] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=19] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623317170617, flush_io_finished_ret_:0, flush_io_finished_round_:26469, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:06:53.869303] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [872][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E799C16-0-0] [lt=2508] ObArchiveSender is running(thread_index=1) [2025-08-14 09:06:53.869303] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [872][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E799C16-0-0] [lt=2508][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1866746, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1866737, ObMalloc#start=1, ObMalloc#end=1, ALLOC_END=0, APPEND_END=7], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3c61 0x113d348b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:55.739154] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [854][T1002_ObXAHbWor][T1002][Y0-0000000000000000-0-0] [lt=59122] XA scheduler heartbeat task statistics(avg_time=489299) [2025-08-14 09:06:55.740259] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:348) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2025-08-14 09:06:55.740383] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=2959886][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a5998, timer:0x7f9935f8df20, task:0x7f99200f5a78, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr15MediumCheckTaskE", scheduled_time:1755161721762022, pushed_time:0, delay:20000000}, running_task_token={this:0x7f991b7a58b8, timer:0x7f9935f8df20, task:0x7f99200f5a38, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr14MediumLoopTaskE", scheduled_time:1755161718796975, pushed_time:1755161718805054, delay:120000000}, total_delay=693978360, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:55.740406] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6418, timer:0x7f994b1dfb70, task:0x7f9920110068, task_type:"N9oceanbase5share22ObIndexUsageReportTaskE", scheduled_time:1755161727099859, pushed_time:0, delay:900000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=688640547, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:55.740418] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6bf8, timer:0x7f994b1dfb70, task:0x7f99201e29f8, task_type:"N9oceanbase5table21ObTableGroupCommitMgr19ObTableGroupOpsTaskE", scheduled_time:1755161732717636, pushed_time:0, delay:1000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=683022782, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:55.740430] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6878, timer:0x7f994b1dfb70, task:0x7f9920110178, task_type:"N9oceanbase5share27ObIndexUsageRefreshConfTaskE", scheduled_time:1755161733727027, pushed_time:0, delay:2000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=682013402, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:55.740441] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6958, timer:0x7f994b1dfb70, task:0x7f9920110810, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161734741162, pushed_time:0, delay:5000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=680999278, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:55.740451] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6a38, timer:0x7f994b1dfb70, task:0x7f9920110af8, task_type:"N9oceanbase5share23ObStorageIOUsageRepoter23TenantIOUsageReportTaskE", scheduled_time:1755161734803774, pushed_time:0, delay:5000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=680936677, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:55.740462] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6e98, timer:0x7f994b1dfb70, task:0x7f99201075a8, task_type:"N9oceanbase3omt11ObTenantSrs27TenantSrsUpdatePeriodicTaskE", scheduled_time:1755161734810627, pushed_time:0, delay:0}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=680929835, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:55.740473] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a64f8, timer:0x7f994b1dfb70, task:0x7f992010e1b0, task_type:"N9oceanbase5table20ObTableObjectPoolMgr26ObTableSessEliminationTaskE", scheduled_time:1755161735735271, pushed_time:0, delay:5000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=680005202, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:55.740484] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a65d8, timer:0x7f994b1dfb70, task:0x7f992010e1c8, task_type:"N9oceanbase5table20ObTableObjectPoolMgr27ObTableSessSysVarUpdateTaskE", scheduled_time:1755161735797729, pushed_time:0, delay:5000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=679942755, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:55.740495] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a66b8, timer:0x7f994b1dfb70, task:0x7f992010e1e0, task_type:"N9oceanbase5table20ObTableObjectPoolMgr28ObTableReqResEliminationTaskE", scheduled_time:1755161735811808, pushed_time:0, delay:5000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=679928687, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:55.740506] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a4ff8, timer:0x7f994b1dfb70, task:0x7f9922ff3e50, task_type:"N9oceanbase3sql3dtl23ObDTLIntermResultGCTaskE", scheduled_time:1755161742809616, pushed_time:0, delay:10000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=672930890, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:55.740519] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a5b58, timer:0x7f99535c3f30, task:0x7f9922ffee90, task_type:"N9oceanbase7storage21ObTenantFreezeInfoMgr24UpdateLSResvSnapshotTaskE", scheduled_time:1755161744768148, pushed_time:0, delay:10000000}, running_task_token={this:0x7f991b7a5a78, timer:0x7f99535c3f30, task:0x7f9922ffee78, task_type:"N9oceanbase7storage21ObTenantFreezeInfoMgr10ReloadTaskE", scheduled_time:1755161737748006, pushed_time:1755161737749908, delay:3000000}, total_delay=670972371, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:55.740531] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a7838, timer:0x7f994b1dfb70, task:0x7f9920110820, task_type:"N9oceanbase10rootserver16ObMViewMdsOpTaskE", scheduled_time:1755161749755625, pushed_time:0, delay:5000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=665984906, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:55.740547] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6db8, timer:0x7f994b1dfb70, task:0x7f99201e33a0, task_type:"N9oceanbase5table20ObTableClientInfoMgr27ObTableClientInfoRetireTaskE", scheduled_time:1755161750768689, pushed_time:0, delay:60000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=664971858, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:55.740558] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a4d58, timer:0x7f994b1dfb70, task:0x7f991f004030, task_type:"N9oceanbase8observer20ObTableQueryASyncMgrE", scheduled_time:1755161750817104, pushed_time:0, delay:0}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=664923454, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:55.740570] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a74b8, timer:0x7f994b1dfb70, task:0x7f9920110730, task_type:"N9oceanbase10rootserver23ObMViewPushSnapshotTaskE", scheduled_time:1755161804754766, pushed_time:0, delay:60000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=610985804, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:55.740581] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a7218, timer:0x7f994b1dfb70, task:0x7f99201107f8, task_type:"N9oceanbase10rootserver24ObMViewCleanSnapshotTaskE", scheduled_time:1755161804755624, pushed_time:0, delay:60000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=610984957, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:55.740592] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a4e38, timer:0x7f99535f3df0, task:0x7f9922ff6bf0, task_type:"N9oceanbase7obmysql15ObEliminateTaskE", scheduled_time:1755161931284195, pushed_time:0, delay:500000}, running_task_token={this:0x7f991b7a4f18, timer:0x7f99535f3df0, task:0x7f9922ff6c40, task_type:"N9oceanbase7obmysql20ObConstructQueueTaskE", scheduled_time:1755161931282757, pushed_time:1755161931285404, delay:500000}, total_delay=484456397, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:55.742360] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=36] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:55.743225] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [553]OB_MALLOC COST TOO MUCH TIME, cost_time=1018291, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1018283], seq:[0] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=1016161, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1016159], seq:[0] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=1017358, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=15, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1017327, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=1017446, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1017379, WASH_BLOCK_END=67], seq:[0, 1, 2] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=1015254, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1015226, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=1015282, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1015264, WASH_BLOCK_END=18], seq:[0, 1, 2] [2025-08-14 09:06:55.751968] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1015323, owner: ObLog, click_count: 5, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=1015289, ALLOC_END=22, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:55.749647] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1017753, owner: ObLog, click_count: 6, time dist:[FORMAT_END=7, ObMalloc#start=0, ObMalloc#end=1017456, ObMalloc#start=29, ObMalloc#end=47, FREE_END=214], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:01:04.756643] EDIAG [SQL] do_close_plan (ob_result_set.cpp:844) [14786][T1002_TimerWK5_FreInfoReload][T1001][YB42AC4D0007-00063C4974599C72-0-0] [lt=22000][errcode=-4006] LOGGER COST TOO MUCH TIME, cost: 350448437, owner: ObLog, click_count: 7, time dist:[FORMAT_END=1011437, ObMalloc#start=1, ObMalloc#end=963270, ObMalloc#start=348473719, ObMalloc#end=4, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x14944de2 0x9395c9f 0x9725f91 0x124dfd77 0x96a5a4d 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x1fa4b892 0x1aec706b 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [385]OB_MALLOC COST TOO MUCH TIME, cost_time=1017257, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1017228, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [2025-08-14 09:06:54.882051] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=23310] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:06:55.747639] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:55.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:55.752079] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=11] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:55.757335] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:55.757702] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:55.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:55.763345] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:06:55.763730] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:55.765197] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 35.020576 loop:26707, slowcnt: 338 [2025-08-14 09:06:55.772829] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:55.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:55.775447] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=23] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:55.863741] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [385]OB_MALLOC COST TOO MUCH TIME, cost_time=1017313, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1017272, WASH_BLOCK_END=41], seq:[0, 1, 2] [2025-08-14 09:06:55.864872] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:06:55.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:55.963786] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:55.972960] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:55.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:55.991519] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=30] PNIO [ratelimit] time: 1755162415991517, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007617, add_bytes: 0 [2025-08-14 09:06:55.999244] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:55.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:56.063846] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:56.163872] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:56.240578] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:56.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:56.250609] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:06:56.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:56.263892] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:56.363951] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:56.463992] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:56.491955] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:06:56.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:56.564051] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:56.664112] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:56.721242] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:56.739948] WDIAG pop_task (ob_timer_service.cpp:565) [513][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f994e1a5df8, timer:0x7f994b3feee0, task:0x7f994b3fef50, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162415734941, pushed_time:0, delay:1000000}, running_task_token={this:0x7f994e1a5fb8, timer:0x7f994b3feee0, task:0x7f994b3fef80, task_type:"N9oceanbase8observer18ObTableLoadService21ObClientTaskPurgeTaskE", scheduled_time:1755162027715356, pushed_time:1755162414738704, delay:1000000}, total_delay=1005005, threashold=1000000, tenant_id=1) [2025-08-14 09:06:56.753344] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=10] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:56.759257] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:56.763448] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=11] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:06:56.764185] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:06:56.764211] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:06:56.231967] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:55.752033] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1017365, owner: ObLog, click_count: 5, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=1017328, ALLOC_END=24, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [383]OB_MALLOC COST TOO MUCH TIME, cost_time=1017104, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1017086], seq:[0, 1, 2] [383]OB_MALLOC COST TOO MUCH TIME, cost_time=1017125, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1017124], seq:[0, 1] [2025-08-14 09:06:55.752310] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1017143, owner: ObLog, click_count: 5, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=1017131, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [477]OB_MALLOC COST TOO MUCH TIME, cost_time=1020143, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=1020121, ALLOC_CHUNK_END=3], seq:[0, 1, 2, 3] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=1019243, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1019221, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=1020197, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1020160, WASH_BLOCK_END=36], seq:[0, 1, 2] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=1019279, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1019260, WASH_BLOCK_END=19], seq:[0, 1, 2] [2025-08-14 09:06:55.754191] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1019329, owner: ObLog, click_count: 5, time dist:[FORMAT_END=11, ObMalloc#start=0, ObMalloc#end=1019295, ALLOC_END=23, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [736]OB_MALLOC COST TOO MUCH TIME, cost_time=1019080, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1019060, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=1019101, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1019089, WASH_BLOCK_END=12], seq:[0, 1, 2] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=1018276, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1018254, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=1018293, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1018285, WASH_BLOCK_END=8], seq:[0, 1, 2] [2025-08-14 09:06:55.754506] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=63][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1019316, owner: ObLog, click_count: 6, time dist:[FORMAT_END=12, ObMalloc#start=0, ObMalloc#end=1019107, ObMalloc#start=18, ObMalloc#end=39, FREE_END=140], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:55.755428] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1018484, owner: ObLog, click_count: 6, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=1018299, ObMalloc#start=19, ObMalloc#end=55, FREE_END=101], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [833]OB_MALLOC COST TOO MUCH TIME, cost_time=1020057, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1020037, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [833]OB_MALLOC COST TOO MUCH TIME, cost_time=1020096, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1020071, WASH_BLOCK_END=24], seq:[0, 1, 2] [389]OB_MALLOC COST TOO MUCH TIME, cost_time=1017425, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1017404, ALLOC_CHUNK_END=3], seq:[0, 1, 2, 3] [389]OB_MALLOC COST TOO MUCH TIME, cost_time=1017448, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1017433, WASH_BLOCK_END=15], seq:[0, 1, 2] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=1013185, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1013164, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=1013200, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1013192, WASH_BLOCK_END=8], seq:[0, 1, 2] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=1012764, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1012745, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=1012783, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1012776, WASH_BLOCK_END=7], seq:[0, 1, 2] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=1011874, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1011851, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [574]OB_MALLOC COST TOO MUCH TIME, cost_time=1011890, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1011883, WASH_BLOCK_END=7], seq:[0, 1, 2] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=1009942, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1009918, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=1009987, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1009958, WASH_BLOCK_END=29], seq:[0, 1, 2] [2025-08-14 09:06:55.758422] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1017848, owner: ObLog, click_count: 6, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=1017455, ObMalloc#start=19, ObMalloc#end=38, FREE_END=325], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [777]OB_MALLOC COST TOO MUCH TIME, cost_time=1004900, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1004878, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=1004927, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1004910, WASH_BLOCK_END=17], seq:[0, 1, 2] [2025-08-14 09:06:55.764288] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=22][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1012118, owner: ObLog, click_count: 6, time dist:[FORMAT_END=22, ObMalloc#start=1, ObMalloc#end=1011895, ObMalloc#start=23, ObMalloc#end=38, FREE_END=139], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [660]OB_MALLOC COST TOO MUCH TIME, cost_time=1003491, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=1003470, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=1003508, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1003499, WASH_BLOCK_END=9], seq:[0, 1, 2] [2025-08-14 09:06:55.762769] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [663][T1001_StandbySc][T1001][YB42AC4D0007-00063C495C7A03DB-0-0] [lt=25][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1013770, owner: ObLog, click_count: 6, time dist:[FORMAT_END=14, ObMalloc#start=0, ObMalloc#end=1013207, ObMalloc#start=28, ObMalloc#end=37, FREE_END=484], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:54.861319] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=21607][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1913881, owner: ObLog, click_count: 6, time dist:[FORMAT_END=892993, ObMalloc#start=0, ObMalloc#end=1020112, ObMalloc#start=21, ObMalloc#end=39, FREE_END=716], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [398]OB_MALLOC COST TOO MUCH TIME, cost_time=994217, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=994194, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [398]OB_MALLOC COST TOO MUCH TIME, cost_time=994235, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=994226, WASH_BLOCK_END=9], seq:[0, 1, 2] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=993712, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=3, ALLOC_CHUNK_END=993695, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=993730, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=993721, WASH_BLOCK_END=9], seq:[0, 1, 2] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=975483, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=975459, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=975501, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=975492, WASH_BLOCK_END=9], seq:[0, 1, 2] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=915505, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=915483, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=915523, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=915514, WASH_BLOCK_END=9], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=622180, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=10, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=622156, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=622198, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=622189, WASH_BLOCK_END=9], seq:[0, 1, 2] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=408278, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=408240, ALLOC_CHUNK_END=10], seq:[0, 1, 2, 3] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=408297, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=408287, WASH_BLOCK_END=10], seq:[0, 1, 2] [683]OB_MALLOC COST TOO MUCH TIME, cost_time=333521, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=333498, ALLOC_CHUNK_END=5], seq:[0, 1, 2, 3] [683]OB_MALLOC COST TOO MUCH TIME, cost_time=333539, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=333530, WASH_BLOCK_END=9], seq:[0, 1, 2] [2025-08-14 09:06:55.771473] EDIAG [RS] skip_check_member_list_change_ (ob_ls_recovery_stat_handler.cpp:109) [777][T1002_TeRec][T1002][YB42AC4D0007-00063C495DDA00B7-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1005996, owner: ObLog, click_count: 6, time dist:[FORMAT_END=8, ObMalloc#start=1, ObMalloc#end=1004935, ObMalloc#start=21, ObMalloc#end=42, FREE_END=989], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x1067fb2c 0x1067ecc2 0x1067e03d 0x1002f2a3 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:55.753268] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [477][T1_StandbySchem][T1][YB42AC4D0007-00063C495B6A03F8-0-0] [lt=18][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1024443, owner: ObLog, click_count: 6, time dist:[FORMAT_END=11, ObMalloc#start=1, ObMalloc#end=1020211, ObMalloc#start=26, ObMalloc#end=48, FREE_END=4146], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [772]OB_MALLOC COST TOO MUCH TIME, cost_time=1010882, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=1010856, ALLOC_CHUNK_END=12], seq:[0, 1, 2, 3] [772]OB_MALLOC COST TOO MUCH TIME, cost_time=1010904, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1010891, WASH_BLOCK_END=13], seq:[0, 1, 2] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=949961, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 4, time dist:[ALLOC_CHUNK_END=7, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=949944, ALLOC_CHUNK_END=3], seq:[0, 1, 2, 3] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=949972, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=949967, WASH_BLOCK_END=5], seq:[0, 1, 2] [2025-08-14 09:06:55.766880] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1011183, owner: ObLog, click_count: 8, time dist:[FORMAT_END=24, ObMalloc#start=1, ObMalloc#end=1010911, ObMalloc#start=20, ObMalloc#end=33, ObMalloc#start=73, ObMalloc#end=37, FREE_END=84], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:55.772981] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1005979, owner: ObLog, click_count: 8, time dist:[FORMAT_END=33, ObMalloc#start=0, ObMalloc#end=1003516, ObMalloc#start=26, ObMalloc#end=42, ObMalloc#start=799, ObMalloc#end=49, FREE_END=1514], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:55.766262] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1012819, owner: ObLog, click_count: 6, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=1009996, ObMalloc#start=23, ObMalloc#end=64, FREE_END=2726], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:55.763294] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1013165, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:56.444737] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1308567, owner: ObLog, click_count: 5, time dist:[FORMAT_END=28, ObMalloc#start=1, ObMalloc#end=333545, ALLOC_END=974992, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [374]OB_MALLOC COST TOO MUCH TIME, cost_time=2973064, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=2973057], seq:[0] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=2973380, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=67, WASH_BLOCK_END=128, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=2973096], seq:[0, 1, 2, 3, 4] [2025-08-14 09:06:55.744924] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2973449, owner: ObLog, click_count: 5, time dist:[FORMAT_END=37, ObMalloc#start=0, ObMalloc#end=2973411, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [553]OB_MALLOC COST TOO MUCH TIME, cost_time=2972387, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1018315], seq:[0, 1] [2025-08-14 09:06:54.872104] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=21940][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3845625, owner: ObLog, click_count: 5, time dist:[FORMAT_END=873218, ObMalloc#start=2, ObMalloc#end=2972401, ALLOC_END=4, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [472]OB_MALLOC COST TOO MUCH TIME, cost_time=2969473, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1016171], seq:[0, 1] [2025-08-14 09:06:55.749599] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=54][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2969507, owner: ObLog, click_count: 5, time dist:[FORMAT_END=24, ObMalloc#start=1, ObMalloc#end=2969481, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:56.173088] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:56.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:56.766492] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=592499][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:56.173|207=593.40ms|210=1us|222=2us|total=593.43ms) [2025-08-14 09:06:56.766524] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4389] timer task use too much time(end=1755162416766523, start=1755162416173086, delta=593437) [2025-08-14 09:06:55.751968] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:56.768846] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1015323][errcode=0] epoll handler too much time :1016890 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:56.768865] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:56.766494] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=533702][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:56.768899] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=2392][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:56.768910] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:55.752033] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:56.770957] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1017365][errcode=0] epoll handler too much time :1018935 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:56.770974] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:55.752310] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:56.771013] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1017143][errcode=0] epoll handler too much time :1018721 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:56.771029] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:55.754191] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:56.775085] WDIAG ev_invoke_pending (ev.c:2136) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1019329][errcode=0] epoll handler too much time :1020906 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:56.775111] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:56.155481] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=18] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:56.728274] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:56.778753] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=50402][errcode=0] epoll handler too much time :50491 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:56.765818] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 36.021196 loop:26707, slowcnt: 348 [2025-08-14 09:06:56.744747] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:56.865573] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:06:56.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:57.754417] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=887495][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:56.865|207=888.84ms|210=0us|222=2us|total=888.87ms) [2025-08-14 09:06:57.754442] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-4389] timer task use too much time(end=1755162417754441, start=1755162416865571, delta=888870) [2025-08-14 09:06:57.178959] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=622243] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:56.768920] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:57.755217] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=984799][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:56.999136] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=15] PNIO [ratelimit] time: 1755162416999134, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007617, add_bytes: 0 [2025-08-14 09:06:56.444737] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fa190, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa470, time_wheel:0x7f99341cdbf0, thread_pool:0x7f99341cd3d0, is_running:1, time_interval:2.00s, expected_run_time:09:06:56.334, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:57.755358] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=1308567][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:56.444|207=1.31s|210=0us|222=2us|total=1.31s) [2025-08-14 09:06:57.755414] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=54][errcode=-4389] timer task use too much time(end=1755162417755413, start=1755162416444734, delta=1310679) [2025-08-14 09:06:56.966651] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:56.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:56.775538] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=13] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:57.727262] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:57.732123] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=985883][errcode=0] epoll handler too much time :987384 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:57.751299] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=13] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:57.768579] WDIAG ev_invoke_pending (ev.c:2136) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=17253][errcode=0] epoll handler too much time :17283 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:57.751601] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:57.768596] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16967][errcode=0] epoll handler too much time :17005 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:57.752985] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:57.768610] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15601][errcode=0] epoll handler too much time :15634 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:57.754061] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:57.769096] WDIAG ev_invoke_pending (ev.c:2136) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15011][errcode=0] epoll handler too much time :15041 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:57.754040] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:57.769111] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15048][errcode=0] epoll handler too much time :15080 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:58.006754] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=755002] PNIO [ratelimit] time: 1755162418006751, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007617, add_bytes: 0 [2025-08-14 09:06:58.254837] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=574639] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:58.455785] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fa190, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_516, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fa470, time_wheel:0x7f99341cdbf0, thread_pool:0x7f99341cd3d0, is_running:1, time_interval:2.00s, expected_run_time:09:06:58.334, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:58.722828] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [683][T1001_FrzTrigge][T1001][Y0-0000000000000000-0-0] [lt=266635][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:58.455|207=267.04ms|210=0us|222=2us|total=267.07ms) [2025-08-14 09:06:55.744924] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72760b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:55.719, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:58.722912] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=2973449][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:55.744|207=2.98s|210=0us|222=2us|total=2.98s) [2025-08-14 09:06:58.722925] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task use too much time(end=1755162418722925, start=1755162415744910, delta=2978015) [2025-08-14 09:06:58.722937] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=2984690, expected_run_ts_us_=1755162415738245, ObClockGenerator::getRealClock()=1755162418722936, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:55.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:58.722951] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:55.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:58.722963] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=2986485, expected_run_ts_us_=1755162415736477, ObClockGenerator::getRealClock()=1755162418722962, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:55.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:58.722975] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:55.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:58.722985] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=2984477, expected_run_ts_us_=1755162415738508, ObClockGenerator::getRealClock()=1755162418722985, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739be30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:55.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:58.722996] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739be30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:55.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:58.723005] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=2991740, expected_run_ts_us_=1755162415731265, ObClockGenerator::getRealClock()=1755162418723005, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:55.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:58.723043] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:55.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:58.723055] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=2980472, expected_run_ts_us_=1755162415742583, ObClockGenerator::getRealClock()=1755162418723055, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7edc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:55.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:58.723092] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7edc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:06:55.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:54.872104] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=21940] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:06:55.749599] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=54][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:55.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:58.723633] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=2969507][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:55.749|207=2.97s|210=0us|222=1us|total=2.97s) [2025-08-14 09:06:58.723653] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4389] timer task use too much time(end=1755162418723653, start=1755162415749596, delta=2974057) [2025-08-14 09:06:57.755254] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:58.723798] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=967051][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:58.723833] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:58.723842] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:58.723864] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:58.723874] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:58.723887] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:58.723907] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:58.723968] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=60][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:58.723995] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:58.724008] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:57.755440] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=787593][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:56.966|207=788.79ms|210=0us|222=2us|total=788.82ms) [2025-08-14 09:06:58.724979] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=968074][errcode=-4389] timer task use too much time(end=1755162418724979, start=1755162416966649, delta=1758330) [2025-08-14 09:02:29.746648] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [319][pnio1][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1002, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162110998126", warnings:[]}, tenant_id_=1002, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [2025-08-14 09:06:57.768102] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=35911] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:58.725568] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=956018][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:57.854520] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:06:57.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:58.726015] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=870174][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:58.726039] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4389] timer task use too much time(end=1755162418726038, start=1755162417854517, delta=871521) [2025-08-14 09:06:55.740284] EDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2982419, owner: ObLog, click_count: 6, time dist:[FORMAT_END=2977577, ObMalloc#start=1, ObMalloc#end=93, ObMalloc#start=29, ObMalloc#end=124, FREE_END=4595], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x1b511e3c 0x95df1fc 0x95ded4c 0x1b4f8508 0x1b4fbf53 0x1b5014e0 0x1b5009d1 0x1b50387c 0x1b8905de 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:57.767979] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=40653][errcode=0] epoll handler too much time :40726 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:06:58.727312] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=957872] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:57.680824] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=349][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1045009, , BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:57.680824] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=349][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:57.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:58.727436] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162418727435, start=1755162417680820, delta=1046615) [2025-08-14 09:06:57.754108] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4015] clock out of order(cur_ts=1755162415755715, cur_ts_=1755162415728969, delta=26746) [2025-08-14 09:02:29.746648] EDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [319][pnio1][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4012] LOGGER COST TOO MUCH TIME, cost: 268572134, owner: ObLog, click_count: 3, time dist:[FORMAT_END=268572133, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1c02872e 0x1c032753 0x1c034ae1 0x97bc914 0x97bc4ca 0x27ad7f11 0x97b81a7 0x27a8a6d6 0x97b7ba0 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:57.751287] INFO [STORAGE] try_shrink (ob_compaction_memory_pool.cpp:518) [510][T1_TimerWK1_MergeMemPool][T1][Y0-0000000000000000-0-0] [lt=7] Compaction Mem Pool current stat: (ret=0, max_block_num=64, total_block_num=8, Chunk list size: =1) [2025-08-14 09:06:58.738932] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [510][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=986151][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f994e1a42d8, timer:0x7f9967075f10, task:0x7f994b3a1cb8, task_type:"N9oceanbase7storage25ObTenantCompactionMemPool17MemPoolShrinkTaskE", scheduled_time:1755162150757827, pushed_time:1755162150758417, delay:120000000}, delay_in_thread_pool=263976552, threashold=500000) [2025-08-14 09:06:58.742645] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=524][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:58.732913] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:200) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=2982419][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2025-08-14 09:06:58.743597] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=204] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:58.744303] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:58.750156] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=664] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:58.751907] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=8] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:58.751947] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:58.753077] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:58.753109] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:58.753603] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:58.754442] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EED9-0-0] [lt=1959101][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:58.754520] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EED9-0-0] [lt=24] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-08-14 09:06:10.727121] EDIAG [SQL.DAS] execute_dist_das_task (ob_data_access_service.cpp:177) [338][Occam][T1][YB42AC4D0007-00063C4960499C96-0-0] [lt=12805][errcode=-4038] LOGGER COST TOO MUCH TIME, cost: 47955779, owner: ObLog, click_count: 5, time dist:[FORMAT_END=47955745, ObMalloc#start=1, ObMalloc#end=5, ALLOC_END=22, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x15f36e49 0x95f5ef0 0x9344f39 0x9343c25 0x9615b91 0x9613406 0x9613912 0x92df373 0x932ff51 0x932e8be 0x96ac524 0x96a672d 0x96a5033 0x97aa16f 0x124eac91 0x97a9944 0x261395f9 0x1f960c89 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:10.727121] WDIAG [SQL.DAS] execute_dist_das_task (ob_data_access_service.cpp:177) [338][Occam][T1][YB42AC4D0007-00063C4960499C96-0-0] [lt=12805][errcode=-4038] do local das task failed(ret=-4038, task_arg={timeout_ts:1755162383740243, ctrl_svr:"172.77.0.7:2882", runner_svr:"172.77.0.7:2882", task_ops:[{parent:{tenant_id:1, task_id:50056315, op_type:2, errcode:-4038, can_part_retry:0, task_started:1, in_part_retry:0, in_stmt_retry:0, need_switch_param:0, trans_desc:{this:0x7f98b2da56e0, tx_id:{txid:21195874}, state:1, addr:"172.77.0.7:2882", tenant_id:1, session_id:1, assoc_session_id:1, client_sid:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1755162356734727, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999983, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1753069619, cluster_version:17180067074, seq_base:1755162225746389, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:2}, snapshot:{this:0x7f995ab50cb8, valid:true, source:5, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:21195874}, scn:{branch:0, seq:2}}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, tablet_id:{id:154}, ls_id:{id:1}, tablet_loc:{tablet_id:{id:154}, ls_id:{id:1}, server:"172.77.0.7:2882", loc_meta:{table_loc_id:18446744073709551614, ref_table_id:154, related_table_ids:[101015, 101016], use_dist_das:0, select_leader:1, is_dup_table:0, is_weak_read:0, unuse_related_pruning:0, is_external_table:0, route_policy:0}, in_retry:0, partition_id:18446744073709551615, first_level_part_id:18446744073709551615}, related_ctdefs:[{ObDASBaseCtDef:{ObDASBaseDef:{op_type:2, children_cnt:0}, table_id:18446744073709551614, index_tid:101015, schema_version:1753069637599528, rowkey_cnt:4, spk_cnt:0, column_ids:[19, 16, 17, 18], column_types:[{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}], column_accuracys:[{length:64, precision:-1, scale:-1}, {length:-1, precision:25, scale:6}, {length:46, precision:-1, scale:-1}, {length:-1, precision:20, scale:0}], old_row_projector:[], new_row_projector:[3, 0, 1, 2], is_total_quantity_log:1, is_ignore:0, is_batch_stmt:0, is_insert_up:0, is_table_api:0, is_main_table_in_fts_ddl:0, tz_info:{id:-1, offset:28800, error_on_overlap_time:false}, table_param:{tenant_schema_version:1753073979411448, data_table:{table_id:101015, schema_version:1753069637599528, table_type:5, index_type:1, index_status:2, shadow_rowkey_column_num:0, doc_id_col_id:18446744073709551615, fulltext_col_id:18446744073709551615, index_name:"idx_server_module", fts_parser_name:"", fts_parser_properties:"", pk_name:"", columns:[{column_id:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:64, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:25, scale:6}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"EXT":"CURRENT_TIMESTAMP"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}, {column_id:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:46, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}, {column_id:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}], read_info:{schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, trans_col_index:-1, mview_old_new_col_index:-1, group_idx_col_index:-1, seq_read_column_count:4, max_col_index:3, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 2, 3], data:0x7f97ccef5b10, count:4, allocator:0x7f97fb64efb0, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 3], data:0x7f97ccef5b30, count:4, allocator:0x7f97fb64efb0, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f97ccef5ad0, count:4, allocator:0x7f97fb64efb0, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, has_all_column_group:true, need_truncate_filter:false}, lob_inrow_threshold:4096}, col_descs:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], col_map:{is_inited_:true, use_array_:true, shadow_use_array_:false, has_:true, has_shadow_:false}}, encrypt_meta:[]}, table_rowkey_cids:[], table_rowkey_types:[]}, {ObDASBaseCtDef:{ObDASBaseDef:{op_type:2, children_cnt:0}, table_id:18446744073709551614, index_tid:101016, schema_version:1753069637600200, rowkey_cnt:4, spk_cnt:0, column_ids:[20, 16, 17, 18], column_types:[{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}], column_accuracys:[{length:64, precision:-1, scale:-1}, {length:-1, precision:25, scale:6}, {length:46, precision:-1, scale:-1}, {length:-1, precision:20, scale:0}], old_row_projector:[], new_row_projector:[4, 0, 1, 2], is_total_quantity_log:1, is_ignore:0, is_batch_stmt:0, is_insert_up:0, is_table_api:0, is_main_table_in_fts_ddl:0, tz_info:{id:-1, offset:28800, error_on_overlap_time:false}, table_param:{tenant_schema_version:1753073979411448, data_table:{table_id:101016, schema_version:1753069637600200, table_type:5, index_type:1, index_status:2, shadow_rowkey_column_num:0, doc_id_col_id:18446744073709551615, fulltext_col_id:18446744073709551615, index_name:"idx_server_event", fts_parser_name:"", fts_parser_properties:"", pk_name:"", columns:[{column_id:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:64, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:25, scale:6}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"EXT":"CURRENT_TIMESTAMP"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}, {column_id:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:46, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}, {column_id:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}], read_info:{schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, trans_col_index:-1, mview_old_new_col_index:-1, group_idx_col_index:-1, seq_read_column_count:4, max_col_index:3, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 2, 3], data:0x7f97ccef6a90, count:4, allocator:0x7f97fb64efb0, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 3], data:0x7f97ccef6ab0, count:4, allocator:0x7f97fb64efb0, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f97ccef6a50, count:4, allocator:0x7f97fb64efb0, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, has_all_column_group:true, need_truncate_filter:false}, lob_inrow_threshold:4096}, col_descs:[column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], col_map:{is_inited_:true, use_array_:true, shadow_use_array_:false, has_:true, has_shadow_:false}}, encrypt_meta:[]}, table_rowkey_cids:[], table_rowkey_types:[]}], related_rtdefs:[{ObDASBaseRtDef:{ObDASBaseRtDef:{op_type:2, children_cnt:0}, timeout_ts:1755162383640243, sql_mode:4194304, prelock:false, tenant_schema_version:1753073979411448, is_for_foreign_key_check:false, affected_rows:0, is_immediate_row_conflict_check:true}, need_fetch_conflict:false, is_duplicated:false, direct_insert_task_id:0, use_put:false, ddl_task_id:0}, {ObDASBaseRtDef:{ObDASBaseRtDef:{op_type:2, children_cnt:0}, timeout_ts:1755162383640243, sql_mode:4194304, prelock:false, tenant_schema_version:1753073979411448, is_for_foreign_key_check:false, affected_rows:0, is_immediate_row_conflict_check:true}, need_fetch_conflict:false, is_duplicated:false, direct_insert_task_id:0, use_put:false, ddl_task_id:0}], task_status:1, related_tablet_ids:[{id:101015}, {id:101016}], das_task_node:, plan_line_id:0}, ins_ctdef:{ObDASBaseCtDef:{ObDASBaseDef:{op_type:2, children_cnt:0}, table_id:18446744073709551614, index_tid:154, schema_version:1753069637597680, rowkey_cnt:3, spk_cnt:0, column_ids:[16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33], column_types:[{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"LONGTEXT", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}], column_accuracys:[{length:-1, precision:25, scale:6}, {length:46, precision:-1, scale:-1}, {length:-1, precision:20, scale:0}, {length:64, precision:-1, scale:-1}, {length:64, precision:-1, scale:-1}, {length:256, precision:-1, scale:-1}, {length:512, precision:-1, scale:-1}, {length:256, precision:-1, scale:-1}, {length:536870911, precision:-1, scale:-1}, {length:256, precision:-1, scale:-1}, {length:512, precision:-1, scale:-1}, {length:256, precision:-1, scale:-1}, {length:512, precision:-1, scale:-1}, {length:256, precision:-1, scale:-1}, {length:512, precision:-1, scale:-1}, {length:256, precision:-1, scale:-1}, {length:512, precision:-1, scale:-1}, {length:512, precision:-1, scale:-1}], old_row_projector:[], new_row_projector:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17], is_total_quantity_log:1, is_ignore:0, is_batch_stmt:0, is_insert_up:0, is_table_api:0, is_main_table_in_fts_ddl:0, tz_info:{id:-1, offset:28800, error_on_overlap_time:false}, table_param:{tenant_schema_version:1753073979411448, data_table:{table_id:154, schema_version:1753069637597680, table_type:0, index_type:0, index_status:0, shadow_rowkey_column_num:0, doc_id_col_id:18446744073709551615, fulltext_col_id:18446744073709551615, index_name:"", fts_parser_name:"", fts_parser_properties:"", pk_name:"", columns:[{column_id:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:25, scale:6}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"EXT":"CURRENT_TIMESTAMP"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}, {column_id:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:46, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}, {column_id:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:true}, {column_id:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:64, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:64, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:256, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:512, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:256, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:24, meta_type:{type:"LONGTEXT", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:536870911, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:25, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:256, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:26, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:512, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:27, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:256, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:28, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:512, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:29, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:256, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:30, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:512, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:31, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:256, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:32, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:512, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}, {column_id:33, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:512, precision:-1, scale:-1}, orig_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, cur_default_value:{"VARCHAR":"", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144, is_data_table_rowkey:false}], read_info:{schema_column_count:18, schema_rowkey_cnt:3, rowkey_cnt:5, trans_col_index:-1, mview_old_new_col_index:-1, group_idx_col_index:-1, seq_read_column_count:3, max_col_index:19, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:18, 0, 1, 2, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19], data:0x7f97ccef4c60, count:18, allocator:0x7f97fb64efb0, capacity:18, init_cnt:18}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:18, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17], data:0x7f97ccef4cb0, count:18, allocator:0x7f97fb64efb0, capacity:18, init_cnt:18}}, cols_desc:{ObIArray:[cnt:18, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=23 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=24 {type:"LONGTEXT", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=25 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=26 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=27 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=28 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=29 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=30 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=31 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=32 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=33 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0], data:0x7f97ccef4b80, count:18, allocator:0x7f97fb64efb0, capacity:18, init_cnt:18}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, has_all_column_group:true, need_truncate_filter:false}, lob_inrow_threshold:4096}, col_descs:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=23 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=24 {type:"LONGTEXT", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=25 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=26 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=27 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=28 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=29 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=30 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=31 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=32 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=33 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0], col_map:{is_inited_:true, use_array_:true, shadow_use_array_:false, has_:true, has_shadow_:false}}, encrypt_meta:[]}, table_rowkey_cids:[], table_rowkey_types:[]}, ins_rtdef:{ObDASBaseRtDef:{ObDASBaseRtDef:{op_type:2, children_cnt:0}, timeout_ts:1755162383640243, sql_mode:4194304, prelock:false, tenant_schema_version:1753073979411448, is_for_foreign_key_check:false, affected_rows:0, is_immediate_row_conflict_check:true}, need_fetch_conflict:false, is_duplicated:false, direct_insert_task_id:0, use_put:false, ddl_task_id:0}, insert_buffer:{mem_attr:tenant_id=1, label=DASInsertBuffer, ctx_id=0, prio=0, buffer_memory:408, buffer_size:1, datum_store:NULL}}], remote_info:NULL}) [2025-08-14 09:06:58.755890] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=978794] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:58.756286] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=186] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:58.757774] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=215] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:58.757805] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=174] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:58.760249] WDIAG [SQL.DAS] execute_das_task (ob_data_access_service.cpp:108) [338][Occam][T1][YB42AC4D0007-00063C4960499C96-0-0] [lt=47955779][errcode=-4038] failed to execute dist das task(ret=-4038) [2025-08-14 09:06:58.761758] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:58.761786] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:58.761897] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=206] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:58.762983] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:58.765051] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=252] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:06:58.766068] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:58.766557] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=421] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:58.775695] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=306][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:58.775720] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=172] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:58.778376] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=246] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:58.782392] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=188] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:58.783136] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=178] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:58.801627] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=181] gc stale ls task succ [2025-08-14 09:06:58.823727] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:58.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:58.825070] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:58.826113] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:58.827512] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:58.835003] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=964702][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:58.923795] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:58.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:58.994409] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:59.014367] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=15] PNIO [ratelimit] time: 1755162419014365, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007614, add_bytes: 0 [2025-08-14 09:06:59.023861] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:58.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:59.042733] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a7678, timer:0x7f994b1dfb70, task:0x7f9920110450, task_type:"N9oceanbase10rootserver21ObMLogMaintenanceTaskE", scheduled_time:1755161754754755, pushed_time:0, delay:0}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=664287977, threashold=1000000, tenant_id=1002) [2025-08-14 09:06:59.123922] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:59.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:59.223947] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:59.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:59.276918] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc4e38, timer:0x7f9970f40f80, task:0x564d6d07c140, task_type:"N9oceanbase7storage25ObRedefTableHeartBeatTaskE", scheduled_time:1755161637759743, pushed_time:0, delay:30000000}, running_task_token={this:0x7f9970fc5fb8, timer:0x7f9970f40f80, task:0x564d727182f0, task_type:"N9oceanbase5share16ObAshRefreshTaskE", scheduled_time:1755161611717613, pushed_time:1755161614757742, delay:120000000}, total_delay=781517174, threashold=1000000, tenant_id=500) [2025-08-14 09:06:59.323190] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=467268] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:06:59.323988] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:59.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:59.369939] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=188] gc stale ls task succ [2025-08-14 09:06:59.424049] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:59.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:59.524120] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:59.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:59.624184] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:59.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:59.721043] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=212] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:59.724302] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:59.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:59.728250] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:59.742380] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:59.744257] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:57.733492] EDIAG [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [807][T1002_TimerWK3_TabletShell][T1002][Y0-0000000000000000-0-0] [lt=2150][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2012021, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2012011, ObMalloc#start=1, ObMalloc#end=4, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1c45bdeb 0x1c45a909 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:57.733492] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [807][T1002_TimerWK3_TabletShell][T1002][Y0-0000000000000000-0-0] [lt=2150] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-08-14 09:06:59.749536] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:59.752261] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:59.752297] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=24] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:59.752341] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:59.752370] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:59.756015] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=21] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:06:59.758263] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:59.758303] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:59.758335] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:59.758358] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:59.760942] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:58.756348] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:981) [368][VTblLocAsyncUp0][T1][YB42AC4D0007-00063C4968299C62-0-0] [lt=11][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-4012, ret=-4012) [2025-08-14 09:06:58.756348] EDIAG [SERVER] query (ob_inner_sql_connection.cpp:981) [368][VTblLocAsyncUp0][T1][YB42AC4D0007-00063C4968299C62-0-0] [lt=11][errcode=-4012] LOGGER COST TOO MUCH TIME, cost: 1005500, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1005485, ObMalloc#start=1, ObMalloc#end=5, ALLOC_END=0, APPEND_END=9], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x124e01a8 0x96a5705 0x96a366f 0x96622b5 0x9661c70 0x9661b3e 0x9661456 0x9a77e58 0x1fcfa594 0x1f7fa1c3 0x2023397e 0x202381d6 0x13ad9dbd 0x13ad61b5 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:59.765274] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:06:59.765416] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:59.765415] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:213) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-08-14 09:06:59.765448] INFO [COMMON] handle (memory_dump.cpp:544) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=32] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-08-14 09:06:59.766319] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] update_check_range(min_check_version=2540, max_check_version=2540, global_version=2651) [2025-08-14 09:06:59.766877] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:06:59.766904] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:06:59.766924] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:06:59.766936] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:06:59.766949] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:06:59.766962] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:06:59.766985] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:06:59.766998] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:06:59.767016] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:06:59.767029] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:06:59.767047] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:06:59.767061] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:06:59.767074] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:06:59.767086] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:06:59.767099] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:06:59.767113] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:06:59.767139] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:06:59.767426] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:06:59.767443] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:06:58.759610] EDIAG [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=1947514][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1016082, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1016074, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1fce4e91 0x1fce455d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:58.759610] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=1947514] blacklist_loop exec finished(cost_time=16, is_enabled=true, send_cnt=0) [2025-08-14 09:06:59.778480] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:59.780275] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=17] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:06:59.782432] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:59.786892] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:59.787263] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:59.787305] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:59.801691] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=21] gc stale ls task succ [2025-08-14 09:06:59.824364] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=53][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:59.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:06:59.826463] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:59.826647] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:59.828124] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:59.835629] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:59.924427] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:59.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:00.018881] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:00.021984] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=37] PNIO [ratelimit] time: 1755162420021982, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007617, add_bytes: 0 [2025-08-14 09:07:00.024492] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:06:59.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:00.124554] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:00.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:00.224616] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:00.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:00.267624] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:07:00.267658] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:07:00.267680] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:07:00.267692] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:07:00.267712] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:07:00.267726] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:07:00.267743] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:07:00.267756] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:07:00.267772] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:07:00.267783] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:07:00.267799] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:07:00.267812] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:07:00.267825] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:07:00.267836] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:07:00.267858] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:07:00.267871] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:07:00.267888] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:07:00.268064] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:07:00.268079] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:07:00.323917] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=20] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:07:00.324680] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:00.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:00.370024] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=23] gc stale ls task succ [2025-08-14 09:07:00.424744] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:00.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:00.524808] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:00.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:00.624871] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:00.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:00.721073] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=31] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:00.724941] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:00.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:00.728224] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:54.756645] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [561][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=6280] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=16, total_destructed_detector=16, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f994b3ee2b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-08-14 09:07:00.743406] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:00.749568] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:00.751251] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=19] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:00.752259] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:00.753505] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:00.753605] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:00.754265] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:00.756114] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=32] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:07:00.756574] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:00.757564] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:00.758397] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:52.767545] EDIAG [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1004558][errcode=0] LOGGER COST TOO MUCH TIME, cost: 7980461, owner: ObLog, click_count: 5, time dist:[FORMAT_END=7980448, ObMalloc#start=2, ObMalloc#end=4, ALLOC_END=0, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x9c6174d 0xfd347d6 0xfd344b7 0xfd34254 0x1c029398 0x97a1ad0 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:52.767545] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1004558] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1755161719775179002, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=690991823, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.77.0.7:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=252386, succ_cluster_heartbeat_count=1905, cluster_heartbeat_interval=100000, local_cluster_version={val:1755161003593738002, v:0}, local_cluster_delta=1407173264, force_self_check=false, weak_read_refresh_interval=100000) [2025-08-14 09:07:00.760955] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:00.765416] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=34] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:07:00.765476] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:00.768275] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:07:00.768311] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:07:00.768346] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:07:00.768359] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:07:00.768379] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:07:00.768392] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:07:00.768407] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:07:00.768421] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:07:00.768449] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:07:00.768461] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:07:00.768477] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:07:00.768490] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:07:00.768503] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:07:00.768515] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:07:00.768535] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:07:00.768548] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:07:00.768565] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:07:00.768740] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:07:00.768754] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:07:00.778561] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=21] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:07:00.780373] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=14] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:07:00.782264] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:00.782394] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:00.794090] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:00.801754] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=43] gc stale ls task succ [2025-08-14 09:07:00.803920] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:00.824947] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:00.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:00.828633] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:00.836725] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:00.925019] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:00.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:00.927120] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:01.025073] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:00.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:01.095805] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:01.125139] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:01.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:01.225200] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:01.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:01.268934] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:07:01.268966] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:07:01.268985] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:07:01.269001] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:07:01.269017] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:07:01.269032] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:07:01.269050] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:07:01.269063] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:07:01.269090] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:07:01.269106] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:07:01.269121] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:07:01.269144] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:07:01.269161] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:07:01.269174] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:07:01.269183] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:07:01.269200] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:07:01.269217] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:07:01.269415] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:07:01.269434] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:07:01.324505] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=29] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:07:01.325271] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:01.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:01.370085] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=23] gc stale ls task succ [2025-08-14 09:07:01.425334] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:01.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:01.525397] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:01.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:01.625461] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:01.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:01.722328] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=38] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:01.725526] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:01.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:01.728258] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=39] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:59.770888] EDIAG [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1981641][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1957359, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1957342, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=1, APPEND_END=11], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1bf2ec62 0x986ebd7 0x986bee6 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:59.777421] EDIAG [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=1979105][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1950850, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1950837, ALLOC_END=1, APPEND_END=12], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1bf2ec62 0x986ebd7 0x986bee6 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:54.756645] EDIAG [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [561][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=6280][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5967830, owner: ObLog, click_count: 5, time dist:[FORMAT_END=5967819, ObMalloc#start=2, ObMalloc#end=4, ALLOC_END=0, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x973b640 0x973b132 0x973a32d 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:59.770888] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1981641] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:06:59.777421] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=1979105] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2025-08-14 09:06:59.777340] WDIAG [SQL] do_close (ob_result_set.cpp:935) [14786][T1002_TimerWK5_FreInfoReload][T1001][YB42AC4D0007-00063C4974599C72-0-0] [lt=350448437][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:01.731270] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [763][T1002_Occam][T1002][YB42AC4D0007-00063C495DB9EED9-0-0] [lt=48][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:06:59.877687] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=3845625][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1850998, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1850995, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:56.731570] EDIAG [SHARE] refresh (ob_task_define.cpp:396) [280][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=972180][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4992419, owner: ObLog, click_count: 5, time dist:[FORMAT_END=4992416, ObMalloc#start=0, ObMalloc#end=2, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1fd6cd17 0x96c37e0 0x1fda2168 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [776]OB_MALLOC COST TOO MUCH TIME, cost_time=1965774, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1965773], seq:[0] [2025-08-14 09:06:58.756822] EDIAG [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2977161, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2977159, ALLOC_END=2, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x12377491 0x1236e8e9 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:58.882635] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=333][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2858993, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2858992, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:58.747798] EDIAG [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2997508, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2997507, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x260b4cc1 0x260b9960 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [776]OB_MALLOC COST TOO MUCH TIME, cost_time=1980946, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=62, WASH_BLOCK_END=1268, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1965791], seq:[0, 1, 2, 3, 4] [2025-08-14 09:06:59.768064] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [776][T1002_StandbySc][T1002][YB42AC4D0007-00063C495DFA03F3-0-0] [lt=18][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1980980, owner: ObLog, click_count: 5, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=1980965, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:59.888369] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=860987][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1870247, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1840309, ObMalloc#start=1, ObMalloc#end=29918, ALLOC_END=19, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:59.868076] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=1913881][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1890767, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1860546, ObMalloc#start=0, ObMalloc#end=29873, ObMalloc#start=18, ObMalloc#end=36, FREE_END=294], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:00.733288] EDIAG pn_print_diag_info (group.c:775) [322][pnio2][T0][Y0-0000000000000000-0-0] [lt=978863][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1027093, owner: ObLog, click_count: 5, time dist:[FORMAT_END=996529, ObMalloc#start=0, ObMalloc#end=30562, ALLOC_END=2, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x27a90dc3 0x27a8aaf2 0x97b7c8d 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:59.766815] EDIAG [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1992394, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1980109, ObMalloc#start=1, ObMalloc#end=12266, ALLOC_END=18, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a5bbf2 0x94da542 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:59.765553] EDIAG [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [411][sql_nio4][T0][Y0-0000000000000000-0-0] [lt=16990][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1993726, owner: ObLog, click_count: 7, time dist:[FORMAT_END=1981489, ObMalloc#start=1, ObMalloc#end=12217, ObMalloc#start=17, ObMalloc#end=2, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a5bbf2 0x94da542 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:59.769282] EDIAG [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1993322, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1988482, ObMalloc#start=1, ObMalloc#end=4665, ObMalloc#start=21, ObMalloc#end=35, FREE_END=118], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a5bbf2 0x94da542 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:59.777379] EDIAG [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [412][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1985344, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1975685, ObMalloc#start=1, ObMalloc#end=9268, ObMalloc#start=25, ObMalloc#end=164, FREE_END=201], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a5bbf2 0x94da542 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:59.877687] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=3845625] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:06:56.731570] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [280][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=972180][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:45, tid:660}]) [2025-08-14 09:06:58.756822] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=5] finish to resize log external storage handler(current_ts=1755162418756815, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-08-14 09:07:01.745577] WDIAG [DETECT] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [561][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=5967830][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:01.738084] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:763) [368][VTblLocAsyncUp0][T0][YB42AC4D0007-00063C4968299C62-0-0] [lt=1005500] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, process_time=281021619) [2025-08-14 09:06:58.882635] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=333][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:01.745987] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=2858993][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:01.746001] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task use too much time(end=1755162421746001, start=1755162418882633, delta=2863368) [2025-08-14 09:07:01.749469] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:58.747798] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=17] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25861652480, fast_delete_log_mode=false) [2025-08-14 09:07:01.761164] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=31] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:01.761186] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:01.761207] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:01.761225] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:06:59.888369] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=860987] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:07:00.733288] INFO pn_print_diag_info (group.c:775) [322][pnio2][T0][Y0-0000000000000000-0-0] [lt=978863] PNIO client_send:0/0, client_queue_time=0, cnt=0, server_send:0/0, server_queue_time=0, cnt=0 [2025-08-14 09:07:01.745904] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1018) [368][VTblLocAsyncUp0][T0][YB42AC4D0007-00063C4968299C62-0-0] [lt=7805][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-4012, ret=-4012) [2025-08-14 09:06:59.766815] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] [sql nio session](*s={this:0x7f98f72d0030, session_id:3221589415, trace_id:YB42AC4D0007-00063C4965FA43DC-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:245}, err:0, last_decode_time:1755161751714366, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1662, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:07:01.762249] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=1992394] [sql nio session](*s={this:0x7f98f72d5030, session_id:3221590200, trace_id:YB42AC4D0007-00063C4965FA43E0-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:248}, err:0, last_decode_time:1755161776777167, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1593, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:06:59.765553] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [411][sql_nio4][T0][Y0-0000000000000000-0-0] [lt=16990] [sql nio session](*s={this:0x7f98f72c3030, session_id:3221590336, trace_id:YB42AC4D0007-00063C49634A0941-0-0, sql_handling_stage:3, sql_initiative_shutdown:false, reader:{fd:255}, err:5, last_decode_time:1755162354747599, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1422, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:07:01.761625] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16619][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:01.731441] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=897447][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:07:01.766005] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:07:01.731636] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=897916][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:06:59.775670] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=17] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:01.767036] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=2977161] snapshot_traffic_info once [2025-08-14 09:06:59.771713] EDIAG [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1993521, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1993501, ObMalloc#start=16, ObMalloc#end=3, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1d189030 0x1d18285e 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:59.777155] EDIAG [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1988161, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1988158, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1d189030 0x1d18285e 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:59.775670] EDIAG [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1988343, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1988343, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x99721a1 0x997a72e 0x2006ed17 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:00.733283] EDIAG pn_print_diag_info (group.c:775) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1045491, owner: ObLog, click_count: 5, time dist:[FORMAT_END=996523, ObMalloc#start=0, ObMalloc#end=48965, ALLOC_END=3, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x27a90dc3 0x27a8aaf2 0x97b7c8d 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:00.745559] EDIAG [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:171) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=10616][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1970356, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1968387, ObMalloc#start=2, ObMalloc#end=61, ObMalloc#start=28, ObMalloc#end=38, FREE_END=1840], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1b538df0 0x1b4f8d72 0x1b4fbf53 0x1b5014e0 0x1b5009d1 0x1b50387c 0x1b8905de 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:59.998839] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2729024, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=8, ALLOC_BLOCK#end=3, FORMAT_END=2729012, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [807]OB_MALLOC COST TOO MUCH TIME, cost_time=969924, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=969917], seq:[0] [2025-08-14 09:07:00.744853] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1984410, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1984408, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [807]OB_MALLOC COST TOO MUCH TIME, cost_time=999372, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=14211, WASH_BLOCK_END=15222, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=969933], seq:[0, 1, 2, 3, 4] [2025-08-14 09:07:00.734118] EDIAG [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [807][T1002_TimerWK3_TabletShell][T1002][Y0-0000000000000000-0-0] [lt=2012021][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1995169, owner: ObLog, click_count: 7, time dist:[FORMAT_END=995780, ObMalloc#start=0, ObMalloc#end=999386, ALLOC_END=3, APPEND_END=0, ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=2], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1c45cbdf 0x1c45b785 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:59.771713] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=35] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:07:01.731315] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:493) [14862][T1001_TimerWK5_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=10004938] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-08-14 09:06:59.777155] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=30] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-08-14 09:07:01.767126] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:497) [510][T1_TimerWK1_SSTableDefragme][T1][Y0-0000000000000000-0-0] [lt=31811] successfully defragment data blocks(ret=0, rewrite_cnt=0, block_used_size_.count()=12) [2025-08-14 09:07:00.733283] INFO pn_print_diag_info (group.c:775) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=25] PNIO client_send:0/0, client_queue_time=0, cnt=0, server_send:0/0, server_queue_time=0, cnt=0 [2025-08-14 09:07:01.780287] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=577][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:07:01.780377] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=85][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:07:01.780395] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=1045491] PNIO [ratelimit] time: 1755162421780393, bytes: 0, bw: 0.000000 MB/s, add_ts: 1758411, add_bytes: 0 [2025-08-14 09:07:01.738337] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:233) [529][T1_RFLWorker][T1][YB42AC4D0007-00063C495B199C16-0-0] [lt=14263] ObRemoteFetchWorker is running(thread_index=0) [2025-08-14 09:07:01.770778] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:118) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=228] try to retry start_working [2025-08-14 09:07:01.782373] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:01.783282] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:01.795663] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:01.805701] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:01.825589] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:01.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:01.830383] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:01.852140] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [280][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=4992419][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:407, tid:472}]) [2025-08-14 09:07:01.906301] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a59b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:01.895, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:01.925653] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:01.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:01.930449] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:01.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:02.016937] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:02.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:02.016967] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:02.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:02.025717] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:01.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:02.027052] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:02.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:02.030510] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:01.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:02.097454] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:02.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:02.218161] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:02.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:02.268453] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:02.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:02.268490] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:02.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:02.520035] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:02.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:02.520063] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:02.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:02.590470] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:02.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:02.711118] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:02.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:02.722588] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=44] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:01.829392] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:02.125780] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:02.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:02.729414] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=602718][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:02.125|207=603.63ms|210=0us|222=2us|total=603.65ms) [2025-08-14 09:07:02.729434] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4389] timer task use too much time(end=1755162422729433, start=1755162422125779, delta=603654) [2025-08-14 09:07:02.130575] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:02.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:02.729462] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=597979][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:02.130|207=598.88ms|210=1us|222=0us|total=598.90ms) [2025-08-14 09:07:02.729474] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] timer task use too much time(end=1755162422729474, start=1755162422130574, delta=598900) [2025-08-14 09:07:02.278535] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:02.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:02.729495] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=450276][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:02.278|207=450.96ms|210=0us|222=2us|total=450.97ms) [2025-08-14 09:07:02.729505] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task use too much time(end=1755162422729505, start=1755162422278532, delta=450973) [2025-08-14 09:07:02.729513] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=5][errcode=0] task scheduled too delay(scheduled_delay=437162, expected_run_ts_us_=1755162422292349, ObClockGenerator::getRealClock()=1755162422729512, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:02.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:02.729528] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:02.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:01.780569] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=1988161] temporary file module use no blocks [2025-08-14 09:06:59.998839] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:02.732019] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=2729024][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:06:59.998|207=2.73s|210=0us|222=2us|total=2.73s) [2025-08-14 09:07:02.732036] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] timer task use too much time(end=1755162422732035, start=1755162419998837, delta=2733198) [2025-08-14 09:07:02.732046] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] task scheduled too delay(scheduled_delay=2487412, expected_run_ts_us_=1755162420244633, ObClockGenerator::getRealClock()=1755162422732046, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:00.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:02.732056] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:00.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:01.767241] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [854][T1002_ObXAHbWor][T1002][Y0-0000000000000000-0-0] [lt=23] XA scheduler heartbeat task statistics(avg_time=977006) [2025-08-14 09:07:00.744853] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:00.734118] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [807][T1002_TimerWK3_TabletShell][T1002][Y0-0000000000000000-0-0] [lt=2012021] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=2, times=1265) [2025-08-14 09:07:02.732301] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1984410][errcode=0] epoll handler too much time :1987457 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:02.732320] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:02.732367] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=1995169][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a5458, timer:0x7f9922ffd748, task:0x7f9922ffd770, task_type:"N9oceanbase7storage10checkpoint16ObEmptyShellTaskE", scheduled_time:1755162393737357, pushed_time:1755162393756601, delay:5000000}, delay_in_thread_pool=21982326, threashold=500000) [2025-08-14 09:07:02.733365] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.726, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:02.733400] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739eb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.721, task_priority:1, with_handle_protected:0}) [2025-08-14 09:06:58.760282] EDIAG [SQL.DAS] execute_all_task (ob_das_ref.cpp:385) [338][Occam][T1][YB42AC4D0007-00063C4960499C96-0-0] [lt=28][errcode=-4038] LOGGER COST TOO MUCH TIME, cost: 3999781, owner: ObLog, click_count: 8, time dist:[FORMAT_END=3998884, ObMalloc#start=1, ObMalloc#end=76, ObMalloc#start=41, ObMalloc#end=66, ObMalloc#start=578, ObMalloc#end=50, FREE_END=85], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x15ebc05e 0x9344205 0x9615b91 0x9613406 0x9613912 0x92df373 0x932ff51 0x932e8be 0x96ac524 0x96a672d 0x96a5033 0x97aa16f 0x124eac91 0x97a9944 0x261395f9 0x1f960c89 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:06:58.765824] EDIAG [RS] get_valid_sts_after (ob_tenant_info_loader.cpp:454) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=326][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3995761, owner: ObLog, click_count: 6, time dist:[FORMAT_END=3995596, ObMalloc#start=1, ObMalloc#end=62, ObMalloc#start=24, ObMalloc#end=41, FREE_END=37], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1001d942 0x996d843 0x996d15a 0x996cfd8 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [416]OB_MALLOC COST TOO MUCH TIME, cost_time=1002934, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1002927], seq:[0] [416]OB_MALLOC COST TOO MUCH TIME, cost_time=1002974, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1002954], seq:[0, 1] [2025-08-14 09:07:01.767285] EDIAG [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1972401, owner: ObLog, click_count: 5, time dist:[FORMAT_END=969414, ObMalloc#start=1, ObMalloc#end=1002982, ALLOC_END=4, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1f9d7fcf 0x99b8be2 0x1f981833 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:01.752090] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] [COMPACTION DAG STATUS] (start_time=1755162410722075, end_time=1755162421746029, dag_status={}) [2025-08-14 09:07:02.733546] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=979968] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:02.733560] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:02.733568] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:02.733575] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:02.733582] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:02.733589] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:02.733596] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:02.733604] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:02.733609] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:02.733618] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:07:02.733628] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:07:02.731837] WDIAG [DETECT] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [561][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] cost too much time:ob_lcl_batch_sender_thread.cpp:virtual void oceanbase::share::detector::ObLCLBatchSenderThread::run1(), (this=|threshold=30.00ms|start at 09:07:01.790|233=940.92ms|total=940.94ms) [2025-08-14 09:07:02.716940] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:515) [510][T1_TimerWK1_SSTableDefragme][T1][Y0-0000000000000000-0-0] [lt=12640] successfully defragment data blocks(rewrite_cnt=0) [2025-08-14 09:07:02.743631] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:02.743651] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:02.729280] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=898524][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:01.829|207=899.89ms|210=0us|222=2us|total=917.29ms) [2025-08-14 09:07:02.746684] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=17377][errcode=-4389] timer task use too much time(end=1755162422746684, start=1755162421829390, delta=917294) [2025-08-14 09:07:02.280550] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:07:02.746784] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=465524][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:07:02.746809] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:07:02.746817] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:07:02.746826] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:07:02.746834] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:07:02.746846] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:07:02.746855] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:07:02.746868] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:07:02.746875] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:07:02.746887] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:07:02.746896] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:07:02.746905] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:07:02.746912] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:07:02.746921] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:07:02.746930] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:07:02.746942] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:07:02.747088] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:07:02.747098] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:07:02.753045] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:02.754163] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:02.767316] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=9306] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:07:02.730378] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] get_fetch_log_dst failed or invalid(ret=0, palf_id=1001, self="172.77.0.7:2882", fetch_log_dst="172.77.0.7:2882") [2025-08-14 09:07:02.764947] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:02.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:02.768722] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=3764][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:02.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:02.765027] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.750, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:02.768755] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=3719][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:02.768765] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d80b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:02.777491] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:02.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:02.807686] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:02.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:02.829516] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:02.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:02.829517] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:02.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:02.830884] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:07:02.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:02.838159] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 42.093538 loop:26707, slowcnt: 399 [2025-08-14 09:07:02.929544] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:02.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:02.929584] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:02.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:03.029607] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:02.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:03.129669] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:03.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:03.229729] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:03.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:03.247265] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:07:03.247286] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:07:03.247303] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:07:03.247311] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:07:03.247320] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:07:03.247329] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:07:03.247340] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:07:03.326672] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=266] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:07:03.728269] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=850] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:02.846758] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:02.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:02.768772] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.016645] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:03.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:03.737277] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=889166][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:02.846|207=890.51ms|210=0us|222=2us|total=890.56ms) [2025-08-14 09:07:03.737287] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=719549][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:03.166|207=720.64ms|210=0us|222=2us|total=720.67ms) [2025-08-14 09:07:03.737315] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=-4389] timer task use too much time(end=1755162423737314, start=1755162422846756, delta=890558) [2025-08-14 09:07:03.737321] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4389] timer task use too much time(end=1755162423737320, start=1755162423016643, delta=720677) [2025-08-14 09:07:03.029648] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:02.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:03.029002] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:03.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:03.737364] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=706635][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:03.296|207=707.71ms|210=0us|222=10us|total=707.73ms) [2025-08-14 09:07:03.737375] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] timer task use too much time(end=1755162423737375, start=1755162423029646, delta=707729) [2025-08-14 09:07:03.737379] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=707303][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:03.290|207=708.38ms|210=0us|222=2us|total=708.39ms) [2025-08-14 09:07:03.247349] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:07:03.737392] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task use too much time(end=1755162423737391, start=1755162423029000, delta=708391) [2025-08-14 09:07:03.737404] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=489306][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:07:03.737425] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:07:03.737440] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:07:03.737429] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=445079, expected_run_ts_us_=1755162423292349, ObClockGenerator::getRealClock()=1755162423737429, ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:03.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:03.737449] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:07:03.737459] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:07:02.782419] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:02.783345] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:03.329790] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:03.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:03.739119] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=408706][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:03.329|207=409.33ms|210=1us|222=1us|total=409.34ms) [2025-08-14 09:07:03.739137] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] timer task use too much time(end=1755162423739136, start=1755162423329788, delta=409348) [2025-08-14 09:07:02.737694] INFO [STORAGE] ~ObITabletFilterOp (ob_tablet_pointer.cpp:751) [510][T1_TimerWK1_SSTableDefragme][T1][Y0-0000000000000000-0-0] [lt=20710] tablet skip filter destructed(total_cnt=0, skip_cnt=0, not_in_mem_cnt=0, invalid_attr_cnt=0, total_tablet_cnt=136172, total_skip_cnt=367, not_in_mem_tablet_cnt=0, invalid_attr_tablet_cnt=0) [2025-08-14 09:07:03.739197] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [510][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=999987][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f994e1a4738, timer:0x7f9956fe7e10, task:0x7f9969bfd0a0, task_type:"N9oceanbase12blocksstable21ObSharedMacroBlockMgr26ObBlockDefragmentationTaskE", scheduled_time:1755162150767467, pushed_time:1755162150768766, delay:30000000}, delay_in_thread_pool=267970197, threashold=500000) [2025-08-14 09:07:03.738504] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=953715][errcode=0] epoll handler too much time :955167 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:02.769358] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1808) [368][VTblLocAsyncUp0][T0][YB42AC4D0007-00063C4968299C62-0-0] [lt=16247][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2025-08-14 09:07:03.738488] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=954623][errcode=0] epoll handler too much time :956079 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:01.767285] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:07:03.250321] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=339][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:03.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:03.737468] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:07:03.744641] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7155][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:07:03.744658] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:07:03.744673] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:07:03.744833] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:07:03.744846] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:07:03.749189] INFO [COMMON] handle (memory_dump.cpp:634) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=27] restore from sigsegv, let's goon~ [2025-08-14 09:07:02.732380] EDIAG alarm_if_necessary (ob_timer_service.cpp:143) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 1016767, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1016758, ObMalloc#start=1, ObMalloc#end=4, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x96ed636 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:03.737284] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=967042][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:02.768|207=968.51ms|210=0us|222=2us|total=968.70ms) [2025-08-14 09:07:03.749873] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12568][errcode=-4389] timer task use too much time(end=1755162423749872, start=1755162422768771, delta=981101) [2025-08-14 09:07:03.749875] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=188] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:03.749893] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] task scheduled too delay(scheduled_delay=991795, expected_run_ts_us_=1755162422758096, ObClockGenerator::getRealClock()=1755162423749892, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891992330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196d710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.749920] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891992330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196d710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.749939] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] task scheduled too delay(scheduled_delay=997274, expected_run_ts_us_=1755162422752664, ObClockGenerator::getRealClock()=1755162423749939, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.749953] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.749966] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=988876, expected_run_ts_us_=1755162422761089, ObClockGenerator::getRealClock()=1755162423749966, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.749978] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.749991] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=989970, expected_run_ts_us_=1755162422760021, ObClockGenerator::getRealClock()=1755162423749991, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.750003] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.750012] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=245] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:03.750015] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=978147, expected_run_ts_us_=1755162422771868, ObClockGenerator::getRealClock()=1755162423750015, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.750028] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.750040] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=974004, expected_run_ts_us_=1755162422776036, ObClockGenerator::getRealClock()=1755162423750040, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72751b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.750054] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72751b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.750075] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] task scheduled too delay(scheduled_delay=973938, expected_run_ts_us_=1755162422776136, ObClockGenerator::getRealClock()=1755162423750075, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d31b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.750086] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d31b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.750098] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=976148, expected_run_ts_us_=1755162422773949, ObClockGenerator::getRealClock()=1755162423750098, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73959b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.773, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.750110] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73959b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.773, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.750124] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=962790, expected_run_ts_us_=1755162422787334, ObClockGenerator::getRealClock()=1755162423750124, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.750136] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.750148] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=962644, expected_run_ts_us_=1755162422787504, ObClockGenerator::getRealClock()=1755162423750148, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72856b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.750160] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72856b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.750172] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=962196, expected_run_ts_us_=1755162422787976, ObClockGenerator::getRealClock()=1755162423750172, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.750186] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.750197] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=960036, expected_run_ts_us_=1755162422790161, ObClockGenerator::getRealClock()=1755162423750197, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919920b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196d590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.790, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.750249] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919920b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196d590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.790, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.750265] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=910181, expected_run_ts_us_=1755162422840084, ObClockGenerator::getRealClock()=1755162423750265, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.840, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.750277] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.840, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.750292] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] task scheduled too delay(scheduled_delay=904168, expected_run_ts_us_=1755162422846124, ObClockGenerator::getRealClock()=1755162423750292, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d54b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ecd90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.846, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.750304] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d54b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ecd90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.846, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.750316] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=902262, expected_run_ts_us_=1755162422848053, ObClockGenerator::getRealClock()=1755162423750316, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ecc10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.848, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.750327] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ecc10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:02.848, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.750351] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.722, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.750376] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.750388] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891826cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891918190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:02.732380] WDIAG alarm_if_necessary (ob_timer_service.cpp:143) [807][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task's elapsed_time is greater than period(token={this:0x7f991b7a5458, timer:0x7f9922ffd748, task:0x7f9922ffd770, task_type:"N9oceanbase7storage10checkpoint16ObEmptyShellTaskE", scheduled_time:1755162393737357, pushed_time:1755162393756601, delay:5000000}, elapsed_time=6993439, period=5000000) [2025-08-14 09:07:03.737448] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:03.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:03.737333] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=728197, expected_run_ts_us_=1755162423009134, ObClockGenerator::getRealClock()=1755162423737332, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:03.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:03.751607] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14250][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:03.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:03.751638] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] task scheduled too delay(scheduled_delay=662321, expected_run_ts_us_=1755162423089316, ObClockGenerator::getRealClock()=1755162423751638, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:03.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:03.751653] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:03.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:03.751673] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] task scheduled too delay(scheduled_delay=547375, expected_run_ts_us_=1755162423204297, ObClockGenerator::getRealClock()=1755162423751672, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:03.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:03.751686] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:03.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:03.752429] INFO [COMMON] handle (memory_dump.cpp:634) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=21] restore from sigsegv, let's goon~ [2025-08-14 09:07:03.744310] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=493227][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:03.250|207=493.98ms|210=0us|222=2us|total=505.76ms) [2025-08-14 09:07:03.756084] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=11755][errcode=-4389] timer task use too much time(end=1755162423756083, start=1755162423250319, delta=505764) [2025-08-14 09:07:03.756102] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=272569, expected_run_ts_us_=1755162423483531, ObClockGenerator::getRealClock()=1755162423756101, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:03.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:03.756129] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:03.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:03.756148] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:03.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:03.756181] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=242] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:02.730064] EDIAG [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=948055][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1041012, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=46, FORMAT_END=1040959, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9826e08 0x98267df 0x982650b 0x98262e0 0x1d188ec1 0x1d17b8ec 0x1d17ad88 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:02.743698] EDIAG [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=961616][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1028439, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1028143, ObMalloc#start=1, ObMalloc#end=36, ObMalloc#start=27, ObMalloc#end=33, FREE_END=199], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9826e08 0x98267df 0x982650b 0x98262e0 0x1d188ec1 0x1d17b8ec 0x1d17ad88 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:02.751435] EDIAG [STORAGE.TRANS] refresh_elr_tenant_config_ (ob_tx_elr_util.cpp:42) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=2583][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1975241, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1975078, ObMalloc#start=0, ObMalloc#end=38, ObMalloc#start=23, ObMalloc#end=30, FREE_END=72], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1b878a47 0x1b8787a8 0x1b878197 0x986bac9 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [396]OB_MALLOC COST TOO MUCH TIME, cost_time=1009102, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1009094], seq:[0] [2025-08-14 09:07:03.760501] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=220] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:03.760532] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=154] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:03.760553] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=161] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:03.760573] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:03.760595] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:03.760635] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=149] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:03.764310] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:03.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:03.764348] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:03.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:03.764499] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.764517] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919845b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.764527] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.764599] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=71][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.764609] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.764637] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.764647] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eacb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:03.767645] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=203] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:02.730064] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=948055] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-08-14 09:07:03.790304] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=185] PNIO [ratelimit] time: 1755162423790302, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007619, add_bytes: 0 [2025-08-14 09:07:03.801906] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=41009] gc stale ls task succ [2025-08-14 09:07:04.370257] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=215] gc stale ls task succ [2025-08-14 09:07:03.781977] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3388] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:04.728641] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=945232][errcode=0] epoll handler too much time :946677 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:03.774579] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.767, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:04.729608] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=953579][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:03.774|207=955.02ms|210=1us|222=1us|total=955.05ms) [2025-08-14 09:07:04.729635] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4389] timer task use too much time(end=1755162424729634, start=1755162423774578, delta=955056) [2025-08-14 09:07:03.781653] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4082] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:03.837394] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:03.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:04.015954] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:04.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:04.327951] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=20] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:07:03.784221] INFO handle (memory_dump.cpp:671) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=18] statistics: tenant_cnt: 6, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 168 270 0 1 2 110 1 0 1 3 7 0 0 1 4 1 1 0 1 5 85 6 0 1 6 13 0 0 1 7 22 2 0 1 8 53 1 0 1 12 17 3 0 1 13 10 1 0 1 14 1 1 0 1 15 1 1 0 1 16 8 7 0 1 18 4 3 0 1 20 2 1 0 1 22 1 1 0 1 24 1 0 0 1 25 6 0 0 500 0 561 274 0 500 7 192 5 2 500 8 41 1 0 500 9 2 1 0 500 10 7 2 0 500 16 57 4 0 500 17 60 7 0 500 22 38 64 0 500 23 27 10 0 508 0 2 7 0 508 8 5 1 0 509 0 2 7 0 509 8 5 1 0 1001 0 147 252 0 1001 2 26 1 0 1001 3 5 0 0 1001 4 1 1 0 1001 5 85 6 0 1001 6 11 0 0 1001 7 30 2 0 1001 8 36 1 0 1001 12 17 3 0 1001 13 3 1 0 1001 14 1 1 0 1001 16 7 4 0 1001 18 4 3 0 1001 20 4 1 0 1001 25 8 0 0 1002 0 263 278 0 1002 2 198 1 0 1002 3 4 0 0 1002 4 1 1 0 1002 5 101 6 0 1002 6 14 0 0 1002 7 6 2 0 1002 8 73 1 0 1002 12 18 3 0 1002 13 7 1 0 1002 14 1 2 0 1002 16 10 7 0 1002 18 9 3 0 1002 20 2 1 0 1002 21 1 0 0 1002 24 1 0 0 1002 25 7 0 0 cost_time: 4017935 [2025-08-14 09:07:04.751308] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=15] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:04.751335] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:04.728259] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:04.751465] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23170][errcode=0] epoll handler too much time :23216 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:04.729648] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=950247, expected_run_ts_us_=1755162423779400, ObClockGenerator::getRealClock()=1755162424729647, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:04.752124] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22440][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:04.752144] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] task scheduled too delay(scheduled_delay=973811, expected_run_ts_us_=1755162423778332, ObClockGenerator::getRealClock()=1755162424752144, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:04.730222] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=947135][errcode=0] epoll handler too much time :948578 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:04.730249] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=891503][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:03.837|207=892.85ms|210=0us|222=1us|total=892.87ms) [2025-08-14 09:07:04.730266] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=713230][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:04.159|207=714.31ms|210=0us|222=1us|total=714.32ms) [2025-08-14 09:07:04.758341] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:04.758396] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:04.758421] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:04.758453] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:04.752155] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:04.766656] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14475][errcode=0] task scheduled too delay(scheduled_delay=988074, expected_run_ts_us_=1755162423778581, ObClockGenerator::getRealClock()=1755162424766656, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:04.766677] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:04.766689] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=996534, expected_run_ts_us_=1755162423770154, ObClockGenerator::getRealClock()=1755162424766689, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:04.766698] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:04.766919] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:04.771030] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=314][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:04.771288] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=290][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:04.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:04.773331] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=5624] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:07:04.779691] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=263] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:07:04.781662] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=242] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:07:04.797917] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=12] PNIO [ratelimit] time: 1755162424797915, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007613, add_bytes: 0 [2025-08-14 09:07:04.801498] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:04.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:04.801986] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=13] gc stale ls task succ [2025-08-14 09:07:04.831822] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=362][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd070, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fd350, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:5.00s, expected_run_time:09:07:04.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:04.831861] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:07:04.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:04.837952] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=283][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:04.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:04.839538] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=347][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 44.094917 loop:26707, slowcnt: 419 [2025-08-14 09:07:04.839763] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=274][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:04.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:04.848983] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fcc90, caller:mds_tenant_service.cpp:mtl_start:117, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_145, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcf70, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:04.849019] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fc8b0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcb90, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:1, time_interval:3.00s, expected_run_time:09:07:04.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:04.871089] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:04.871688] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=1890767] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:07:04.880849] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=1850998] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:07:04.896171] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=1870247] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:07:04.938008] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:04.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:04.939787] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:04.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:04.971161] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:04.996726] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1374][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:04.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.022884] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:05.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.038040] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:04.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.039850] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:04.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.071222] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:05.138103] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:05.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.139913] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:05.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.171281] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:05.238172] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:05.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.239955] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:05.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.248248] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:05.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.258283] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:05.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.263774] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=372][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:07:05.263808] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:07:05.263844] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:07:05.263861] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:07:05.263878] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:07:05.263896] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:07:05.263913] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:07:05.263931] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:07:05.263953] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:07:05.263978] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:07:05.263999] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:07:05.264017] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:07:05.264042] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:07:05.264058] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:07:05.264075] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:07:05.264090] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:07:05.264111] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:07:05.264297] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:07:05.264317] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:07:05.271311] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:05.274342] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:05.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.304537] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:05.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.330707] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=401723] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:07:05.338246] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:05.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.339975] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:05.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.370350] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=13] gc stale ls task succ [2025-08-14 09:07:05.371385] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:05.438290] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:05.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.440043] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:05.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.471453] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:05.499749] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:05.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.525830] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:05.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.538312] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:05.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.540087] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:05.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.571511] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:05.638424] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:05.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.640150] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:05.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.671574] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:04.752186] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=21887][errcode=-4389] timer task use too much time(end=1755162424752186, start=1755162424015952, delta=736234) [2025-08-14 09:07:05.721580] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=285] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [864]OB_MALLOC COST TOO MUCH TIME, cost_time=1946209, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1946208], seq:[0] [864]OB_MALLOC COST TOO MUCH TIME, cost_time=1951304, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=65, WASH_BLOCK_END=4999, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1946229], seq:[0, 1, 2, 3, 4] [2025-08-14 09:07:03.767936] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=276][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1951334, owner: ObLog, click_count: 5, time dist:[FORMAT_END=14, ObMalloc#start=1, ObMalloc#end=1951313, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [396]OB_MALLOC COST TOO MUCH TIME, cost_time=1964466, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=44, WASH_BLOCK_END=60, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1009124], seq:[0, 1, 2, 3, 4] [2025-08-14 09:07:03.744469] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1975896, owner: ObLog, click_count: 11, time dist:[ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=83, ALLOC_BLOCK#start=11237, ALLOC_BLOCK#end=42, FORMAT_END=40, ObMalloc#start=1, ObMalloc#end=1964484, ALLOC_END=2, APPEND_END=3, ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=4], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:03.767936] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=276] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:07:03.744469] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:05.723418] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1975896][errcode=0] epoll handler too much time :1978958 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:05.723440] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:04.766707] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=983056, expected_run_ts_us_=1755162423783651, ObClockGenerator::getRealClock()=1755162424766707, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72716b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a4910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723467] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=955311][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72716b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a4910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723496] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:04.766|207=956.79ms|210=0us|222=2us|total=956.80ms) [2025-08-14 09:07:05.723512] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] timer task use too much time(end=1755162425723512, start=1755162424766707, delta=956805) [2025-08-14 09:07:05.723525] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1935497, expected_run_ts_us_=1755162423788027, ObClockGenerator::getRealClock()=1755162425723525, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723541] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723554] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1935287, expected_run_ts_us_=1755162423788267, ObClockGenerator::getRealClock()=1755162425723554, ret=0, ret="OB_SUCCESS", *this={this:0x7f98918100b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723567] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918100b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723579] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=1942367, expected_run_ts_us_=1755162423781212, ObClockGenerator::getRealClock()=1755162425723579, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723591] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723615] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] task scheduled too delay(scheduled_delay=1929820, expected_run_ts_us_=1755162423793794, ObClockGenerator::getRealClock()=1755162425723615, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.793, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723628] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.793, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723641] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1931241, expected_run_ts_us_=1755162423792400, ObClockGenerator::getRealClock()=1755162425723641, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180fe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723653] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180fe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723674] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] task scheduled too delay(scheduled_delay=1917926, expected_run_ts_us_=1755162423805747, ObClockGenerator::getRealClock()=1755162425723674, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.805, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723689] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.805, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723702] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1911146, expected_run_ts_us_=1755162423812555, ObClockGenerator::getRealClock()=1755162425723701, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73927b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.812, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723714] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73927b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.812, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723728] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1908261, expected_run_ts_us_=1755162423815466, ObClockGenerator::getRealClock()=1755162425723728, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7399090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.815, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723740] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7399090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.815, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723755] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=1907456, expected_run_ts_us_=1755162423816298, ObClockGenerator::getRealClock()=1755162425723755, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723774] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723787] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1896458, expected_run_ts_us_=1755162423827329, ObClockGenerator::getRealClock()=1755162425723787, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73954b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.827, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723799] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73954b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.827, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723815] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=1850130, expected_run_ts_us_=1755162423873684, ObClockGenerator::getRealClock()=1755162425723815, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.873, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723827] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:03.873, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723854] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] task scheduled too delay(scheduled_delay=1475681, expected_run_ts_us_=1755162424248172, ObClockGenerator::getRealClock()=1755162425723854, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.248, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723867] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.248, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723889] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=988360, expected_run_ts_us_=1755162424735529, ObClockGenerator::getRealClock()=1755162425723889, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7270030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a3b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.735, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723903] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7270030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a3b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.735, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723917] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=984010, expected_run_ts_us_=1755162424739906, ObClockGenerator::getRealClock()=1755162425723916, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ecb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.739, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723929] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ecb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.739, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723942] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=991183, expected_run_ts_us_=1755162424732759, ObClockGenerator::getRealClock()=1755162425723942, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7353730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7396990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723954] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7353730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7396990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723969] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=987107, expected_run_ts_us_=1755162424736861, ObClockGenerator::getRealClock()=1755162425723969, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723981] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.723995] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=983838, expected_run_ts_us_=1755162424740156, ObClockGenerator::getRealClock()=1755162425723994, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ea30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724007] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ea30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724020] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=969721, expected_run_ts_us_=1755162424754299, ObClockGenerator::getRealClock()=1755162425724020, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919afbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724034] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919afbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724048] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=965516, expected_run_ts_us_=1755162424758532, ObClockGenerator::getRealClock()=1755162425724048, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919854b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724060] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919854b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724072] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=965339, expected_run_ts_us_=1755162424758733, ObClockGenerator::getRealClock()=1755162425724072, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724084] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724097] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=965127, expected_run_ts_us_=1755162424758970, ObClockGenerator::getRealClock()=1755162425724097, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724108] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724128] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] task scheduled too delay(scheduled_delay=971219, expected_run_ts_us_=1755162424752908, ObClockGenerator::getRealClock()=1755162425724128, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d45b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724140] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d45b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:04.781736] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:05.724152] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=957560, expected_run_ts_us_=1755162424766592, ObClockGenerator::getRealClock()=1755162425724152, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72747b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724160] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=940999][errcode=0] epoll handler too much time :942432 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:05.724165] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72747b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724178] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=957769, expected_run_ts_us_=1755162424766408, ObClockGenerator::getRealClock()=1755162425724178, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724190] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724202] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=948581, expected_run_ts_us_=1755162424775620, ObClockGenerator::getRealClock()=1755162425724202, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724220] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724254] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] task scheduled too delay(scheduled_delay=948599, expected_run_ts_us_=1755162424775655, ObClockGenerator::getRealClock()=1755162425724254, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724270] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724283] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=952381, expected_run_ts_us_=1755162424771901, ObClockGenerator::getRealClock()=1755162425724282, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724295] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724314] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] task scheduled too delay(scheduled_delay=951343, expected_run_ts_us_=1755162424772971, ObClockGenerator::getRealClock()=1755162425724314, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724328] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724341] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=938481, expected_run_ts_us_=1755162424785859, ObClockGenerator::getRealClock()=1755162425724341, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724352] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724379] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] task scheduled too delay(scheduled_delay=939218, expected_run_ts_us_=1755162424785160, ObClockGenerator::getRealClock()=1755162425724379, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724393] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724405] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=920325, expected_run_ts_us_=1755162424804079, ObClockGenerator::getRealClock()=1755162425724405, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919836b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724417] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919836b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724430] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=894030, expected_run_ts_us_=1755162424830400, ObClockGenerator::getRealClock()=1755162425724430, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724450] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724466] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] task scheduled too delay(scheduled_delay=893441, expected_run_ts_us_=1755162424831025, ObClockGenerator::getRealClock()=1755162425724466, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.831, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724479] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.831, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724492] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=862358, expected_run_ts_us_=1755162424862133, ObClockGenerator::getRealClock()=1755162425724491, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.862, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.724503] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:04.862, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:04.782247] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21911] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:05.724843] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=941161][errcode=0] epoll handler too much time :942606 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:03.753122] EDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=1970356][errcode=-4038] LOGGER COST TOO MUCH TIME, cost: 1971499, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1971494, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1b4fd0db 0x1b4fc327 0x1b5014e0 0x1b5009d1 0x1b50387c 0x1b8905de 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:03.753122] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=1970356][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162415740247], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:07:05.728338] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:05.714841] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=961196][errcode=0] task scheduled too delay(scheduled_delay=1705704, expected_run_ts_us_=1755162424009134, ObClockGenerator::getRealClock()=1755162425714839, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:04.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.729263] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14399][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:04.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.729304] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] task scheduled too delay(scheduled_delay=1639987, expected_run_ts_us_=1755162424089316, ObClockGenerator::getRealClock()=1755162425729304, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:04.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.729324] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:04.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.729338] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1525040, expected_run_ts_us_=1755162424204297, ObClockGenerator::getRealClock()=1755162425729337, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:04.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.729348] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:04.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.729383] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=0] task scheduled too delay(scheduled_delay=1002789, expected_run_ts_us_=1755162424726594, ObClockGenerator::getRealClock()=1755162425729383, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd110, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd3f0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:10.00s, expected_run_time:09:07:04.726, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.729394] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd110, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd3f0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:10.00s, expected_run_time:09:07:04.726, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.729404] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=973917, expected_run_ts_us_=1755162424755487, ObClockGenerator::getRealClock()=1755162425729404, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:04.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.729414] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:04.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:04.752175] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=21892][errcode=-4389] timer task use too much time(end=1755162424752175, start=1755162423837392, delta=914783) [2025-08-14 09:07:05.729496] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=975840][errcode=0] task scheduled too delay(scheduled_delay=935561, expected_run_ts_us_=1755162424793934, ObClockGenerator::getRealClock()=1755162425729496, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:04.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.729538] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:04.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:03.772738] EDIAG [RPC] handle_resp (ob_poc_rpc_proxy.cpp:192) [319][pnio1][T0][Y0-0000000000000000-0-0] [lt=268572134][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1960299, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1960291, ObMalloc#start=0, ObMalloc#end=4, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x27a861af 0x97bcba5 0x97bc4ca 0x27ad7f11 0x97b81a7 0x27a8a6d6 0x97b7ba0 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:05.726741] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=1972401] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:07:05.735255] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8440] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:07:05.735310] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:07:05.735363] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:07:05.735403] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS CHANNEL] device_id=0x7f99667cc0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:07:05.735422] INFO [COMMON] print_status (ob_io_struct.cpp:2837) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CHANNEL] device_id=0x7f99667cb2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-08-14 09:07:03.772738] WDIAG [RPC] handle_resp (ob_poc_rpc_proxy.cpp:192) [319][pnio1][T0][Y0-0000000000000000-0-0] [lt=268572134][errcode=0] async_cb handler cost too much time(total_time=273992285, decode_time=-1755162149746646, process_time=0, session_destroy_time=1755162423738931, ret=0, pcode=1815) [2025-08-14 09:07:05.736044] WDIAG stat_cleanup (debug.c:35) [319][pnio1][T0][Y0-0000000000000000-0-0] [lt=1960299][errcode=0] PNIO [delay_warn] cost too much time: 275988934us, procedure: pn_pktc_resp_cb [2025-08-14 09:07:05.738690] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:05.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.740284] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:05.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.727645] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1971499][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1002, srr:[mts=1755162415740247], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:07:05.741738] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14062] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162415740247]}) [2025-08-14 09:07:05.741765] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21] gts statistics(tenant_id=1002, gts_rpc_cnt=0, get_gts_cache_cnt=0, get_gts_with_stc_cnt=1, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-08-14 09:07:05.741788] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=22] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1755161762796561], gts:1755161762721798002, latest_srr:[mts=1755162415740247]}) [2025-08-14 09:07:05.743520] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:05.746506] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:05.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.746668] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.746698] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918114b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.746710] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728aa90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.746728] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.746741] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.746754] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.749044] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=159] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:05.751288] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=11] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:05.751306] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:05.752378] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:05.743263] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=174] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:05.753328] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10048][errcode=0] epoll handler too much time :10076 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:05.753412] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=944750] [MEMORY] tenant: 1, limit: 2147483648 hold: 1013174272 cache_hold: 56623104 cache_used: 56623104 cache_item_count: 27 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 444993536 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 230686720 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 1163264 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 483328 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 51617792 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 1458176 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 14204928 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 111149056 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35651584 limit= 429496720 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 8003584 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= PS_CACHE_CTX_ID hold_bytes= 65536 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 14827520 limit= 9223372036854775807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 3481600 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3964928 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 212992 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 884736 limit= 9223372036854775807 [2025-08-14 09:07:05.754123] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:05.754916] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:05.755771] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=233] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:05.756565] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:05.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.757255] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=305] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [374]OB_MALLOC COST TOO MUCH TIME, cost_time=970535, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=970525], seq:[0] [2025-08-14 09:07:05.758458] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:05.761335] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=192] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:05.761359] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:05.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.761492] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=118][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:05.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.765843] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:05.779819] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=25] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:07:05.781763] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=21] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:07:05.781936] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:05.782254] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:05.802018] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:07:05.829560] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:05.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:05.929614] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:05.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.012769] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:06.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.012796] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:06.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.029675] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:05.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.093182] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:06.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.129691] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:06.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.213894] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:06.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.229753] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:06.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.264130] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:06.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.264154] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:06.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.329788] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:06.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.370414] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=26] gc stale ls task succ [2025-08-14 09:07:06.429850] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:06.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.515532] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:06.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.515568] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:06.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.529910] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:06.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.596012] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:06.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.629946] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:06.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.716734] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:06.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.727037] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=16] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 444993536 used= 389812256 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 70 washed_blocks= 287 washed_size= 70221824 [MEMORY] request_cached_chunk_cnt= 6 [MEMORY] hold= 389812256 used= 379555363 count= 32901 avg_used= 11536 mod=SUMMARY [2025-08-14 09:07:06.729483] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=2420] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 230686720 used= 228884480 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 block_cnt= 110 chunk_cnt= 110 mod=Memstore [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:07:06.729512] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=22] [MEMORY] tenant_id= 1 ctx_id= EXECUTE_CTX_ID hold= 1163264 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 15 washed_size= 13516800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:07:06.729528] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:07:06.729544] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 51617792 used= 47452416 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 84 washed_blocks= 334 washed_size= 126640128 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 47452416 used= 43070492 count= 3992 avg_used= 10789 mod=SUMMARY [2025-08-14 09:07:06.729558] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= WORK_AREA hold= 1458176 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 13 washed_blocks= 66 washed_size= 25804800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:07:06.729573] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 14204928 used= 10970224 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 277 washed_size= 31932416 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10970224 used= 6078924 count= 18573 avg_used= 327 mod=SUMMARY [2025-08-14 09:07:06.729588] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 111149056 used= 108896256 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 block_cnt= 211 chunk_cnt= 53 mod=CoStack [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 mod=SUMMARY [2025-08-14 09:07:06.729609] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 56623104 used= 56623104 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 56623104 used= 0 count= 27 avg_used= 0 mod=SUMMARY [2025-08-14 09:07:06.729625] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 34365376 limit= 429496720 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 34365376 used= 31086608 count= 475 avg_used= 65445 mod=SUMMARY [2025-08-14 09:07:06.729640] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 8003584 used= 4947392 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 10 washed_blocks= 23 washed_size= 12967936 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4947392 used= 4793344 count= 604 avg_used= 7936 mod=SUMMARY [2025-08-14 09:07:06.729654] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:07:06.729672] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 65536 used= 31024 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 2031616 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 31024 used= 27058 count= 20 avg_used= 1352 mod=SUMMARY [2025-08-14 09:07:06.729687] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= RPC_CTX_ID hold= 14827520 used= 6135808 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 6135808 used= 4051328 count= 253 avg_used= 16013 mod=SUMMARY [2025-08-14 09:07:06.729704] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 3481600 used= 2391232 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 10 washed_size= 4907008 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2391232 used= 2367040 count= 36 avg_used= 65751 mod=SUMMARY [2025-08-14 09:07:06.729728] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=15] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 3964928 used= 3080192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 3 washed_size= 229376 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3080192 used= 2983936 count= 376 avg_used= 7936 mod=SUMMARY [2025-08-14 09:07:06.729743] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= SCHEMA_SERVICE hold= 212992 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1884160 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:07:06.729759] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:07:06.729772] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= MERGE_NORMAL_CTX_ID hold= 884736 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 6 washed_blocks= 9 washed_size= 11698176 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:07:06.730180] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:06.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.739369] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=296][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:06.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.729240] WDIAG pktc_do_cb_exception (pktc_resp.h:23) [319][pnio1][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] PNIO pktc_req hasn't be flushed before callback, pkt_id=559725, sock=0x7f995f5e54a8, code=-47 [2025-08-14 09:07:06.740853] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=256][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:06.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.740889] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11981][errcode=0] epoll handler too much time :12011 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:06.742834] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=258][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:06.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.745169] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=30] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:06.745255] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [381]OB_MALLOC COST TOO MUCH TIME, cost_time=991189, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=991187], seq:[0] [374]OB_MALLOC COST TOO MUCH TIME, cost_time=991561, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=970556], seq:[0, 1] [2025-08-14 09:07:06.749343] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][YB42AC4D0007-00063C495F8A1985-0-0] [lt=1016767][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a7ad8, timer:0x7f9935f59f10, task:0x7f9935f6c710, task_type:"N9oceanbase5table22OBTTLTimerPeriodicTaskE", scheduled_time:1755162397769934, pushed_time:1755162397771917, delay:10000000}, delay_in_thread_pool=25978805, threashold=500000) [2025-08-14 09:07:06.749406] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][YB42AC4D0007-00063C495F8A1986-0-0] [lt=31][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a79f8, timer:0x7f99535c95e0, task:0x7f9935f6c340, task_type:"N9oceanbase5table22OBTTLTimerPeriodicTaskE", scheduled_time:1755162397769947, pushed_time:1755162398737260, delay:10000000}, delay_in_thread_pool=28012132, threashold=500000) [2025-08-14 09:07:06.749904] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:06.750006] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 500, limit: 9223372036854775807 hold: 691027968 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 333369344 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 35291136 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 85983232 limit= 9223372036854775807 [MEMORY] ctx_id= LIBEASY hold_bytes= 3145728 limit= 9223372036854775807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 14680064 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 3063808 limit= 9223372036854775807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 37584896 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 78925824 limit= 9223372036854775807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 98983936 limit= 9223372036854775807 [2025-08-14 09:07:06.750043] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 333369344 used= 301696048 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 499 washed_blocks= 2229 washed_size= 922714112 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 301696048 used= 294263701 count= 12718 avg_used= 23137 mod=SUMMARY [2025-08-14 09:07:06.750069] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 500 ctx_id= GLIBC hold= 35291136 used= 21987056 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 186 washed_blocks= 811 washed_size= 367362048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 21987056 used= 10377061 count= 131699 avg_used= 78 mod=SUMMARY [2025-08-14 09:07:06.750086] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 500 ctx_id= CO_STACK hold= 85983232 used= 83091456 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 83091456 used= 82936896 count= 161 avg_used= 515136 block_cnt= 161 chunk_cnt= 41 mod=CoStack [MEMORY] hold= 83091456 used= 82936896 count= 161 avg_used= 515136 mod=SUMMARY [2025-08-14 09:07:06.750101] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= LIBEASY hold= 3145728 used= 3070816 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1048576 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3070816 used= 2859784 count= 223 avg_used= 12824 mod=SUMMARY [2025-08-14 09:07:06.750114] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= LOGGER_CTX_ID hold= 16777216 used= 14565376 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 14565376 used= 14558208 count= 7 avg_used= 2079744 mod=SUMMARY [2025-08-14 09:07:06.750137] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= RPC_CTX_ID hold= 3063808 used= 1482752 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 57 washed_blocks= 70 washed_size= 116473856 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1482752 used= 979328 count= 61 avg_used= 16054 mod=SUMMARY [2025-08-14 09:07:06.750151] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= PKT_NIO hold= 37584896 used= 36013568 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 50 washed_blocks= 71 washed_size= 99016704 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 36013568 used= 35017984 count= 125 avg_used= 280143 mod=SUMMARY [2025-08-14 09:07:06.750175] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 500 ctx_id= SCHEMA_SERVICE hold= 78925824 used= 68937344 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 22 washed_size= 5767168 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 68937344 used= 67674815 count= 4860 avg_used= 13924 mod=SUMMARY [2025-08-14 09:07:06.750190] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= UNEXPECTED_IN_500 hold= 98983936 used= 97174864 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 97174864 used= 86783232 count= 3014 avg_used= 28793 mod=SUMMARY [2025-08-14 09:07:06.750205] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant: 508, limit: 1073741824 hold: 15613952 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 5128192 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10485760 limit= 9223372036854775807 [2025-08-14 09:07:06.750216] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 508 ctx_id= DEFAULT_CTX_ID hold= 5128192 used= 3765328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3765328 used= 3723912 count= 84 avg_used= 44332 mod=SUMMARY [2025-08-14 09:07:05.757282] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.750400] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=991613][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:05.757|207=993.12ms|210=1us|222=1us|total=993.14ms) [2025-08-14 09:07:06.750425] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4389] timer task use too much time(end=1755162426750424, start=1755162425757280, delta=993144) [2025-08-14 09:07:06.750438] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1006885, expected_run_ts_us_=1755162425743552, ObClockGenerator::getRealClock()=1755162426750437, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.750457] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.750483] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 508 ctx_id= CO_STACK hold= 10485760 used= 10321920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10321920 used= 10302720 count= 20 avg_used= 515136 mod=SUMMARY [2025-08-14 09:07:06.750531] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant: 509, limit: 1073741824 hold: 15613952 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 5128192 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10485760 limit= 9223372036854775807 [2025-08-14 09:07:06.750543] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 509 ctx_id= DEFAULT_CTX_ID hold= 5128192 used= 3765328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3765328 used= 3723912 count= 84 avg_used= 44332 mod=SUMMARY [2025-08-14 09:07:06.750568] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 509 ctx_id= CO_STACK hold= 10485760 used= 10321920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10321920 used= 10302720 count= 20 avg_used= 515136 mod=SUMMARY [2025-08-14 09:07:06.750618] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant: 1001, limit: 1073741824 hold: 608747520 cache_hold: 44040192 cache_used: 44040192 cache_item_count: 21 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 302309376 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 54525952 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 3080192 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 483328 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 45514752 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 5521408 limit= 157286400 [MEMORY] ctx_id= GLIBC hold_bytes= 15499264 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 73424896 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35651584 limit= 214748360 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 3252224 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 14680064 limit= 1073741800 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 5570560 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3866624 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 1179648 limit= 9223372036854775807 [2025-08-14 09:07:06.750644] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1001 ctx_id= DEFAULT_CTX_ID hold= 302309376 used= 268508240 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 81 washed_blocks= 406 washed_size= 113410048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 268508240 used= 262389910 count= 3782 avg_used= 69378 mod=SUMMARY [2025-08-14 09:07:06.750656] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= MEMSTORE_CTX_ID hold= 54525952 used= 54099968 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 54099968 used= 54075216 count= 26 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:07:06.750666] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= EXECUTE_CTX_ID hold= 3080192 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 8 washed_size= 7405568 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:07:06.750677] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:07:06.750688] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= PLAN_CACHE_CTX_ID hold= 45514752 used= 36552320 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 77 washed_blocks= 201 washed_size= 132743168 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 36552320 used= 33009690 count= 3323 avg_used= 9933 mod=SUMMARY [2025-08-14 09:07:06.750698] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= WORK_AREA hold= 5521408 used= 0 limit= 157286400 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 9 washed_blocks= 40 washed_size= 17547264 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:07:06.750709] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= GLIBC hold= 15499264 used= 10889728 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 25 washed_blocks= 164 washed_size= 47415296 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10889728 used= 6020141 count= 18313 avg_used= 328 mod=SUMMARY [2025-08-14 09:07:06.750721] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= CO_STACK hold= 73424896 used= 71221248 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 32 washed_blocks= 34 washed_size= 2072576 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 71221248 used= 71088768 count= 138 avg_used= 515136 block_cnt= 138 chunk_cnt= 36 mod=CoStack [MEMORY] hold= 71221248 used= 71088768 count= 138 avg_used= 515136 mod=SUMMARY [2025-08-14 09:07:06.750736] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= KVSTORE_CACHE_ID hold= 44040192 used= 44040192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 44040192 used= 0 count= 21 avg_used= 0 mod=SUMMARY [2025-08-14 09:07:06.750747] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 33906624 limit= 214748360 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33906624 used= 30629648 count= 468 avg_used= 65447 mod=SUMMARY [2025-08-14 09:07:06.750772] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=21] [MEMORY] tenant_id= 1001 ctx_id= TX_CALLBACK_CTX_ID hold= 3252224 used= 2342592 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 3 washed_blocks= 8 washed_size= 3039232 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2342592 used= 2269696 count= 286 avg_used= 7936 mod=SUMMARY [2025-08-14 09:07:06.750783] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:07:06.750796] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= RPC_CTX_ID hold= 14680064 used= 4202496 limit= 1073741800 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4202496 used= 2779776 count= 171 avg_used= 16256 mod=SUMMARY [2025-08-14 09:07:06.750818] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= TX_DATA_TABLE hold= 5570560 used= 2342784 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 8 washed_size= 2818048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2342784 used= 2319424 count= 30 avg_used= 77314 mod=SUMMARY [2025-08-14 09:07:06.750831] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= MDS_DATA_ID hold= 3866624 used= 2785280 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 5 washed_size= 4521984 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2785280 used= 2698240 count= 340 avg_used= 7936 mod=SUMMARY [2025-08-14 09:07:06.750848] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= MERGE_NORMAL_CTX_ID hold= 1179648 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 8 washed_blocks= 24 washed_size= 15597568 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:07:06.750874] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant: 1002, limit: 8589934592 hold: 1945841664 cache_hold: 48234496 cache_used: 48234496 cache_item_count: 23 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 1009610752 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 415236096 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 1114112 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 950272 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 211812352 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 1441792 limit= 429496725 [MEMORY] ctx_id= GLIBC hold_bytes= 12582912 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 149913600 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 37748736 limit= 1717986900 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 7766016 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 5783552 limit= 8589934500 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 5718016 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3047424 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 1032192 limit= 9223372036854775807 [2025-08-14 09:07:06.750905] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= DEFAULT_CTX_ID hold= 1009610752 used= 919207280 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 13 [MEMORY] hold= 185860096 used= 185534336 count= 114 avg_used= 1627494 block_cnt= 114 chunk_cnt= 104 mod=MysqlRequesReco [MEMORY] hold= 167792640 used= 167772192 count= 1 avg_used= 167772192 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 83927040 used= 83886080 count= 2 avg_used= 41943040 block_cnt= 2 chunk_cnt= 2 mod=LogGroupBuffer [MEMORY] hold= 67190784 used= 67109344 count= 4 avg_used= 16777336 block_cnt= 4 chunk_cnt= 4 mod=MrgMemPoolChk [MEMORY] hold= 919207280 used= 901963089 count= 9503 avg_used= 94913 mod=SUMMARY [2025-08-14 09:07:06.750926] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= MEMSTORE_CTX_ID hold= 415236096 used= 411992064 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 411992064 used= 411803568 count= 198 avg_used= 2079816 block_cnt= 198 chunk_cnt= 198 mod=Memstore [MEMORY] hold= 411992064 used= 411803568 count= 198 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:07:06.750933] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= EXECUTE_CTX_ID hold= 1114112 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 7 washed_size= 7274496 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:07:06.750946] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1002 ctx_id= TRANS_CTX_MGR_ID hold= 950272 used= 933888 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1146880 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 933888 used= 921984 count= 2 avg_used= 460992 mod=SUMMARY [2025-08-14 09:07:06.750956] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= PLAN_CACHE_CTX_ID hold= 211812352 used= 201995008 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 176930880 used= 160228962 count= 13450 avg_used= 11912 block_cnt= 13450 chunk_cnt= 101 mod=SqlPhyPlan [MEMORY] hold= 201995008 used= 183048143 count= 17906 avg_used= 10222 mod=SUMMARY [2025-08-14 09:07:06.750964] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= WORK_AREA hold= 1441792 used= 0 limit= 429496725 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 14 washed_blocks= 54 washed_size= 27918336 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:07:06.750973] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1002 ctx_id= GLIBC hold= 12582912 used= 11425280 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 11425280 used= 6322556 count= 19216 avg_used= 329 mod=SUMMARY [2025-08-14 09:07:06.750988] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1002 ctx_id= CO_STACK hold= 149913600 used= 147087360 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 68 washed_blocks= 70 washed_size= 3178496 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 147087360 used= 146813760 count= 285 avg_used= 515136 block_cnt= 285 chunk_cnt= 73 mod=CoStack [MEMORY] hold= 147087360 used= 146813760 count= 285 avg_used= 515136 mod=SUMMARY [2025-08-14 09:07:06.751004] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1002 ctx_id= KVSTORE_CACHE_ID hold= 48234496 used= 48234496 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 48234496 used= 0 count= 23 avg_used= 0 mod=SUMMARY [2025-08-14 09:07:06.751016] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1002 ctx_id= META_OBJ_CTX_ID hold= 37748736 used= 35741632 limit= 1717986900 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 35741632 used= 32457488 count= 496 avg_used= 65438 mod=SUMMARY [2025-08-14 09:07:06.751026] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= TX_CALLBACK_CTX_ID hold= 7766016 used= 6934400 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 17 washed_size= 6914048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 6934400 used= 6729728 count= 848 avg_used= 7936 mod=SUMMARY [2025-08-14 09:07:06.751036] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= LOB_CTX_ID hold= 147456 used= 16384 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 16384 used= 15872 count= 2 avg_used= 7936 mod=SUMMARY [2025-08-14 09:07:06.751048] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= RPC_CTX_ID hold= 5783552 used= 5005312 limit= 8589934500 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 10 washed_blocks= 10 washed_size= 15187968 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 5005312 used= 3299200 count= 209 avg_used= 15785 mod=SUMMARY [2025-08-14 09:07:06.751483] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=19] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:06.752795] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=40] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:06.752929] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:06.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.756055] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:06.756422] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:06.757255] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:06.758442] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:06.761320] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:06.765869] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [372]OB_MALLOC COST TOO MUCH TIME, cost_time=1002248, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1002245], seq:[0] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=1002281, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1002276], seq:[0, 1] [2025-08-14 09:07:05.764517] EDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] LOGGER COST TOO MUCH TIME, cost: 1002315, owner: ObLog, click_count: 5, time dist:[FORMAT_END=17, ObMalloc#start=1, ObMalloc#end=1002290, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x13b06ffb 0x958a39e 0xfd28f5f 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:05.764517] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:07:06.768396] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=1002315][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:07:06.768430] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:07:06.768448] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:07:06.768460] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:07:06.768486] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:07:06.768507] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:07:06.768526] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:07:06.768547] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:07:06.768564] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:07:06.768581] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:07:06.768600] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:07:06.768618] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:07:06.768635] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:07:06.768659] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:07:06.768677] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:07:06.768690] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:07:06.768857] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:07:06.768886] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [381]OB_MALLOC COST TOO MUCH TIME, cost_time=1010735, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=69, WASH_BLOCK_END=141, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=991204], seq:[0, 1, 2, 3, 4] [2025-08-14 09:07:05.741812] EDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4038] LOGGER COST TOO MUCH TIME, cost: 1025959, owner: ObLog, click_count: 5, time dist:[FORMAT_END=15200, ObMalloc#start=0, ObMalloc#end=1010752, ALLOC_END=1, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1ba227a3 0x1b89078a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:02:32.736278] EDIAG [STORAGE] get_ls_tx_data_memory_info_ (ob_tenant_freezer.cpp:735) [491][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 150780296, owner: ObLog, click_count: 7, time dist:[FORMAT_END=62, ObMalloc#start=7, ObMalloc#end=63, ObMalloc#start=150780154, ObMalloc#end=5, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1c438438 0x1c43678a 0x1c43b877 0x1c47767a 0x9835119 0x9834fb5 0x98349ee 0xfd28f5f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:05.741812] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1002}) [2025-08-14 09:07:06.769346] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1025959] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1002}) [2025-08-14 09:07:06.769526] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:06.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.769561] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:06.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.753076] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:06.769933] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16829][errcode=0] epoll handler too much time :16865 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:06.755398] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:06.769965] WDIAG ev_invoke_pending (ev.c:2136) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14545][errcode=0] epoll handler too much time :14578 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:06.750471] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=1007315, expected_run_ts_us_=1755162425743155, ObClockGenerator::getRealClock()=1755162426750471, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770219] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19715][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770261] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=37][errcode=0] task scheduled too delay(scheduled_delay=1013360, expected_run_ts_us_=1755162425756899, ObClockGenerator::getRealClock()=1755162426770260, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fc830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770277] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fc830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770291] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=1014102, expected_run_ts_us_=1755162425756188, ObClockGenerator::getRealClock()=1755162426770291, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770305] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770319] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=1013999, expected_run_ts_us_=1755162425756319, ObClockGenerator::getRealClock()=1755162426770318, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a54b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770332] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a54b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770346] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=1009798, expected_run_ts_us_=1755162425760547, ObClockGenerator::getRealClock()=1755162426770346, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72edc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770359] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72edc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770372] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1005154, expected_run_ts_us_=1755162425765218, ObClockGenerator::getRealClock()=1755162426770372, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72edb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770398] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72edb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770411] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=1004288, expected_run_ts_us_=1755162425766122, ObClockGenerator::getRealClock()=1755162426770411, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770423] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770436] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1001803, expected_run_ts_us_=1755162425768633, ObClockGenerator::getRealClock()=1755162426770436, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770458] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770474] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=1009475, expected_run_ts_us_=1755162425760998, ObClockGenerator::getRealClock()=1755162426770473, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770486] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770499] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=999665, expected_run_ts_us_=1755162425770834, ObClockGenerator::getRealClock()=1755162426770499, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770520] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770533] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=991194, expected_run_ts_us_=1755162425779339, ObClockGenerator::getRealClock()=1755162426770533, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728ac10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770548] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728ac10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770562] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=967657, expected_run_ts_us_=1755162425802905, ObClockGenerator::getRealClock()=1755162426770562, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.802, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770574] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.802, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770588] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=969113, expected_run_ts_us_=1755162425801474, ObClockGenerator::getRealClock()=1755162426770588, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770600] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770623] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] task scheduled too delay(scheduled_delay=966313, expected_run_ts_us_=1755162425804310, ObClockGenerator::getRealClock()=1755162426770623, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728f9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770636] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728f9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770650] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=952109, expected_run_ts_us_=1755162425818540, ObClockGenerator::getRealClock()=1755162426770650, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eae10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770671] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eae10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770685] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=947778, expected_run_ts_us_=1755162425822907, ObClockGenerator::getRealClock()=1755162426770685, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770704] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770718] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=905359, expected_run_ts_us_=1755162425865358, ObClockGenerator::getRealClock()=1755162426770717, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770730] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770743] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=903970, expected_run_ts_us_=1755162425866773, ObClockGenerator::getRealClock()=1755162426770743, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.866, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770762] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:05.866, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770784] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=441684, expected_run_ts_us_=1755162426329099, ObClockGenerator::getRealClock()=1755162426770784, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770796] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770814] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72756b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770827] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.724, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770840] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a36b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770856] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919bfbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aa690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770868] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919911b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196cc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770881] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d40b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770894] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770906] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770921] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770934] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770954] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.744, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770967] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aaf90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770980] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aba10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.770996] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.771008] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196cb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.771021] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.771033] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.751, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.771046] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.771130] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=280][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:06.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.772456] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=361][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:06.591304] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=251][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fe030, caller:ob_table_lock_service.cpp:start:276, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE6$_1273, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe310, time_wheel:0x7f99535f3990, thread_pool:0x7f99535f3170, is_running:1, time_interval:10.00s, expected_run_time:09:07:06.426, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.772512] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=180931][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:06.591|207=181.21ms|210=0us|222=1us|total=181.22ms) [2025-08-14 09:07:06.772530] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] timer task use too much time(end=1755162426772530, start=1755162426591302, delta=181228) [2025-08-14 09:07:06.739824] WDIAG delay_warn (debug.h:103) [319][pnio1][T0][Y0-0000000000000000-0-0] [lt=10567][errcode=0] PNIO [delay_warn] pktc_flush_cb delay high: 276994942, start_us=1755162149744061 [2025-08-14 09:07:06.751060] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= TX_DATA_TABLE hold= 5718016 used= 4521856 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 9 washed_blocks= 16 washed_size= 13156352 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4521856 used= 4480128 count= 40 avg_used= 112003 mod=SUMMARY [2025-08-14 09:07:06.775431] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=24318] [MEMORY] tenant_id= 1002 ctx_id= MDS_DATA_ID hold= 3047424 used= 1998848 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 4 washed_size= 1146880 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1998848 used= 1936384 count= 244 avg_used= 7936 mod=SUMMARY [2025-08-14 09:07:06.775446] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1002 ctx_id= MDS_CTX_ID hold= 147456 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:07:06.775461] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1002 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:07:06.775473] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1002 ctx_id= MERGE_NORMAL_CTX_ID hold= 1032192 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 12 washed_size= 13647872 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:07:06.776442] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=229] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:07:06.779933] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:07:06.780324] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919acf10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.780363] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.781856] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=11] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:07:06.782896] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:06.783213] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:06.790413] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919abe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.790447] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.790458] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199fe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919acd90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.790476] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e80b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.790494] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a31b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ede10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.801325] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:06.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.802081] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=12] gc stale ls task succ [2025-08-14 09:07:06.810503] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.806, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.810549] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.810564] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.810582] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.800, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.810601] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728e330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7290690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.805, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.810628] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.803, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.813149] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=191] PNIO [ratelimit] time: 1755162426813147, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007615, add_bytes: 0 [2025-08-14 09:07:06.830249] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:06.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.830599] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.824, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.833734] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=309][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:07:06.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.839411] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=58][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:06.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.840661] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.832, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.840891] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:06.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.841136] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=316][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 46.096515 loop:26707, slowcnt: 439 [2025-08-14 09:07:06.870792] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ade10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.864, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.870834] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.870853] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919adc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.870868] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919adb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.872509] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:06.880853] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72865b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.878, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.900944] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728de30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7290b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.899, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.910986] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:06.902, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.939475] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:06.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.940955] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:06.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.972569] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:06.996660] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:06.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.012372] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:07.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.012418] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:07.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.023766] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:07.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.039536] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:06.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.040976] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:06.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.072598] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:07.092783] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:07.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.122191] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.111, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.139600] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:07.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.141024] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:07.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.172666] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:07.213425] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:07.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.239665] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:07.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.241086] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:07.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.248125] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:07.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.263709] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:07.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.263752] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:07.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.272699] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:07.275348] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:07.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.305532] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:07.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.335110] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=194] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:07:07.339724] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:07.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.341145] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:07.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.370485] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=12] gc stale ls task succ [2025-08-14 09:07:07.372758] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:07.439784] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:07.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.441208] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:07.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.472806] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:07.515211] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:07.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.515258] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:07.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.526899] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:07.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.539844] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:07.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.541238] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:07.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.572849] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:07.595655] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:07.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.639892] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:07.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.641308] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:07.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.672913] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:07.716222] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:07.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.722697] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=239] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:07:07.725327] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=208] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:07.734185] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:06.930270] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:06.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.258200] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:07.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.739471] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=480540][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:07.258|207=481.27ms|210=0us|222=1us|total=481.31ms) [2025-08-14 09:07:07.739515] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=43][errcode=-4389] timer task use too much time(end=1755162427739515, start=1755162427258198, delta=481317) [2025-08-14 09:07:07.739533] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=256001, expected_run_ts_us_=1755162427483531, ObClockGenerator::getRealClock()=1755162427739532, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:07.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.739551] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:07.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.769366] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [854][T1002_ObXAHbWor][T1002][Y0-0000000000000000-0-0] [lt=963410] XA scheduler heartbeat task statistics(avg_time=4) [2025-08-14 09:07:07.740140] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:07.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.729210] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [510][T1_TimerWK1_TabletShell][T1][Y0-0000000000000000-0-0] [lt=13] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-08-14 09:07:07.269067] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:07:07.740952] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=471167][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:07:07.740975] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:07:07.740987] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:07:07.740999] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:06:37.726, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:06:07.726], idx=471, next_allow_print_ts="09:07:07.726") [2025-08-14 09:07:07.741011] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:07:07.741024] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:07:07.741040] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:07:07.741072] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:07:07.741090] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:07:07.741102] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:07:07.741118] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:07:07.741132] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:07:07.741145] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:07:07.741157] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:07:07.741169] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:07:07.741183] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:07:07.741199] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:07:07.741372] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:07:07.741395] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:07:07.741502] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:07.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.739362] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=807818][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:06.930|207=809.04ms|210=0us|222=50us|total=811.35ms) [2025-08-14 09:07:07.741627] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=2261][errcode=-4389] timer task use too much time(end=1755162427741627, start=1755162426930268, delta=811359) [2025-08-14 09:07:06.729210] EDIAG [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [510][T1_TimerWK1_TabletShell][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1009672, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1009667, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1c45bdeb 0x1c45a909 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:07.743141] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [510][T1_TimerWK1_TabletShell][T1][Y0-0000000000000000-0-0] [lt=1009672] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=1256) [2025-08-14 09:07:07.743167] WDIAG alarm_if_necessary (ob_timer_service.cpp:129) [510][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4389] timer task too much delay in priority queue(token={this:0x7f994e1a5378, timer:0x7f994b3efe78, task:0x7f994b3efea0, task_type:"N9oceanbase7storage10checkpoint16ObEmptyShellTaskE", scheduled_time:1755162179758191, pushed_time:1755162183825422, delay:5000000}, delay_in_priority_queue=4067231, threashold=1000000) [2025-08-14 09:07:07.743190] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [510][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f994e1a5378, timer:0x7f994b3efe78, task:0x7f994b3efea0, task_type:"N9oceanbase7storage10checkpoint16ObEmptyShellTaskE", scheduled_time:1755162179758191, pushed_time:1755162183825422, delay:5000000}, delay_in_thread_pool=239913800, threashold=500000) [2025-08-14 09:07:07.746373] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.746423] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.746442] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891991430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ce10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.749915] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:07.752805] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:07.753026] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:07.753108] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=18] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:07.753140] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:07.754310] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:07.755644] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:07.755892] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:07.756482] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aac90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.756515] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.756534] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.756567] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e62b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.756582] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73940b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.757251] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:07.758622] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:07.759665] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=65][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:07.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.761215] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:07.765884] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:07.766535] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eeb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.766568] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73936b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eec90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.766583] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.769216] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=51][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:07.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.769263] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:07.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.772973] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:07.774744] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:07.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.776599] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.776642] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72765b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.776657] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.776672] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.776685] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.780051] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:07:07.780252] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=25] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:07:07.781958] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=21] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:07:07.782890] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:07.786664] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aab10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.774, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.786695] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739ef90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.786711] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a45b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.786725] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eee10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.786748] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73931b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.791687] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:07.796730] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.796763] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.796778] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7273c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.796792] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d27b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.796807] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d94b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.796822] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.780, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.796846] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739bbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.780, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.796861] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fa10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.796876] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f727dc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f729ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.802143] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=22] gc stale ls task succ [2025-08-14 09:07:07.804990] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:07.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.820785] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=22] PNIO [ratelimit] time: 1755162427820783, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007636, add_bytes: 0 [2025-08-14 09:07:07.834393] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:07:07.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.840203] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:07.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.841565] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:07.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.841668] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 47.097047 loop:26707, slowcnt: 449 [2025-08-14 09:07:07.857250] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fc8b0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcb90, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:1, time_interval:3.00s, expected_run_time:09:07:07.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.873047] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:07.940249] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=57][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:07.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.941626] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:07.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:07.973112] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:07.983722] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=337][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc5a78, timer:0x7f9970f8c420, task:0x564d6e8e31f8, task_type:"N9oceanbase3omt19ObTenantTimezoneMgr15AddTenantTZTaskE", scheduled_time:1755161702763912, pushed_time:0, delay:5000000}, running_task_token={this:0x7f9970fc66b8, timer:0x7f9970f8c420, task:0x564d6e8e3228, task_type:"N9oceanbase3omt19ObTenantTimezoneMgr18UpdateTenantTZTaskE", scheduled_time:1755161702763810, pushed_time:1755161702764275, delay:5000000}, total_delay=725219809, threashold=1000000, tenant_id=500) [2025-08-14 09:07:07.991184] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:07.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.026640] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:08.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.040336] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:07.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.041686] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:07.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.073177] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:08.140423] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:08.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.141750] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:08.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.173260] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:08.240485] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:08.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.241584] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:07:08.241610] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:07:08.241630] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:07:08.241641] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:07:08.241654] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:06:37.726, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:06:07.726], idx=471, next_allow_print_ts="09:07:07.726") [2025-08-14 09:07:08.241665] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:07:08.242576] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:08.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.252635] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:08.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.273332] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:08.278410] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:08.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.308592] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:08.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.340505] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:08.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.344633] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=20] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:07:08.371251] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:07:08.373412] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:08.449244] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:08.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.473478] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:08.494108] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:08.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.529882] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:08.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.549324] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:08.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.573511] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:08.649397] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:08.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.673573] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:08.721920] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:08.722822] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=30] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:07:08.728601] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:07.753093] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:08.730456] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=975884][errcode=0] epoll handler too much time :977368 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:07.739441] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=35312][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:07:07.796890] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f727d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f729af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:08.731850] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=933542][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:07.796|207=934.96ms|210=0us|222=2us|total=934.99ms) [2025-08-14 09:07:08.731883] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4389] timer task use too much time(end=1755162428731882, start=1755162427796889, delta=934993) [2025-08-14 09:07:08.731897] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=946051, expected_run_ts_us_=1755162427785844, ObClockGenerator::getRealClock()=1755162428731896, ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:08.731926] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.841695] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:07.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.010732] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:08.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.241678] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:07:08.732012] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=489586][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:07:08.732031] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:07:08.732048] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:07:08.732059] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:07:08.732076] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:07:08.241808] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:08.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.732108] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=489556][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:08.241|207=490.30ms|210=0us|222=2us|total=490.31ms) [2025-08-14 09:07:08.744931] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=12801][errcode=-4389] timer task use too much time(end=1755162428744930, start=1755162428241806, delta=503124) [2025-08-14 09:07:08.731963] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=888919][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:07.841|207=890.27ms|210=0us|222=1us|total=890.28ms) [2025-08-14 09:07:08.744970] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=12987][errcode=-4389] timer task use too much time(end=1755162428744970, start=1755162427841694, delta=903276) [2025-08-14 09:07:08.731985] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=720160][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:08.107|207=721.25ms|210=1us|222=1us|total=721.26ms) [2025-08-14 09:07:08.744992] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12987][errcode=-4389] timer task use too much time(end=1755162428744991, start=1755162428010730, delta=734261) [2025-08-14 09:07:08.745003] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=0] task scheduled too delay(scheduled_delay=739513, expected_run_ts_us_=1755162428005487, ObClockGenerator::getRealClock()=1755162428745001, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:08.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.745028] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:08.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.732089] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:07:08.745103] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12989][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:07:08.745136] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:07:08.745145] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:07:08.745155] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:07:08.745168] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:07:08.731942] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=938199, expected_run_ts_us_=1755162427793742, ObClockGenerator::getRealClock()=1755162428731941, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891814fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.793, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:08.749759] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:08.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.749888] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:08.744657] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:08.755798] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:08.745042] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=655725, expected_run_ts_us_=1755162428089316, ObClockGenerator::getRealClock()=1755162428745042, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:08.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.756575] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11513][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:08.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.756602] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=0] task scheduled too delay(scheduled_delay=552304, expected_run_ts_us_=1755162428204297, ObClockGenerator::getRealClock()=1755162428756601, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:08.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.756614] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:08.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.745436] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13472][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891814fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.793, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:08.745413] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:07:08.756842] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11406][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:07:08.757280] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:08.757321] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:08.761189] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:08.765893] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:08.756664] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11208][errcode=0] task scheduled too delay(scheduled_delay=960216, expected_run_ts_us_=1755162427796447, ObClockGenerator::getRealClock()=1755162428756664, ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:08.771194] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14504][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:08.771226] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] task scheduled too delay(scheduled_delay=963953, expected_run_ts_us_=1755162427807273, ObClockGenerator::getRealClock()=1755162428771226, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7facb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:08.771256] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7facb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:08.756632] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc190, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc470, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:10.00s, expected_run_time:09:07:08.720, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.771919] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:08.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.773742] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=34] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:07:08.773761] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:07:08.773771] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:08.780141] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:07:08.780354] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=24] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:07:08.782052] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=14] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:07:08.802109] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:08.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.802207] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=19] gc stale ls task succ [2025-08-14 09:07:08.812172] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40b7f0, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40bad0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:10.00s, expected_run_time:09:07:08.802, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.828403] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=23] PNIO [ratelimit] time: 1755162428828400, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007617, add_bytes: 0 [2025-08-14 09:07:08.834979] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:07:08.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.845023] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:08.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.849793] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:08.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.851126] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 48.106505 loop:26707, slowcnt: 459 [2025-08-14 09:07:08.873739] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:08.945083] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:08.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.949855] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:08.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.973799] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:09.023413] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:09.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.045142] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:08.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.049921] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:08.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.073863] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:09.145207] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:09.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.149938] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:09.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.173924] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:09.245282] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=53][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:09.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.250000] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:09.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.257027] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:07:09.257060] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:07:09.257081] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:07:09.257105] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:07:09.257122] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:06:37.726, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:06:07.726], idx=471, next_allow_print_ts="09:07:07.726") [2025-08-14 09:07:09.257140] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:07:09.257154] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:07:09.257175] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:07:09.257194] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:07:09.257216] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:07:09.257245] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:07:09.257266] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:07:09.257284] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:07:09.257301] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:07:09.257309] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:07:09.257326] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:07:09.257336] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:07:09.257350] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:07:09.257537] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:07:09.257554] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:07:09.273984] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:09.274846] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:09.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.304983] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:09.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.345314] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:09.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.350072] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:09.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.351496] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=18] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:07:09.371336] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=22] gc stale ls task succ [2025-08-14 09:07:09.374043] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:09.445388] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:09.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.450135] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:09.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.474094] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:09.526124] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:09.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.545452] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:09.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.550199] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:09.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.574155] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:09.645484] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:09.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.650263] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:09.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.674340] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:09.720980] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=40] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:09.724727] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=27] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:07:09.727442] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [861]OB_MALLOC COST TOO MUCH TIME, cost_time=998828, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=998819], seq:[0] [861]OB_MALLOC COST TOO MUCH TIME, cost_time=998868, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=998860], seq:[0, 1] [2025-08-14 09:07:07.740080] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=31764][errcode=-4018] LOGGER COST TOO MUCH TIME, cost: 1989035, owner: ObLog, click_count: 5, time dist:[FORMAT_END=990134, ObMalloc#start=0, ObMalloc#end=998876, ALLOC_END=1, APPEND_END=24], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3b4b 0x113d34aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [686]OB_MALLOC COST TOO MUCH TIME, cost_time=1000423, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=38, WASH_BLOCK_END=492, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=24471], seq:[0, 1, 2, 3, 4] [2025-08-14 09:07:06.739582] EDIAG [COMMON] loop_running_dag_net_list (ob_tenant_dag_scheduler.cpp:3845) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=172][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2989472, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1989027, ObMalloc#start=1, ObMalloc#end=1000439, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x20061064 0x2006f829 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [357]OB_MALLOC COST TOO MUCH TIME, cost_time=1001585, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=60, WASH_BLOCK_END=139, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=25239], seq:[0, 1, 2, 3, 4] [2025-08-14 09:07:06.767393] EDIAG [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=30537][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2962799, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1961192, ObMalloc#start=2, ObMalloc#end=1001600, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x18bdb084 0x18bda6b1 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [474]OB_MALLOC COST TOO MUCH TIME, cost_time=992510, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=16, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=992482], seq:[0, 1, 2] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=992526, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=992524], seq:[0, 1] [387]OB_MALLOC COST TOO MUCH TIME, cost_time=991173, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=991150], seq:[0, 1, 2] [387]OB_MALLOC COST TOO MUCH TIME, cost_time=991183, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=991181], seq:[0, 1] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=983489, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=983467], seq:[0, 1, 2] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=983497, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=983497], seq:[0, 1] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=981496, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=981477], seq:[0, 1, 2] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=981508, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=981507], seq:[0, 1] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=981502, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=981488], seq:[0, 1, 2] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=981510, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=981510], seq:[0, 1] [383]OB_MALLOC COST TOO MUCH TIME, cost_time=981175, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=981160], seq:[0, 1, 2] [383]OB_MALLOC COST TOO MUCH TIME, cost_time=981182, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=981182], seq:[0, 1] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=979331, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=979313], seq:[0, 1, 2] [391]OB_MALLOC COST TOO MUCH TIME, cost_time=979339, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=979339], seq:[0, 1] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=977980, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=21, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=977950], seq:[0, 1, 2] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=978001, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=978000], seq:[0, 1] [872]OB_MALLOC COST TOO MUCH TIME, cost_time=977642, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 5, time dist:[ALLOC_CHUNK_END=6, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=977619], seq:[0, 1, 2, 3, 4] [872]OB_MALLOC COST TOO MUCH TIME, cost_time=977654, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=977653], seq:[0, 1] [2025-08-14 09:07:07.750166] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [872][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E799C16-0-0] [lt=1866746][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1982659, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1004996, ObMalloc#start=0, ObMalloc#end=977659, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3c61 0x113d348b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:07.740080] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=31764][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:07:08.771301] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14644][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:08.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.732686] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=959926][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:08.771|207=961.38ms|210=0us|222=3us|total=961.41ms) [2025-08-14 09:07:09.732720] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4389] timer task use too much time(end=1755162429732720, start=1755162428771300, delta=961420) [2025-08-14 09:07:09.732735] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=977246, expected_run_ts_us_=1755162428755487, ObClockGenerator::getRealClock()=1755162429732734, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:08.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.732759] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:08.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.732779] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=643463, expected_run_ts_us_=1755162429089316, ObClockGenerator::getRealClock()=1755162429732779, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:09.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.732793] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:09.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.732807] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=528509, expected_run_ts_us_=1755162429204297, ObClockGenerator::getRealClock()=1755162429732807, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:09.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.732819] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:09.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:06.739582] INFO [COMMON] loop_running_dag_net_list (ob_tenant_dag_scheduler.cpp:3845) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=172] loop running dag net list(ret=0, running_dag_net_list_size=0, slow_dag_net_cnt=0, finished_dag_net_list_size=0) [2025-08-14 09:07:08.771273] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=775616, expected_run_ts_us_=1755162427995656, ObClockGenerator::getRealClock()=1755162428771273, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199fbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919acc10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.995, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.733615] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=960886][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199fbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919acc10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:07.995, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.733640] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:08.771|207=962.36ms|210=0us|222=3us|total=962.38ms) [2025-08-14 09:07:09.733654] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task use too much time(end=1755162429733653, start=1755162428771272, delta=962381) [2025-08-14 09:07:09.733676] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1014740, expected_run_ts_us_=1755162428718935, ObClockGenerator::getRealClock()=1755162429733676, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728bb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.718, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.733689] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728bb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.718, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.733710] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] task scheduled too delay(scheduled_delay=1013904, expected_run_ts_us_=1755162428719806, ObClockGenerator::getRealClock()=1755162429733710, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180fbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.719, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:08.782718] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:09.733742] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180fbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.719, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.733759] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=949602][errcode=0] epoll handler too much time :951049 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:09.733772] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] task scheduled too delay(scheduled_delay=986029, expected_run_ts_us_=1755162428747742, ObClockGenerator::getRealClock()=1755162429733772, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.733792] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.733822] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] task scheduled too delay(scheduled_delay=985387, expected_run_ts_us_=1755162428748434, ObClockGenerator::getRealClock()=1755162429733822, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.733834] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.733848] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=984773, expected_run_ts_us_=1755162428749075, ObClockGenerator::getRealClock()=1755162429733848, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.733869] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.733882] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=978155, expected_run_ts_us_=1755162428755726, ObClockGenerator::getRealClock()=1755162429733882, ret=0, ret="OB_SUCCESS", *this={this:0x7f98918105b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.733894] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918105b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.733906] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=968511, expected_run_ts_us_=1755162428765395, ObClockGenerator::getRealClock()=1755162429733906, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.733920] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.733933] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=970299, expected_run_ts_us_=1755162428763633, ObClockGenerator::getRealClock()=1755162429733932, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.733945] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.733957] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=965683, expected_run_ts_us_=1755162428768273, ObClockGenerator::getRealClock()=1755162429733957, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891815730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.733971] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891815730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.733984] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=963588, expected_run_ts_us_=1755162428770395, ObClockGenerator::getRealClock()=1755162429733984, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eca90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.733996] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eca90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734008] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=963391, expected_run_ts_us_=1755162428770617, ObClockGenerator::getRealClock()=1755162429734008, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e94b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734031] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e94b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734046] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=963259, expected_run_ts_us_=1755162428770786, ObClockGenerator::getRealClock()=1755162429734045, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734058] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734071] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=958949, expected_run_ts_us_=1755162428775121, ObClockGenerator::getRealClock()=1755162429734071, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180ea30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734091] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180ea30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734104] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=951234, expected_run_ts_us_=1755162428782869, ObClockGenerator::getRealClock()=1755162429734104, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734144] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734157] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=950052, expected_run_ts_us_=1755162428784105, ObClockGenerator::getRealClock()=1755162429734157, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734168] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734180] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=945093, expected_run_ts_us_=1755162428789087, ObClockGenerator::getRealClock()=1755162429734180, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196be10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734192] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196be10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734206] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=948868, expected_run_ts_us_=1755162428785337, ObClockGenerator::getRealClock()=1755162429734205, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728d430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7290f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734217] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728d430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7290f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734229] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=938464, expected_run_ts_us_=1755162428795764, ObClockGenerator::getRealClock()=1755162429734229, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734251] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734263] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=938357, expected_run_ts_us_=1755162428795906, ObClockGenerator::getRealClock()=1755162429734263, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73922b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734277] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73922b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734290] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=938253, expected_run_ts_us_=1755162428796036, ObClockGenerator::getRealClock()=1755162429734290, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734301] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734313] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=935176, expected_run_ts_us_=1755162428799136, ObClockGenerator::getRealClock()=1755162429734313, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.799, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734325] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.799, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734338] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=941404, expected_run_ts_us_=1755162428792934, ObClockGenerator::getRealClock()=1755162429734338, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734375] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734389] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=925178, expected_run_ts_us_=1755162428809210, ObClockGenerator::getRealClock()=1755162429734389, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.809, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734409] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.809, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734421] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=929561, expected_run_ts_us_=1755162428804860, ObClockGenerator::getRealClock()=1755162429734421, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eaa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734436] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eaa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734449] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=896910, expected_run_ts_us_=1755162428837539, ObClockGenerator::getRealClock()=1755162429734449, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73945b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.837, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734461] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73945b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.837, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734475] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=822707, expected_run_ts_us_=1755162428911768, ObClockGenerator::getRealClock()=1755162429734475, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.911, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734487] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:08.911, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734529] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:09.718, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734543] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:09.726, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.734555] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a67b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7397290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:09.728, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:06.767393] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=30537] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:07:08.741247] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8030, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8310, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:10.00s, expected_run_time:09:07:08.730, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.735322] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=992570][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:08.741|207=994.07ms|210=0us|222=2us|total=994.09ms) [2025-08-14 09:07:09.735343] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task use too much time(end=1755162429735342, start=1755162428741244, delta=994098) [2025-08-14 09:07:09.735355] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1001822, expected_run_ts_us_=1755162428733531, ObClockGenerator::getRealClock()=1755162429735354, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:08.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.735371] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:08.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.735387] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=990753, expected_run_ts_us_=1755162428744633, ObClockGenerator::getRealClock()=1755162429735386, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:08.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.735414] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:08.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:08.742816] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:09.735523] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=991206][errcode=0] epoll handler too much time :992718 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:08.750560] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=30] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:09.735567] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=983518][errcode=0] epoll handler too much time :985015 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:08.752598] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=11] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:09.735613] WDIAG ev_invoke_pending (ev.c:2136) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=981529][errcode=0] epoll handler too much time :983019 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:08.752641] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:09.735655] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=981527][errcode=0] epoll handler too much time :983021 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:08.753011] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:09.735698] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=981202][errcode=0] epoll handler too much time :982696 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:08.754895] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:09.735738] WDIAG ev_invoke_pending (ev.c:2136) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=979360][errcode=0] epoll handler too much time :980853 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:07.750166] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [872][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E799C16-0-0] [lt=1866746] ObArchiveSender is running(thread_index=1) [2025-08-14 09:07:08.782877] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:09.738818] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=954494][errcode=0] epoll handler too much time :955947 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:08.845072] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:08.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.739353] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=892925][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:08.845|207=894.28ms|210=0us|222=2us|total=894.30ms) [2025-08-14 09:07:09.739373] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4389] timer task use too much time(end=1755162429739372, start=1755162428845071, delta=894301) [2025-08-14 09:07:09.743254] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:09.743838] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919d9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:09.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.743883] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7277c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eac90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:09.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.743992] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9347] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:09.746003] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:09.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.750544] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:09.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.750630] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:09.750695] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=10] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:09.751565] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:09.751874] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:09.751904] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:09.753819] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:09.753858] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:09.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.753886] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:09.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:05.727669] EDIAG [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=1041012][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4022569, owner: ObLog, click_count: 5, time dist:[FORMAT_END=4022554, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=7, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0xfd250b6 0xfd247d3 0xfd244ea 0xfd242a2 0x1d19629b 0x1d17baa3 0x1d17ad94 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:05.727664] EDIAG [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=1028439][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4023312, owner: ObLog, click_count: 3, time dist:[FORMAT_END=4023308, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0xfd250b6 0xfd247d3 0xfd244ea 0xfd242a2 0x1d19629b 0x1d17baa3 0x1d17ad94 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:07.747841] EDIAG [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=2997508][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2010145, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2010129, ObMalloc#start=1, ObMalloc#end=11, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x260b4cc1 0x260b9960 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:07.739634] EDIAG [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [561][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=5795][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2021620, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2021585, ALLOC_END=1, APPEND_END=34], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x973b640 0x973b132 0x973a32d 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:09.755728] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:09.756203] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:09.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.733600] INFO [COMMON] loop_finished_dag_net_list (ob_tenant_dag_scheduler.cpp:3868) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=2989472] loop finsihed dag net list(ret=0, finished_dag_net_list_size=0) [2025-08-14 09:07:05.727669] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=1041012] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:07:05.727664] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=1028439] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=10626, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-08-14 09:07:09.734227] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=949230] blacklist_loop exec finished(cost_time=17, is_enabled=true, send_cnt=0) [2025-08-14 09:07:09.757253] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:09.757737] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:07:09.757761] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:07:09.757780] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:07:09.757793] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:07:09.757814] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:06:37.726, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:06:07.726], idx=471, next_allow_print_ts="09:07:07.726") [2025-08-14 09:07:09.757826] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:07:09.757838] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:07:09.757853] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:07:09.757874] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:07:09.757890] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:07:09.757902] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:07:09.757915] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=677, timeout ts=09:07:09.730, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:06:39.730], idx=677, next_allow_print_ts="09:07:39.730") [2025-08-14 09:07:09.757930] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:07:09.757951] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:07:09.757965] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:07:09.757976] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:07:09.757989] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:07:09.758002] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:07:09.758019] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:07:09.758192] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:07:09.758207] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:07:09.758799] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:09.001721] EDIAG [COMMON] generate_mod_stat_task (memory_dump.cpp:213) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1725141, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1725140, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf98b221 0xf98aea2 0x96b1d30 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:09.740521] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:355) [834][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=20] finish check and clear obj lock(ls->get_ls_id()={id:1001}) [2025-08-14 09:07:09.760841] WDIAG [RPC.OBRPC] in_black (ob_net_keepalive.cpp:257) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7980461][errcode=-4016] keepalive thread maybe not work, last_write_ts: 1755162235783184 [2025-08-14 09:07:09.760980] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:07.747841] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=2997508] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=25861750784, fast_delete_log_mode=false) [2025-08-14 09:07:09.763392] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:09.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.763431] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:09.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.763886] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7277230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:09.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.763909] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7272330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a5090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:09.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.763926] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919822b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:09.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.763946] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:09.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.763967] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d71b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:09.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.763988] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:09.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.764001] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a27b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:09.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.764017] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a40b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:09.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.764042] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7faf30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:09.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:07.739634] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [561][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=5795] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=16, total_destructed_detector=16, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f994b3ee2b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-08-14 09:07:09.764342] WDIAG [DETECT] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [561][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=2021620][errcode=0] cost too much time:ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t), (this=|threshold=100.00ms|start at 09:07:09.764|total=2.05s) [2025-08-14 09:07:09.765536] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:09.773903] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7352330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196fc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:09.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.774423] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:09.779658] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:09.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.780264] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=25] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:07:09.780490] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=20] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:07:09.781846] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:09.781877] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:09.782156] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:07:09.783938] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aca90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:09.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.783972] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f734fc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891918610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:09.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.794001] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:09.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.794025] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:09.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.802286] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:07:09.804061] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:09.794, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.809794] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:09.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.814123] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:09.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.814148] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:09.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.814162] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:09.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.814173] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7397b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:09.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.814183] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918299b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891916210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:09.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.834225] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:09.825, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.834263] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:09.826, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.834279] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:09.820, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.834291] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72879b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:09.828, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.839447] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:09.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.846062] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:09.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.846494] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd070, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fd350, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:5.00s, expected_run_time:09:07:09.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.846532] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:07:09.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.850606] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:09.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.851708] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 49.107087 loop:26707, slowcnt: 469 [2025-08-14 09:07:09.854312] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:09.846, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.874485] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:09.946126] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:09.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.950681] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:09.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.974545] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:10.025859] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:10.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.046188] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:09.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.050735] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:09.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.146256] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:10.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.150797] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:10.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.246320] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:10.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.250841] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:10.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.258533] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:07:10.258559] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:07:10.258589] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:07:10.258598] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:07:10.258608] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:06:37.726, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:06:07.726], idx=471, next_allow_print_ts="09:07:07.726") [2025-08-14 09:07:10.258617] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:07:10.258627] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:07:10.258639] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:07:10.258649] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:07:10.258662] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:07:10.258670] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:07:10.258684] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:07:10.258694] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:07:10.258704] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:07:10.258713] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:07:10.258723] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:07:10.258733] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:07:10.258746] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:07:10.258911] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:07:10.258930] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:07:10.277328] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:10.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.307470] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:10.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.346382] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:10.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.350871] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:10.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.355491] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=21] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:07:10.371398] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=22] gc stale ls task succ [2025-08-14 09:07:10.446445] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:10.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.450935] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:10.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.528860] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:10.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.546509] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:10.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.551011] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:10.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.646569] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:10.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.651064] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:10.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.997567] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:09.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.728068] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=729390][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:09.997|207=730.50ms|210=0us|222=2us|total=730.54ms) [2025-08-14 09:07:09.939501] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:09.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.728109] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=40][errcode=-4389] timer task use too much time(end=1755162430728108, start=1755162429997565, delta=730543) [2025-08-14 09:07:10.728127] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=483492, expected_run_ts_us_=1755162430244633, ObClockGenerator::getRealClock()=1755162430728126, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:10.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.728149] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:10.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.728115] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=787420][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:09.939|207=788.61ms|210=0us|222=2us|total=788.66ms) [2025-08-14 09:07:10.728162] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=47][errcode=-4389] timer task use too much time(end=1755162430728162, start=1755162429939499, delta=788663) [2025-08-14 09:07:10.014765] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:10.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.074604] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:10.728185] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=652590][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:10.728184] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=712338][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:10.147|207=713.42ms|210=0us|222=1us|total=713.43ms) [2025-08-14 09:07:10.728195] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] timer task use too much time(end=1755162430728195, start=1755162430014763, delta=713432) [2025-08-14 09:07:10.728202] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=0] task scheduled too delay(scheduled_delay=722714, expected_run_ts_us_=1755162430005487, ObClockGenerator::getRealClock()=1755162430728201, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:10.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.728220] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:10.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.728254] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] task scheduled too delay(scheduled_delay=638937, expected_run_ts_us_=1755162430089316, ObClockGenerator::getRealClock()=1755162430728254, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:10.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.728264] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:10.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.728277] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=523980, expected_run_ts_us_=1755162430204297, ObClockGenerator::getRealClock()=1755162430728277, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:10.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.728287] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:10.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.728956] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:288) [884][T1002_ArcFetche][T1002][YB42AC4D0007-00063C495ED99C16-0-0] [lt=48936] ObArchiveFetcher is running(thread_index=1) [2025-08-14 09:07:10.729058] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72760b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.719, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:10.721061] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=67] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:09.001721] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:213) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-08-14 09:07:10.729518] INFO [COMMON] handle (memory_dump.cpp:544) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=1725141] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-08-14 09:07:10.729947] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [834][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19244][errcode=-4389] run this task use too much time(*p_task_={this:0x7f99319fe030, caller:ob_table_lock_service.cpp:start:276, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE6$_1273, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe310, time_wheel:0x7f99535f3990, thread_pool:0x7f99535f3170, is_running:0, time_interval:10.00s, expected_run_time:09:07:16.426, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.729971] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:409) [834][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:07:10.729|total=14.15s) [2025-08-14 09:07:10.730522] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] update_check_range(min_check_version=2540, max_check_version=2540, global_version=2652) [2025-08-14 09:07:10.731670] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=15278] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:10.731699] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=27] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:10.724832] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=29] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:07:10.728934] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:10.739208] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10255][errcode=0] epoll handler too much time :10280 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:10.743291] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:10.744881] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:10.731705] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:10.745852] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=14124] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:10.745896] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=43] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:09.761106] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=8] finish to resize log external storage handler(current_ts=1755162428770304, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-08-14 09:07:10.746636] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:10.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:09.741545] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:594) [491][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=150780296] TxData Memory Statistic : (Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=0, Tenant Active TxData Memory(MB)=1, Freeze TxData Trigger Memory(MB)=102, Total TxDataTable Hold Memory(MB)=3, Total TxDataTable Memory Limit(MB)=409) [2025-08-14 09:07:10.748253] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:10.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.749291] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:10.749328] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:10.749342] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739be30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:10.749355] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:10.749587] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:10.752331] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:10.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.752402] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:10.752437] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=15] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:10.752460] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:10.756194] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:10.756418] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:10.756445] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:10.757283] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:10.745907] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:10.758876] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12947] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:10.758890] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:10.758906] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:10.759651] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:07:10.759677] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:07:10.759697] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:07:10.759711] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:07:10.759725] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:06:37.726, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:06:07.726], idx=471, next_allow_print_ts="09:07:07.726") [2025-08-14 09:07:10.759738] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:07:10.759753] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:07:10.759771] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:07:10.759792] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:07:10.759810] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:07:10.759823] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:07:10.759842] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:07:10.759856] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:07:10.759871] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:07:10.759882] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:07:10.759895] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:07:10.759908] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:07:10.759931] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:07:10.759957] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:10.760102] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:07:10.760118] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:07:10.760200] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:10.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.760242] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7edc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:10.760264] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ebe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:10.760278] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:10.760291] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.750, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:10.760305] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:10.760318] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728d1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7291110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:10.760970] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:10.765489] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=983514] snapshot_traffic_info once [2025-08-14 09:07:10.766496] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:10.771092] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:10.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.771128] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:10.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.771169] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aa990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:10.771186] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:10.771199] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919afe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:10.771212] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d62b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:10.771225] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:10.773743] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:10.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.758366] INFO [ARCHIVE] produce_log_fetch_task_ (ob_archive_sequencer.cpp:166) [731][T1001_ArcSeq][T1001][YB42AC4D0007-00063C495C999C16-0-0] [lt=1969932] archive round not in doing status, just skip(key={incarnation:-1, dest_id:-1, round:-1}, state={status:"INVALID"}) [2025-08-14 09:07:10.749747] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=30] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:07:10.749831] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=33] report RowHolderMapper summary info(count=2, bkt_cnt=248) [477]OB_MALLOC COST TOO MUCH TIME, cost_time=929354, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=929352], seq:[0] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=920010, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=920009], seq:[0] [477]OB_MALLOC COST TOO MUCH TIME, cost_time=945942, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=929379], seq:[0, 1] [553]OB_MALLOC COST TOO MUCH TIME, cost_time=962374, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=962367], seq:[0] [553]OB_MALLOC COST TOO MUCH TIME, cost_time=962674, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=116, WASH_BLOCK_END=162, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=962392], seq:[0, 1, 2, 3, 4] [2025-08-14 09:07:10.780567] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=10] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:07:10.782263] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=8] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:07:10.782286] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=10] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:07:10.782531] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:10.782556] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:10.782569] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:10.782579] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7397a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:10.782584] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:10.846702] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:10.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.847618] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:07:10.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.859876] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 50.115255 loop:26707, slowcnt: 479 [2025-08-14 09:07:10.946767] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:10.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.046832] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:10.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.146897] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:11.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.246960] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:11.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.260298] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:07:11.260333] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:07:11.260351] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:07:11.260360] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:07:11.260369] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:06:37.726, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:06:07.726], idx=471, next_allow_print_ts="09:07:07.726") [2025-08-14 09:07:11.260378] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:07:11.260403] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:07:11.260424] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:07:11.260434] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:07:11.260447] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:07:11.260456] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:07:11.260469] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:07:11.260479] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:07:11.260489] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:07:11.260497] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:07:11.260506] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:07:11.260516] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:07:11.260537] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:07:11.260702] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:07:11.260713] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:07:11.347022] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:11.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.447087] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:11.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.547153] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:11.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.647217] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:11.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.782612] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:11.722372] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=938337][errcode=0] epoll handler too much time :939765 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:10.797495] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.722439] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=923542][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:10.797|207=924.94ms|210=0us|222=2us|total=924.97ms) [2025-08-14 09:07:11.722464] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4389] timer task use too much time(end=1755162431722463, start=1755162430797493, delta=924970) [2025-08-14 09:07:11.018740] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:11.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.828246] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:10.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.722521] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=892921][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:10.828|207=894.29ms|210=0us|222=2us|total=894.30ms) [2025-08-14 09:07:11.723702] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1178][errcode=-4389] timer task use too much time(end=1755162431723701, start=1755162430828229, delta=895472) [2025-08-14 09:07:11.722475] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=941412, expected_run_ts_us_=1755162430781061, ObClockGenerator::getRealClock()=1755162431722474, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.723756] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1278][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.723771] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] task scheduled too delay(scheduled_delay=941112, expected_run_ts_us_=1755162430782658, ObClockGenerator::getRealClock()=1755162431723771, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.723781] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.722516] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=702707][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:11.187|207=703.77ms|210=0us|222=2us|total=703.80ms) [2025-08-14 09:07:11.723790] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=926967, expected_run_ts_us_=1755162430796823, ObClockGenerator::getRealClock()=1755162431723790, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.737897] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14083][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.731297] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8766][errcode=-4389] timer task use too much time(end=1755162431731296, start=1755162431018738, delta=712558) [2025-08-14 09:07:11.737922] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] task scheduled too delay(scheduled_delay=935704, expected_run_ts_us_=1755162430802217, ObClockGenerator::getRealClock()=1755162431737922, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.802, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.737922] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=6612][errcode=0] task scheduled too delay(scheduled_delay=732434, expected_run_ts_us_=1755162431005487, ObClockGenerator::getRealClock()=1755162431737922, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:11.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:10.749925] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=25] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:07:10.747782] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [319][pnio1][T1][Y0-0000000000000000-0-0] [lt=33393][errcode=-4012] tenant weak read service cluster heartbeat RPC fail(ret=-4012, rcode={code:-4012, msg:"weak read service cluster heartbeat rpc timeout, tenant_id=1, svr="172.77.0.7:2882", timeout=9000000, send_ts=1755162112740348", warnings:[]}, tenant_id_=1, dst="172.77.0.7:2882", cluster_service_tablet_id={id:226}) [303]OB_MALLOC COST TOO MUCH TIME, cost_time=935970, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=935947], seq:[0, 1, 2] [303]OB_MALLOC COST TOO MUCH TIME, cost_time=935985, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=935983], seq:[0, 1] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=933807, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=933784], seq:[0, 1, 2] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=933819, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=933817], seq:[0, 1] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=936067, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=920020], seq:[0, 1] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=910287, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=11, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=910254], seq:[0, 1, 2] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=910300, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=910299], seq:[0, 1] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=895045, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=9, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=895027], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=895079, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=895079], seq:[0, 1] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=886339, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=8, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=886322], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=886348, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=886347], seq:[0, 1] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=879461, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=12, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=879439], seq:[0, 1, 2] [652]OB_MALLOC COST TOO MUCH TIME, cost_time=879484, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=879483], seq:[0, 1] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=748249, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=14, ALLOC_CHUNK_END=13, ALLOC_CHUNK_END=748217], seq:[0, 1, 2] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=748257, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=748256], seq:[0, 1] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=380356, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=29, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=380317], seq:[0, 1, 2] [419]OB_MALLOC COST TOO MUCH TIME, cost_time=380366, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=380365], seq:[0, 1] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=369361, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 3, time dist:[ALLOC_CHUNK_END=15, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=369336], seq:[0, 1, 2] [730]OB_MALLOC COST TOO MUCH TIME, cost_time=369773, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=369771], seq:[0, 1] [2025-08-14 09:07:10.802363] INFO handle (memory_dump.cpp:671) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] statistics: tenant_cnt: 6, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 168 270 0 1 2 110 1 0 1 3 7 0 0 1 4 1 1 0 1 5 85 6 0 1 6 13 0 0 1 7 22 2 0 1 8 53 1 0 1 12 17 3 0 1 13 10 1 0 1 14 1 1 0 1 15 1 1 0 1 16 8 7 0 1 18 4 3 0 1 20 2 1 0 1 22 1 1 0 1 24 1 0 0 1 25 6 0 0 500 0 561 274 0 500 7 192 5 0 500 8 41 1 0 500 9 2 1 0 500 10 3 1 0 500 16 57 4 0 500 17 60 7 0 500 22 38 64 0 500 23 27 10 0 508 0 2 7 0 508 8 5 1 0 509 0 2 7 0 509 8 5 1 0 1001 0 147 252 0 1001 2 26 1 0 1001 3 5 0 0 1001 4 1 1 0 1001 5 85 6 0 1001 6 11 0 0 1001 7 30 2 0 1001 8 36 1 0 1001 12 17 3 0 1001 13 3 1 0 1001 14 1 1 0 1001 16 7 4 0 1001 18 4 3 0 1001 20 4 1 0 1001 25 8 0 0 1002 0 263 278 0 1002 2 198 1 0 1002 3 4 0 0 1002 4 1 1 0 1002 5 101 6 0 1002 6 14 0 0 1002 7 6 2 0 1002 8 73 1 0 1002 12 18 3 0 1002 13 7 1 0 1002 14 1 2 0 1002 16 10 7 0 1002 18 9 3 0 1002 20 2 1 0 1002 21 1 0 0 1002 24 1 0 0 1002 25 7 0 0 cost_time: 71877 [2025-08-14 09:07:10.804547] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:10.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.739805] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=936005] [MEMORY] tenant: 1, limit: 2147483648 hold: 1013174272 cache_hold: 56623104 cache_used: 56623104 cache_item_count: 27 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 444993536 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 230686720 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 1163264 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 483328 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 51617792 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 1458176 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 14204928 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 111149056 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35651584 limit= 429496720 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 8003584 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= PS_CACHE_CTX_ID hold_bytes= 65536 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 14827520 limit= 9223372036854775807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 3481600 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3964928 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 212992 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 884736 limit= 9223372036854775807 [2025-08-14 09:07:10.802314] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=13] gc stale ls task succ [2025-08-14 09:07:11.739817] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=933853][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:10.804|207=935.27ms|210=1us|222=1us|total=935.29ms) [2025-08-14 09:07:11.739841] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4389] timer task use too much time(end=1755162431739840, start=1755162430804545, delta=935295) [2025-08-14 09:07:11.739854] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 444993536 used= 389804064 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 70 washed_blocks= 287 washed_size= 70221824 [MEMORY] request_cached_chunk_cnt= 6 [MEMORY] hold= 389804064 used= 379547427 count= 32900 avg_used= 11536 mod=SUMMARY [2025-08-14 09:07:11.739878] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 230686720 used= 228884480 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 block_cnt= 110 chunk_cnt= 110 mod=Memstore [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:07:11.739857] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=729824, expected_run_ts_us_=1755162431010031, ObClockGenerator::getRealClock()=1755162431739856, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:11.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.739893] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= EXECUTE_CTX_ID hold= 1163264 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 15 washed_size= 13516800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:07:11.739889] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:11.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.739908] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:07:11.739922] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 51617792 used= 47452416 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 84 washed_blocks= 334 washed_size= 126640128 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 47452416 used= 43070492 count= 3992 avg_used= 10789 mod=SUMMARY [2025-08-14 09:07:11.739936] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= WORK_AREA hold= 1458176 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 13 washed_blocks= 66 washed_size= 25804800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:07:11.739950] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 14204928 used= 10970224 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 277 washed_size= 31932416 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10970224 used= 6078924 count= 18573 avg_used= 327 mod=SUMMARY [2025-08-14 09:07:10.828271] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:11.739974] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 111149056 used= 108896256 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 block_cnt= 211 chunk_cnt= 53 mod=CoStack [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 mod=SUMMARY [2025-08-14 09:07:11.739995] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 56623104 used= 56623104 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 56623104 used= 0 count= 27 avg_used= 0 mod=SUMMARY [2025-08-14 09:07:11.740007] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=910355][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:11.740009] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 34365376 limit= 429496720 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 34365376 used= 31086608 count= 475 avg_used= 65445 mod=SUMMARY [2025-08-14 09:07:11.740031] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 8003584 used= 4947392 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 10 washed_blocks= 23 washed_size= 12967936 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4947392 used= 4793344 count= 604 avg_used= 7936 mod=SUMMARY [2025-08-14 09:07:11.740047] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:07:11.740061] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 65536 used= 31024 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 2031616 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 31024 used= 27058 count= 20 avg_used= 1352 mod=SUMMARY [2025-08-14 09:07:10.843616] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=324] PNIO [ratelimit] time: 1755162430843613, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007614, add_bytes: 0 [2025-08-14 09:07:11.740078] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= RPC_CTX_ID hold= 14827520 used= 6135808 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 6135808 used= 4051328 count= 253 avg_used= 16013 mod=SUMMARY [2025-08-14 09:07:11.740096] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 3481600 used= 2391232 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 10 washed_size= 4907008 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2391232 used= 2367040 count= 36 avg_used= 65751 mod=SUMMARY [2025-08-14 09:07:10.852397] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:10.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.740114] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 3964928 used= 3080192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 3 washed_size= 229376 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3080192 used= 2983936 count= 376 avg_used= 7936 mod=SUMMARY [2025-08-14 09:07:11.740122] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=886381][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:10.852|207=887.73ms|210=0us|222=1us|total=887.74ms) [2025-08-14 09:07:11.740131] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= SCHEMA_SERVICE hold= 212992 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1884160 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:07:11.740140] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] timer task use too much time(end=1755162431740139, start=1755162430852395, delta=887744) [2025-08-14 09:07:11.740154] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=16] [MEMORY] tenant_id= 1 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:07:11.740168] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= MERGE_NORMAL_CTX_ID hold= 884736 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 6 washed_blocks= 9 washed_size= 11698176 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:07:11.740183] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 500, limit: 9223372036854775807 hold: 693125120 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 333369344 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 35291136 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 85983232 limit= 9223372036854775807 [MEMORY] ctx_id= LIBEASY hold_bytes= 3145728 limit= 9223372036854775807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 16777216 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 3063808 limit= 9223372036854775807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 37584896 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 78925824 limit= 9223372036854775807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 98983936 limit= 9223372036854775807 [2025-08-14 09:07:11.740222] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=18] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 333369344 used= 301812656 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 499 washed_blocks= 2229 washed_size= 922714112 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 301812656 used= 294377165 count= 12719 avg_used= 23144 mod=SUMMARY [2025-08-14 09:07:11.740261] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= GLIBC hold= 35291136 used= 21991152 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 186 washed_blocks= 812 washed_size= 367362048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 21991152 used= 10379125 count= 131700 avg_used= 78 mod=SUMMARY [2025-08-14 09:07:11.740277] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= CO_STACK hold= 85983232 used= 83091456 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 83091456 used= 82936896 count= 161 avg_used= 515136 block_cnt= 161 chunk_cnt= 41 mod=CoStack [MEMORY] hold= 83091456 used= 82936896 count= 161 avg_used= 515136 mod=SUMMARY [2025-08-14 09:07:11.740291] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= LIBEASY hold= 3145728 used= 3070816 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1048576 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3070816 used= 2859784 count= 223 avg_used= 12824 mod=SUMMARY [2025-08-14 09:07:11.737933] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.802, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.740304] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 500 ctx_id= LOGGER_CTX_ID hold= 16777216 used= 4161536 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4161536 used= 4159488 count= 2 avg_used= 2079744 mod=SUMMARY [2025-08-14 09:07:11.740311] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=2371][errcode=0] task scheduled too delay(scheduled_delay=925979, expected_run_ts_us_=1755162430814330, ObClockGenerator::getRealClock()=1755162431740310, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.814, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.740329] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 500 ctx_id= RPC_CTX_ID hold= 3063808 used= 1482752 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 57 washed_blocks= 70 washed_size= 116473856 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1482752 used= 979328 count= 61 avg_used= 16054 mod=SUMMARY [2025-08-14 09:07:11.740333] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.814, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.740346] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 500 ctx_id= PKT_NIO hold= 37584896 used= 36013568 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 50 washed_blocks= 71 washed_size= 99016704 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 36013568 used= 35017984 count= 125 avg_used= 280143 mod=SUMMARY [2025-08-14 09:07:11.740345] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=922525, expected_run_ts_us_=1755162430817820, ObClockGenerator::getRealClock()=1755162431740345, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.817, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.740357] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.817, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.740372] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= SCHEMA_SERVICE hold= 78925824 used= 68937344 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 22 washed_size= 5767168 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 68937344 used= 67674815 count= 4860 avg_used= 13924 mod=SUMMARY [2025-08-14 09:07:11.740371] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=921398, expected_run_ts_us_=1755162430818973, ObClockGenerator::getRealClock()=1755162431740371, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.740383] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.740386] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= UNEXPECTED_IN_500 hold= 98983936 used= 97174864 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 97174864 used= 86783232 count= 3014 avg_used= 28793 mod=SUMMARY [2025-08-14 09:07:11.740395] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=921276, expected_run_ts_us_=1755162430819118, ObClockGenerator::getRealClock()=1755162431740395, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.819, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.740402] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant: 508, limit: 1073741824 hold: 15613952 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 5128192 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10485760 limit= 9223372036854775807 [2025-08-14 09:07:11.740406] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7393bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.819, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.740423] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 508 ctx_id= DEFAULT_CTX_ID hold= 5128192 used= 3765328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3765328 used= 3723912 count= 84 avg_used= 44332 mod=SUMMARY [2025-08-14 09:07:11.740427] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] task scheduled too delay(scheduled_delay=926934, expected_run_ts_us_=1755162430813492, ObClockGenerator::getRealClock()=1755162431740427, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.813, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.740440] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.813, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.740448] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 508 ctx_id= CO_STACK hold= 10485760 used= 10321920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10321920 used= 10302720 count= 20 avg_used= 515136 mod=SUMMARY [2025-08-14 09:07:11.740452] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=911617, expected_run_ts_us_=1755162430828835, ObClockGenerator::getRealClock()=1755162431740452, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919831b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.828, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.740464] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919831b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.828, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.740476] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=905436, expected_run_ts_us_=1755162430835039, ObClockGenerator::getRealClock()=1755162431740476, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.835, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.740487] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.835, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.740499] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant: 509, limit: 1073741824 hold: 15613952 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 5128192 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10485760 limit= 9223372036854775807 [2025-08-14 09:07:11.740501] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=892813, expected_run_ts_us_=1755162430847687, ObClockGenerator::getRealClock()=1755162431740500, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.847, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.740509] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 509 ctx_id= DEFAULT_CTX_ID hold= 5128192 used= 3765328 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3765328 used= 3723912 count= 84 avg_used= 44332 mod=SUMMARY [2025-08-14 09:07:11.740512] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.847, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.740524] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=895776, expected_run_ts_us_=1755162430844747, ObClockGenerator::getRealClock()=1755162431740523, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.844, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.740532] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 509 ctx_id= CO_STACK hold= 10485760 used= 10321920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10321920 used= 10302720 count= 20 avg_used= 515136 mod=SUMMARY [2025-08-14 09:07:11.740534] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.844, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.740546] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=895572, expected_run_ts_us_=1755162430844974, ObClockGenerator::getRealClock()=1755162431740546, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.844, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.740562] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:10.844, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.740580] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant: 1001, limit: 1073741824 hold: 608747520 cache_hold: 44040192 cache_used: 44040192 cache_item_count: 21 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 302309376 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 54525952 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 3080192 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 483328 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 45514752 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 5521408 limit= 157286400 [MEMORY] ctx_id= GLIBC hold_bytes= 15499264 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 73424896 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35651584 limit= 214748360 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 3252224 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 14680064 limit= 1073741800 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 5570560 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3866624 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 1179648 limit= 9223372036854775807 [2025-08-14 09:07:11.740583] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=125235, expected_run_ts_us_=1755162431615348, ObClockGenerator::getRealClock()=1755162431740583, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ebbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:11.615, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.740595] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ebbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:11.615, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.740605] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= DEFAULT_CTX_ID hold= 302309376 used= 268516432 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 81 washed_blocks= 406 washed_size= 113410048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 268516432 used= 262397846 count= 3783 avg_used= 69362 mod=SUMMARY [2025-08-14 09:07:11.740608] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919abd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:11.721, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.740620] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1001 ctx_id= MEMSTORE_CTX_ID hold= 54525952 used= 54099968 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 54099968 used= 54075216 count= 26 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:07:11.740619] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919abb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:11.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.740630] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= EXECUTE_CTX_ID hold= 3080192 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 8 washed_size= 7405568 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:07:11.740632] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:11.726, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.740642] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:07:11.740644] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fdc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:11.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.740652] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= PLAN_CACHE_CTX_ID hold= 45514752 used= 36552320 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 77 washed_blocks= 201 washed_size= 132743168 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 36552320 used= 33009690 count= 3323 avg_used= 9933 mod=SUMMARY [2025-08-14 09:07:11.740656] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:11.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.740663] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= WORK_AREA hold= 5521408 used= 0 limit= 157286400 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 9 washed_blocks= 40 washed_size= 17547264 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:07:11.740668] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728be10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:11.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.740674] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= GLIBC hold= 15499264 used= 10889728 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 25 washed_blocks= 164 washed_size= 47415296 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10889728 used= 6020141 count= 18313 avg_used= 328 mod=SUMMARY [2025-08-14 09:07:11.740686] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= CO_STACK hold= 73424896 used= 71221248 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 32 washed_blocks= 34 washed_size= 2072576 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 71221248 used= 71088768 count= 138 avg_used= 515136 block_cnt= 138 chunk_cnt= 36 mod=CoStack [MEMORY] hold= 71221248 used= 71088768 count= 138 avg_used= 515136 mod=SUMMARY [2025-08-14 09:07:11.740702] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1001 ctx_id= KVSTORE_CACHE_ID hold= 44040192 used= 44040192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 44040192 used= 0 count= 21 avg_used= 0 mod=SUMMARY [2025-08-14 09:07:11.740712] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 33906624 limit= 214748360 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33906624 used= 30629648 count= 468 avg_used= 65447 mod=SUMMARY [2025-08-14 09:07:11.740723] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= TX_CALLBACK_CTX_ID hold= 3252224 used= 2342592 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 3 washed_blocks= 8 washed_size= 3039232 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2342592 used= 2269696 count= 286 avg_used= 7936 mod=SUMMARY [2025-08-14 09:07:11.740733] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:07:11.740745] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= RPC_CTX_ID hold= 14680064 used= 4227072 limit= 1073741800 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4227072 used= 2796032 count= 172 avg_used= 16256 mod=SUMMARY [2025-08-14 09:07:11.740758] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1001 ctx_id= TX_DATA_TABLE hold= 5570560 used= 2342784 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 8 washed_size= 2818048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2342784 used= 2319424 count= 30 avg_used= 77314 mod=SUMMARY [2025-08-14 09:07:11.740772] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1001 ctx_id= MDS_DATA_ID hold= 3866624 used= 2785280 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 5 washed_size= 4521984 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2785280 used= 2698240 count= 340 avg_used= 7936 mod=SUMMARY [2025-08-14 09:07:11.740788] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1001 ctx_id= MERGE_NORMAL_CTX_ID hold= 1179648 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 8 washed_blocks= 24 washed_size= 15597568 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:07:11.740814] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant: 1002, limit: 8589934592 hold: 1945841664 cache_hold: 48234496 cache_used: 48234496 cache_item_count: 23 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 1009610752 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 415236096 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 1114112 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 950272 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 211812352 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 1441792 limit= 429496725 [MEMORY] ctx_id= GLIBC hold_bytes= 12582912 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 149913600 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 37748736 limit= 1717986900 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 7766016 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 5783552 limit= 8589934500 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 5718016 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3047424 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 1032192 limit= 9223372036854775807 [2025-08-14 09:07:11.740844] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= DEFAULT_CTX_ID hold= 1009610752 used= 919357296 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 13 [MEMORY] hold= 185860096 used= 185534336 count= 114 avg_used= 1627494 block_cnt= 114 chunk_cnt= 104 mod=MysqlRequesReco [MEMORY] hold= 167792640 used= 167772192 count= 1 avg_used= 167772192 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 83927040 used= 83886080 count= 2 avg_used= 41943040 block_cnt= 2 chunk_cnt= 2 mod=LogGroupBuffer [MEMORY] hold= 67190784 used= 67109344 count= 4 avg_used= 16777336 block_cnt= 4 chunk_cnt= 4 mod=MrgMemPoolChk [MEMORY] hold= 919357296 used= 902107601 count= 9510 avg_used= 94858 mod=SUMMARY [2025-08-14 09:07:11.740859] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1002 ctx_id= MEMSTORE_CTX_ID hold= 415236096 used= 411992064 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 411992064 used= 411803568 count= 198 avg_used= 2079816 block_cnt= 198 chunk_cnt= 198 mod=Memstore [MEMORY] hold= 411992064 used= 411803568 count= 198 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:07:11.740869] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= EXECUTE_CTX_ID hold= 1114112 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 7 washed_size= 7274496 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:07:11.740879] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= TRANS_CTX_MGR_ID hold= 950272 used= 933888 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1146880 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 933888 used= 921984 count= 2 avg_used= 460992 mod=SUMMARY [2025-08-14 09:07:11.740891] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= PLAN_CACHE_CTX_ID hold= 211812352 used= 201995008 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 176930880 used= 160228962 count= 13450 avg_used= 11912 block_cnt= 13450 chunk_cnt= 101 mod=SqlPhyPlan [MEMORY] hold= 201995008 used= 183048143 count= 17906 avg_used= 10222 mod=SUMMARY [2025-08-14 09:07:11.740900] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= WORK_AREA hold= 1441792 used= 0 limit= 429496725 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 14 washed_blocks= 54 washed_size= 27918336 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:07:11.740917] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1002 ctx_id= GLIBC hold= 12582912 used= 11425280 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 11425280 used= 6322556 count= 19216 avg_used= 329 mod=SUMMARY [2025-08-14 09:07:11.740931] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1002 ctx_id= CO_STACK hold= 149913600 used= 147087360 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 68 washed_blocks= 70 washed_size= 3178496 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 147087360 used= 146813760 count= 285 avg_used= 515136 block_cnt= 285 chunk_cnt= 73 mod=CoStack [MEMORY] hold= 147087360 used= 146813760 count= 285 avg_used= 515136 mod=SUMMARY [2025-08-14 09:07:11.740946] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= KVSTORE_CACHE_ID hold= 48234496 used= 48234496 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 48234496 used= 0 count= 23 avg_used= 0 mod=SUMMARY [2025-08-14 09:07:11.740956] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= META_OBJ_CTX_ID hold= 37748736 used= 35741632 limit= 1717986900 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 35741632 used= 32457488 count= 496 avg_used= 65438 mod=SUMMARY [2025-08-14 09:07:11.740966] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= TX_CALLBACK_CTX_ID hold= 7766016 used= 6934400 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 17 washed_size= 6914048 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 6934400 used= 6729728 count= 848 avg_used= 7936 mod=SUMMARY [2025-08-14 09:07:11.740977] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= LOB_CTX_ID hold= 147456 used= 16384 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 16384 used= 15872 count= 2 avg_used= 7936 mod=SUMMARY [2025-08-14 09:07:11.740989] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= RPC_CTX_ID hold= 5783552 used= 5005312 limit= 8589934500 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 10 washed_blocks= 10 washed_size= 15187968 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 5005312 used= 3299200 count= 209 avg_used= 15785 mod=SUMMARY [2025-08-14 09:07:11.741008] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=14] [MEMORY] tenant_id= 1002 ctx_id= TX_DATA_TABLE hold= 5718016 used= 4521856 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 9 washed_blocks= 16 washed_size= 13156352 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4521856 used= 4480128 count= 40 avg_used= 112003 mod=SUMMARY [2025-08-14 09:07:11.741020] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= MDS_DATA_ID hold= 3047424 used= 1998848 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 4 washed_size= 1146880 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1998848 used= 1936384 count= 244 avg_used= 7936 mod=SUMMARY [2025-08-14 09:07:11.741030] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= MDS_CTX_ID hold= 147456 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:07:11.741043] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1002 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:07:11.741054] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1002 ctx_id= MERGE_NORMAL_CTX_ID hold= 1032192 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 12 washed_size= 13647872 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:07:10.860286] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fc8b0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcb90, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:1, time_interval:3.00s, expected_run_time:09:07:10.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.741139] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=879519][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:10.860|207=880.85ms|210=0us|222=1us|total=880.87ms) [2025-08-14 09:07:11.741160] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4389] timer task use too much time(end=1755162431741159, start=1755162430860284, delta=880875) [2025-08-14 09:07:10.991802] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:10.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.741243] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=748296][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:10.991|207=749.43ms|210=0us|222=10us|total=749.45ms) [2025-08-14 09:07:11.741257] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] timer task use too much time(end=1755162431741257, start=1755162430991800, delta=749457) [2025-08-14 09:07:11.741272] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=496638, expected_run_ts_us_=1755162431244633, ObClockGenerator::getRealClock()=1755162431741271, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:11.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.741291] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:11.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.360394] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=10] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:07:11.371443] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=22] gc stale ls task succ [2025-08-14 09:07:11.722320] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8380] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:11.741836] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19484][errcode=0] epoll handler too much time :19528 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:11.730045] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=36] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:11.741871] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11808][errcode=0] epoll handler too much time :11837 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:11.737948] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:11.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.741947] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=3937][errcode=0] task scheduled too delay(scheduled_delay=652630, expected_run_ts_us_=1755162431089316, ObClockGenerator::getRealClock()=1755162431741947, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:11.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.741970] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:11.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.741984] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=537687, expected_run_ts_us_=1755162431204297, ObClockGenerator::getRealClock()=1755162431741984, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:11.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.741995] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:11.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.733656] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=7032] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:07:11.737499] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:07:06.768281] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=26132] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", [SlidingWindow]={palf_id:1001, self:"172.77.0.7:2882", lsn_allocator:{max_log_id:8571658, max_lsn:7323687369, max_scn:{val:1755161749741193002, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:7278610047}, reuse_lsn:{lsn:7323686131}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:7281744329}}, last_submit_lsn:{lsn:7323686131}, last_submit_end_lsn:{lsn:7323687369}, last_submit_log_id:8571658, last_submit_log_pid:49, max_flushed_lsn:{lsn:7323686009}, max_flushed_end_lsn:{lsn:7323686131}, max_flushed_log_pid:49, committed_end_lsn:{lsn:7323686009}, last_slide_log_id:8571656, last_slide_scn:{val:1755161747742105001, v:0}, last_slide_lsn:{lsn:7323685887}, last_slide_end_lsn:{lsn:7323686009}, last_slide_log_pid:49, last_slide_log_accum_checksum:2505458373, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f990112a530}, [StateMgr]={this:0x7f990112e9b0, self:"172.77.0.7:2882", palf_id:1001, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"PENDING", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:49}, leader:"0.0.0.0:0", leader_epoch:-1, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:7323687369}, scan_disk_log_finished:true, last_check_start_id:8571657, is_changing_config_with_arb:false, reconfirm_start_time_us:1755161767744317, palf_role_change_cb:0x7f9901135470, allow_vote:true}, [ConfigMgr]={palf_id:1001, self:"172.77.0.7:2882", alive_paxos_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:49, prev:{version:1, config:{config_version:{proposal_id:48, config_seq:47}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:49, config_seq:48}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_mode_pid:49}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_end_lsn:{lsn:7323685240}, prev_mode_pid:49}, reconfig_barrier:{prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_end_lsn:{lsn:7323685240}, prev_mode_pid:49}, persistent_config_version:{proposal_id:49, config_seq:48}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"deault_region", register_time_us:1755161739755312, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1755161739755312, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f990112bc70}, [ModeMgr]={palf_id:1001, self:"172.77.0.7:2882", applied_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, accepted_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1001, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1753069671378640001, v:0}, base_lsn_for_block_gc:{lsn:7247314944}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:49}, log_config_meta:{version:3, proposal_id:49, prev:{version:1, config:{config_version:{proposal_id:48, config_seq:47}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:49, config_seq:48}, log_sync_memberlist:1{server:"172.77.0.7:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:45, prev_lsn:{lsn:7323685081}, prev_mode_pid:49}, log_snapshot_meta:{version:2, base_lsn:{lsn:7247314944}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1753069671378640000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:49, mode_version:0, access_mode:1, ref_scn:{val:1753069671378640000, v:0}}}, log_meta_storage:{log_tail:{lsn:765952}, readable_log_tail:{lsn:765952}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7381524480}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/meta", dir_fd:232, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66338816, block_header_serialize_buf_:0x7f9901130bb8, flashback_version:0}, log_storage:{log_tail:{lsn:7323686131}, readable_log_tail:{lsn:7323686131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:7314419712}, min_scn:{val:1755157128863258005, v:0}, curr_block_id:109, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:234, min_block_id:0, max_block_id:110, curr_writable_block_id:109}, logical_block_size_:67104768, curr_block_writable_size_:57838349, block_header_serialize_buf_:0x7f99011328b8, flashback_version:0}, palf_epoch:2, last_purge_throttling_ts:1755161769777679, this:0x7f990112f0f0}, [Reconfirm]={palf_id:1001, self:"172.77.0.7:2882", state:"WAITING_LOG_FLUSHED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:1755161769777679, this:0x7f990112eb30}) [2025-08-14 09:07:11.746571] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:11.746619] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:11.747246] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:11.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.742970] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5460] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:07:10.768540] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1432) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-08-14 09:07:06.746908] EDIAG [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=969186][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4994090, owner: ObLog, click_count: 7, time dist:[FORMAT_END=4994080, ObMalloc#start=0, ObMalloc#end=4, ALLOC_END=0, APPEND_END=6, ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=2], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xff22be4 0x97a9cd7 0x97a9238 0x97a9307 0x97a8fe2 0x986bc46 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:06.749960] EDIAG [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=186][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4991701, owner: ObLog, click_count: 3, time dist:[FORMAT_END=4991695, ALLOC_END=2, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xff22be4 0x97a9cd7 0x97a9238 0x97a9307 0x97a8fe2 0x986bc46 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:06.746908] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=969186] get_min_unreplayed_log_info(lsn={lsn:5282737655}, scn={val:1755100947825033001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161624848121, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:13063726, ret_code_:0}}, next_to_submit_lsn_:{lsn:5282737655}, next_to_submit_scn_:{val:1755100947825033001, v:0}, base_lsn_:{lsn:4831543296}, base_scn_:{val:1755063445733337001, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7605000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:5282737655}, end_lsn:{lsn:5282737655}, read_buf:{buf_len_:3747840, buf_:0x7f98f7605000}, block_size:67104768, log_storage_:0x7f993e529bf0, log_storage_:{log_tail:{lsn:5282737655}, readable_log_tail:{lsn:5282737655}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:5234171904}, min_scn:{val:1755152244802535004, v:0}, curr_block_id:78, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:206, min_block_id:42, max_block_id:79, curr_writable_block_id:78}, logical_block_size_:67104768, curr_block_writable_size_:18539017, block_header_serialize_buf_:0x7f993e52a8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:85, read_io_size:301367296, read_disk_cost_ts:2364604, start_lsn:{lsn:4831543296}}}}}}) [2025-08-14 09:07:06.749960] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=186] get_min_unreplayed_log_info(lsn={lsn:8461098908}, scn={val:1755100947600130001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161920778323, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:23084275, ret_code_:0}}, next_to_submit_lsn_:{lsn:8461098908}, next_to_submit_scn_:{val:1755100947600130001, v:0}, base_lsn_:{lsn:7515734016}, base_scn_:{val:1753840867171901045, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6e05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:8461098908}, end_lsn:{lsn:8461098908}, read_buf:{buf_len_:3747840, buf_:0x7f98f6e05000}, block_size:67104768, log_storage_:0x7f99297a3bf0, log_storage_:{log_tail:{lsn:8461098908}, readable_log_tail:{lsn:8461098908}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8455200768}, min_scn:{val:1755160384851871002, v:0}, curr_block_id:126, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:215, min_block_id:93, max_block_id:127, curr_writable_block_id:126}, logical_block_size_:67104768, curr_block_writable_size_:61206628, block_header_serialize_buf_:0x7f99297a48b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:198, read_io_size:699822080, read_disk_cost_ts:4178035, start_lsn:{lsn:7515734016}}}}}}) [2025-08-14 09:07:11.749514] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:11.750825] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72847b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:11.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.751490] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:11.751516] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=10] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:11.752312] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:11.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.752483] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:11.753254] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:11.754261] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:11.755805] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:11.757269] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:11.757813] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:11.760861] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:07:11.760904] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:07:11.760903] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:11.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.760929] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:07:11.760940] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891806f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:11.751, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.760953] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739fb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:11.750, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.761019] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=87][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:07:11.761060] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:06:37.726, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:06:07.726], idx=471, next_allow_print_ts="09:07:07.726") [2025-08-14 09:07:11.761079] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:07:11.761096] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:07:11.761192] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:07:11.761213] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:07:11.761230] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:07:11.761261] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:07:11.761278] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:07:11.761291] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:07:11.761304] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:07:11.761315] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:07:11.761375] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:07:11.761389] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:07:11.761404] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:07:11.761595] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:07:11.761610] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:07:11.761924] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:11.762277] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:11.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.762304] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:11.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.765399] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:11.770674] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:11.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.770968] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73539b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7396b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:11.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.780672] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=20] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:07:11.781474] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:11.782374] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=9] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:07:11.782385] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=17] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:07:11.782535] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:11.800968] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:11.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.801145] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:11.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.802379] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=936093] gc stale ls task succ [2025-08-14 09:07:11.811206] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d22b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eaf90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:11.808, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.821265] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:11.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.821293] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:11.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.823775] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:11.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.840112] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=41] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:07:11.840138] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:07:11.840148] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:11.840211] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:11.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.841399] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:11.831, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.841437] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:11.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.847309] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:11.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.848284] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:07:11.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.851242] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=895105] PNIO [ratelimit] time: 1755162431851229, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007616, add_bytes: 0 [2025-08-14 09:07:11.851461] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ac10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:11.849, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.860438] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 51.115816 loop:26707, slowcnt: 489 [2025-08-14 09:07:11.861519] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:11.852, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.871582] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:11.861, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.901761] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919859b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:11.896, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.923839] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:11.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.940133] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:11.940274] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:11.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.947357] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:11.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.998620] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:11.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.017706] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:12.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.017738] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:12.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.023857] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:12.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.023901] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:11.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.040169] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:12.040336] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:11.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.047394] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:11.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.098195] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:12.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.123964] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:12.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.140223] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:12.140393] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:12.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.147452] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:12.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.218931] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:12.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.224014] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:12.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.240265] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:12.240455] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:12.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.247512] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:12.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.250049] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:12.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.250095] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:12.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.262017] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:07:12.262061] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:07:12.262084] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:07:12.262101] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:07:12.262118] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:06:37.726, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:06:07.726], idx=471, next_allow_print_ts="09:07:07.726") [2025-08-14 09:07:12.262130] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:07:12.262143] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:07:12.262171] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:07:12.262184] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:07:12.262201] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:07:12.262212] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:07:12.262229] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:07:12.262256] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:07:12.262280] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:07:12.262293] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:07:12.262306] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:07:12.262319] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:07:12.262336] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:07:12.262502] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:07:12.262517] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:07:12.269227] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:12.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.269275] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:12.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.275286] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:12.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.305501] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:12.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.324042] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:12.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.340328] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:12.340519] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:12.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.347552] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:12.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.371506] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=369808] gc stale ls task succ [2025-08-14 09:07:12.424099] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:12.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.440389] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:12.440580] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:12.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.441850] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=380391] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:07:12.447599] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:12.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.491473] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:12.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.510632] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:12.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.510667] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:12.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.524162] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:12.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.526659] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:12.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.540427] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:12.540602] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:12.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.547660] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:12.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.591137] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:12.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.624224] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:12.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.640485] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:12.640672] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:12.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.647704] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:12.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.711867] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:12.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.715780] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1049) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=4991701] get_min_unreplayed_log_info(lsn={lsn:8461098908}, scn={val:1755100947600130001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161920778323, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:23084275, ret_code_:0}}, next_to_submit_lsn_:{lsn:8461098908}, next_to_submit_scn_:{val:1755100947600130001, v:0}, base_lsn_:{lsn:7515734016}, base_scn_:{val:1753840867171901045, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f6e05000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:8461098908}, end_lsn:{lsn:8461098908}, read_buf:{buf_len_:3747840, buf_:0x7f98f6e05000}, block_size:67104768, log_storage_:0x7f99297a3bf0, log_storage_:{log_tail:{lsn:8461098908}, readable_log_tail:{lsn:8461098908}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:8455200768}, min_scn:{val:1755160384851871002, v:0}, curr_block_id:126, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:215, min_block_id:93, max_block_id:127, curr_writable_block_id:126}, logical_block_size_:67104768, curr_block_writable_size_:61206628, block_header_serialize_buf_:0x7f99297a48b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:198, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:198, read_io_size:699822080, read_disk_cost_ts:4178035, start_lsn:{lsn:7515734016}}}}}}, ret=0) [2025-08-14 09:07:12.716225] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1049) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=4994090] get_min_unreplayed_log_info(lsn={lsn:5282737655}, scn={val:1755100947825033001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1755161624848121, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:13063726, ret_code_:0}}, next_to_submit_lsn_:{lsn:5282737655}, next_to_submit_scn_:{val:1755100947825033001, v:0}, base_lsn_:{lsn:4831543296}, base_scn_:{val:1755063445733337001, v:0}, iterator_:{iterator_impl:{buf_:0x7f98f7605000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:5282737655}, end_lsn:{lsn:5282737655}, read_buf:{buf_len_:3747840, buf_:0x7f98f7605000}, block_size:67104768, log_storage_:0x7f993e529bf0, log_storage_:{log_tail:{lsn:5282737655}, readable_log_tail:{lsn:5282737655}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:5234171904}, min_scn:{val:1755152244802535004, v:0}, curr_block_id:78, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:206, min_block_id:42, max_block_id:79, curr_writable_block_id:78}, logical_block_size_:67104768, curr_block_writable_size_:18539017, block_header_serialize_buf_:0x7f993e52a8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:false, hot_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:85, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:85, read_io_size:301367296, read_disk_cost_ts:2364604, start_lsn:{lsn:4831543296}}}}}}, ret=0) [2025-08-14 09:07:06.768281] EDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=26132][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5941280, owner: ObLog, click_count: 5, time dist:[FORMAT_END=4962187, ObMalloc#start=0, ObMalloc#end=4996, ALLOC_END=7, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x9bb826f 0xf9861e5 0xf985ee2 0xf985cc1 0xfc0e73d 0x993f560 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:11.754105] INFO [SERVER] check_tenant_memory (ob_plugin_vector_index_scheduler.cpp:493) [807][T1002_TimerWK3_TTLTabletMgr][T1002][YB42AC4D0007-00063C495F8A1988-0-0] [lt=13] get vector mem limit size(ret=0, ret="OB_SUCCESS", tenant_id=1002, current_memory_config=0) [2025-08-14 09:07:12.727436] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:12.727561] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:12.735380] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:12.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.735843] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:12.724, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:12.740548] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:12.740739] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:12.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.742510] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:12.742869] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=9092] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:07:12.744759] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=39] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:12.744886] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:12.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.747767] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:12.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.749260] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:12.751255] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:12.751283] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=9] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:12.751990] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:12.752999] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:12.754305] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:12.755829] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:12.756280] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=51][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:12.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.756323] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=355][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:12.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:12.756343] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:12.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:11.775469] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [564][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=29720] XA scheduler heartbeat task statistics(avg_time=490966) [2025-08-14 09:07:12.757492] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=128] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:12.757821] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:12.721102] INFO [STORAGE.TRANS] statistics (ob_location_adapter.cpp:70) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] location adapter statistics(renew_access=5, total_access=5, error_count=0, renew_rate=8.333333135e-01) [2025-08-14 09:07:12.761249] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:12.762701] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:07:12.762737] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:07:12.762757] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:07:12.762769] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:07:12.762780] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:06:37.726, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:06:07.726], idx=471, next_allow_print_ts="09:07:07.726") [2025-08-14 09:07:12.762799] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:07:12.762811] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:07:12.762827] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:07:12.762839] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:07:12.762856] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:07:12.762874] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:07:12.762899] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:07:12.762912] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:07:12.762925] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:07:12.762936] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:07:12.762947] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:07:12.762960] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:07:12.762996] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:07:12.763158] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:07:12.763172] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:07:12.765413] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:12.766383] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:12.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:12.770183] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=52][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:12.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.770523] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=335][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:12.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.776477] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:12.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:12.776513] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:12.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:12.776526] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d36b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ebb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:12.767, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:12.776538] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:12.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:12.779050] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:12.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.780774] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=28] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:07:12.782438] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:12.782481] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:12.782480] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=11] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:07:12.783203] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=21] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:07:12.786474] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:12.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:12.786498] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72739b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a5e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:12.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:12.786510] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72779b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eab10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:12.777, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:12.786523] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739b6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728bc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:12.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:12.796887] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919902b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196c390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:12.786, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:12.796926] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:12.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:12.796937] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:12.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:12.802444] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=15] gc stale ls task succ [2025-08-14 09:07:12.806925] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7277730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:12.794, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:12.806953] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:12.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:12.809404] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:12.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.827069] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:12.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:12.835458] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=67][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:12.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.837132] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:12.820, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:12.840605] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:12.840791] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:12.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.847196] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:12.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:12.847801] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:12.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.848887] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:07:12.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.858848] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=25] PNIO [ratelimit] time: 1755162432858845, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007616, add_bytes: 0 [2025-08-14 09:07:12.935523] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=58][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:12.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.940624] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:12.940849] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:12.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.947850] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:12.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:12.968022] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196aa90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:12.952, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:12.968071] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:12.953, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:12.988154] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891992fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196de90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:12.970, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:12.998213] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:12.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.011630] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:13.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.011672] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:13.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.020736] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:13.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.035593] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=75][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:12.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.040643] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:13.040910] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:12.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.047896] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:12.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.092078] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:13.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.135656] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:13.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.140702] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:13.140963] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:13.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.147931] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:13.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.212780] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:13.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.235721] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:13.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.240760] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:13.240996] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:13.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.247956] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:13.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.249872] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:13.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.259957] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:13.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.263034] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:13.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.263068] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:13.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.264577] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:07:13.264617] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:07:13.264641] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:07:13.264658] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:07:13.264675] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:06:37.726, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:06:07.726], idx=471, next_allow_print_ts="09:07:07.726") [2025-08-14 09:07:13.264702] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:07:13.264721] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:07:13.264742] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:07:13.264760] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:07:13.264780] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:07:13.264796] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:07:13.264817] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:07:13.264835] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:07:13.264852] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:07:13.264876] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:07:13.264893] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:07:13.264911] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:07:13.264932] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:07:13.265110] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:07:13.265130] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:07:13.272182] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:13.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.302368] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:13.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.335782] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:13.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.340831] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:13.341058] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:13.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.348019] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:13.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.371570] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=21] gc stale ls task succ [2025-08-14 09:07:13.435793] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:13.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.440882] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:13.441099] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:13.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.442413] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=25] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:07:13.448085] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:13.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.491201] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:13.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.514421] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:13.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.514464] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:13.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.523517] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:13.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.535853] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:13.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.540944] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:13.541161] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:13.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.548149] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:13.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.594821] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:13.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.635914] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:13.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.641003] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:13.641223] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:13.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.648211] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:13.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.715808] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:13.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.721500] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:13.727913] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:13.739448] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:13.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.741065] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:13.741246] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:13.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.743125] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:07:13.743334] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:13.746422] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:13.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.746977] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:13.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:13.747009] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:13.737, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:13.747024] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e71b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:13.739, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:13.747037] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918119b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:13.737, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:13.747289] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=71] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:13.748275] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:13.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.749306] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:13.751267] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=10] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:13.751886] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:13.752979] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:13.753009] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:13.757700] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72774b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:13.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:13.757731] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fbe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:13.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:13.759025] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:13.759272] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:13.759508] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:13.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.759757] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 08:59:58.729291] EDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [954][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8E9F-0-0] [lt=1008227][errcode=-4341] LOGGER COST TOO MUCH TIME, cost: 434374630, owner: ObLog, click_count: 7, time dist:[FORMAT_END=20154, ObMalloc#start=1, ObMalloc#end=24478, ObMalloc#start=434329986, ObMalloc#end=3, ALLOC_END=1, APPEND_END=7], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1c03240b 0x1c031da7 0x1c03844f 0x984d055 0x9280839 0x926c154 0x12c5a0c9 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 08:59:58.729291] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc (ob_weak_read_service.cpp:223) [954][T1002_L0_G0][T1002][YB42AC4D0007-00063C49612B8E9F-0-0] [lt=1008227][errcode=-4341] tenant weak read service process cluster heartbeat RPC fail(ret=-4341, ret="OB_NOT_IN_SERVICE", tenant_id=1002, req={req_server:"172.77.0.7:2882", version:{val:1755161758221798000, v:0}, valid_part_count:0, total_part_count:2, generate_timestamp:1755161968826921}, twrs={inited:true, tenant_id:1002, self:"172.77.0.7:2882", svr_version_mgr:{server_version:{version:{val:1755161758221798000, v:0}, total_part_count:2, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755161997747090}, server_version_for_stat:{version:{val:18446744073709551615, v:3}, total_part_count:1, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1755161997832893}}, cluster_service:{current_version:{val:18446744073709551615, v:3}, min_version:{val:18446744073709551615, v:3}, max_version:{val:18446744073709551615, v:3}}}) [2025-08-14 09:07:13.763321] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:13.765280] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:07:13.765311] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:07:13.765341] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:07:13.765353] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:07:13.765366] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:06:37.726, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:06:07.726], idx=471, next_allow_print_ts="09:07:07.726") [2025-08-14 09:07:13.765377] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:07:13.765396] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:07:13.765412] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:07:13.765424] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:07:13.765449] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:07:13.765460] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:07:13.765476] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:07:13.765489] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:07:13.765502] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:07:13.765513] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:07:13.765525] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:07:13.765538] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:07:13.765553] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:07:13.765600] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=26] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:13.765708] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:07:13.765724] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:07:13.768454] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:13.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:13.768486] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:13.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:13.768496] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc0b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:13.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:13.769490] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=60][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:13.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.769537] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:13.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.778465] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:13.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.778522] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919aef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:13.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:13.778541] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7270a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a4190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:13.762, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:13.778551] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:13.764, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:13.778560] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:13.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:13.778573] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:13.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:13.780827] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=35] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:07:13.782073] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:13.782594] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=17] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:07:13.783385] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:07:13.783475] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:13.788577] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f734de30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891919810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:13.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:13.788619] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:13.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:13.788634] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:13.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:13.788648] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d59b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:13.773, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:13.788661] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ea030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efa10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:13.777, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:13.798587] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eba10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:13.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:13.798622] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ecfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:13.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:13.802509] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=18] gc stale ls task succ [2025-08-14 09:07:13.808648] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:13.796, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:13.808648] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:13.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.808683] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:13.790, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:13.838759] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:13.827, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:13.839506] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=48][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:13.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.841122] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:13.841300] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:13.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.843143] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fc8b0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcb90, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:1, time_interval:3.00s, expected_run_time:09:07:13.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.848327] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:13.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.848822] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739af30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196bc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:13.837, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:13.849470] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:07:13.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.858894] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:13.841, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:13.858936] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:13.849, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:13.858948] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:13.849, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:13.866462] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=25] PNIO [ratelimit] time: 1755162433866460, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007615, add_bytes: 0 [2025-08-14 09:07:13.909134] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:13.892, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:13.929256] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:13.917, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:13.939318] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:13.922, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:13.939561] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:13.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.941185] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:13.941362] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:13.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.943405] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=372][errcode=0] EASY SLOW: thread: 7f9954179300, lastrun: 1755162433.759726 cost: 0.183677 loop:26787, slowcnt: 1 [2025-08-14 09:07:13.948393] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:13.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:13.990991] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:13.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.010971] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:14.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.011009] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:14.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.029888] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:14.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.039627] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:13.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.041251] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:14.041402] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:13.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.048458] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:13.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.091483] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:14.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.139693] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:14.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.141313] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:14.141465] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:14.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.148527] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:14.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.212180] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:14.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.239772] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=49][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:14.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.241375] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:14.241494] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:14.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.242565] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:14.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.248592] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:14.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.252604] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:14.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.262510] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:14.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.262544] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:14.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.265875] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:07:14.265903] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:07:14.265922] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:07:14.265935] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:07:14.265947] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:06:37.726, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:06:07.726], idx=471, next_allow_print_ts="09:07:07.726") [2025-08-14 09:07:14.265959] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:07:14.265972] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:07:14.265988] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:07:14.266001] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:07:14.266017] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:07:14.266029] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:07:14.266055] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:07:14.266068] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:07:14.266081] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:07:14.266094] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:07:14.266107] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:07:14.266129] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:07:14.266145] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:07:14.266336] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:07:14.266350] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:07:14.271211] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:14.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.301393] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:14.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.339810] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:14.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.341444] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:14.341559] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:14.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.348663] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:14.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.371658] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=22] gc stale ls task succ [2025-08-14 09:07:14.439874] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:14.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.441505] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:14.441604] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:14.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.443819] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=23] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:07:14.448727] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:14.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.493932] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:14.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.514045] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:14.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.514084] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:14.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.522615] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=53][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:14.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.539902] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:14.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.541565] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:14.541668] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:14.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.548791] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:14.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.594549] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:14.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.639964] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=57][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:14.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.641922] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:14.641952] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:14.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.648853] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:14.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.715314] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:14.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.727947] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:14.728244] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:14.734569] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.728, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:14.734631] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=59][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f734c030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891807390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:14.735721] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd110, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd3f0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:10.00s, expected_run_time:09:07:14.726, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.739979] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:14.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.741992] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:14.742016] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:14.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.743194] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:14.743203] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=21] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:07:14.744304] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:14.744974] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aae10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:14.745008] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:14.745291] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=37565] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2025-08-14 09:07:14.745023] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e67b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:14.745389] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=363][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:14.746285] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:14.733, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.748917] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:14.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.749320] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:14.752891] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:14.754972] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=19] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:14.755014] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:14.755038] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:14.755069] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:14.756285] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919bfe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aa810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:14.756331] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:14.756347] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:14.756362] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:14.756381] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:14.757684] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:14.761252] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:05:09.746726] EDIAG [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1006721, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1006202, ObMalloc#start=0, ObMalloc#end=78, ObMalloc#start=30, ObMalloc#end=60, FREE_END=351], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1f832f91 0x97c36fe 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:14.765576] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:14.766531] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:07:14.766570] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:07:14.766591] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:07:14.766610] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:07:14.766624] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:06:37.726, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:06:07.726], idx=471, next_allow_print_ts="09:07:07.726") [2025-08-14 09:07:14.766640] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:07:14.766658] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:07:14.766678] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:07:14.766697] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:07:14.766727] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:07:14.766744] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:07:14.766762] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:07:14.766788] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:07:14.766806] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:07:14.766823] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:07:14.766840] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:07:14.766855] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:07:14.766881] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:07:14.767046] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:07:14.767067] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:07:14.777772] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=44][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:14.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.780922] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=28] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:07:14.782702] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=20] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:07:14.783457] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=30] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:07:14.802575] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=17] gc stale ls task succ [2025-08-14 09:07:14.807934] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:14.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.842075] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:14.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.848980] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:14.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.850634] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd070, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fd350, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:5.00s, expected_run_time:09:07:14.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.850658] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:07:14.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.874079] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=23] PNIO [ratelimit] time: 1755162434874076, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007616, add_bytes: 0 [2025-08-14 09:07:14.881650] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=25232] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:07:14.894653] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=987893] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:07:14.914983] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=25299] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:07:14.942117] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:14.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.944186] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=37][errcode=0] EASY SLOW: thread: 7f9954179300, lastrun: 1755162433.759726 cost: 1.184458 loop:26787, slowcnt: 11 [2025-08-14 09:07:14.949046] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:14.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:15.029261] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:15.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:15.042147] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=50][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:14.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:15.049109] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:14.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:15.142208] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:15.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:15.149172] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:15.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:15.242245] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=52][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:15.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:15.249244] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:15.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:15.267256] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:07:15.267291] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:07:15.267316] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:07:15.267332] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:07:15.267348] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:06:37.726, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:06:07.726], idx=471, next_allow_print_ts="09:07:07.726") [2025-08-14 09:07:15.267363] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:07:15.267381] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:07:15.267396] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:07:15.267411] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:07:15.267427] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:07:15.267447] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:07:15.267461] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:07:15.267483] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:07:15.267496] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:07:15.267511] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:07:15.267527] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:07:15.267541] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:07:15.267561] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:07:15.267730] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:07:15.267758] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:07:15.270640] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:15.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:15.300786] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:15.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:15.342290] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:15.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:15.349304] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:15.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:15.371720] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:07:15.442351] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:15.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:15.444463] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=18] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:07:15.449366] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:15.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:15.522051] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:15.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:15.542388] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:15.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:15.549429] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:15.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:15.642458] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:15.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:15.649492] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:15.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:15.721023] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=40] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:15.733580] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:15.742521] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:15.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:15.743177] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:15.743328] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=20] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-08-14 09:07:15.744348] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:15.749274] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:15.749555] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:15.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:15.751970] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=19] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:15.752007] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:15.752037] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:15.752686] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:15.754036] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:15.757562] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:15.761068] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:15.770393] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:15.770464] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:07:15.770500] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:07:15.770521] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:07:15.770533] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:07:15.770544] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:06:37.726, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:06:07.726], idx=471, next_allow_print_ts="09:07:07.726") [2025-08-14 09:07:15.770554] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:07:15.770566] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:07:15.770580] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:07:15.770592] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:07:15.770607] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:07:15.770617] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:07:15.770637] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:07:15.770669] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:07:15.770681] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:07:15.770691] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:07:15.770702] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:07:15.770722] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:07:15.770737] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:07:15.770891] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:07:15.770905] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:07:13.768124] EDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=5941280][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2000008, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=8, ALLOC_BLOCK#end=17, FORMAT_END=1999976, ObMalloc#start=0, ObMalloc#end=2, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xfc0e8a4 0x993f693 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:12.871974] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2894964, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2894958, ObMalloc#start=1, ObMalloc#end=1, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:14.756375] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1014024, owner: ObLog, click_count: 3, time dist:[FORMAT_END=9, ALLOC_END=1014011, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:14.767763] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1004899, owner: ObLog, click_count: 3, time dist:[FORMAT_END=19, ALLOC_END=1004876, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:11.754306] EDIAG [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4017277, owner: ObLog, click_count: 3, time dist:[FORMAT_END=4017272, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1af3d850 0x1af3c294 0x20071394 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:13.768124] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=5941280] [PALF_DUMP](palf_id=1001, self="172.77.0.7:2882", ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [2025-08-14 09:07:12.871974] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 52.127353 loop:26707, slowcnt: 499 [2025-08-14 09:07:14.756375] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.771966] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1014024][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:14.756|207=1.02s|210=0us|222=2us|total=1.02s) [2025-08-14 09:07:15.771987] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task use too much time(end=1755162435771986, start=1755162434756375, delta=1015611) [2025-08-14 09:07:15.772001] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1016040, expected_run_ts_us_=1755162434755958, ObClockGenerator::getRealClock()=1755162435772000, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772018] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772034] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] task scheduled too delay(scheduled_delay=992241, expected_run_ts_us_=1755162434779792, ObClockGenerator::getRealClock()=1755162435772034, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72707b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a4010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772046] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72707b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a4010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772058] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=983300, expected_run_ts_us_=1755162434788758, ObClockGenerator::getRealClock()=1755162435772058, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772069] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772081] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=973208, expected_run_ts_us_=1755162434798873, ObClockGenerator::getRealClock()=1755162435772081, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739acb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196bb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.798, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772093] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739acb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196bb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.798, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772105] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=965445, expected_run_ts_us_=1755162434806660, ObClockGenerator::getRealClock()=1755162435772105, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.806, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772116] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.806, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772128] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=962221, expected_run_ts_us_=1755162434809907, ObClockGenerator::getRealClock()=1755162435772128, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.809, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772142] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.809, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772153] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=956425, expected_run_ts_us_=1755162434815728, ObClockGenerator::getRealClock()=1755162435772153, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.815, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772164] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.815, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772176] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=941234, expected_run_ts_us_=1755162434830942, ObClockGenerator::getRealClock()=1755162435772176, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772187] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e7430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772200] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=940154, expected_run_ts_us_=1755162434832045, ObClockGenerator::getRealClock()=1755162435772200, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.832, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772211] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.832, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772223] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=931031, expected_run_ts_us_=1755162434841191, ObClockGenerator::getRealClock()=1755162435772223, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.841, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772243] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.841, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772254] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=912782, expected_run_ts_us_=1755162434859472, ObClockGenerator::getRealClock()=1755162435772254, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.859, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772267] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:14.859, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772290] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73516b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196f510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.716, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772301] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.720, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772313] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.727, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772324] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.727, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772337] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72851b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.721, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772349] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.737, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772360] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fd4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772372] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772384] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891807210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772395] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d6cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772408] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e8830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:15.772419] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d67b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:14.767763] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:14.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:15.774206] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1004899][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:14.767|207=1.01s|210=1us|222=1us|total=1.01s) [2025-08-14 09:07:15.774221] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] timer task use too much time(end=1755162435774221, start=1755162434767761, delta=1006460) [2025-08-14 09:07:15.774239] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1018743, expected_run_ts_us_=1755162434755487, ObClockGenerator::getRealClock()=1755162435774231, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:14.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:14.782608] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:14.782632] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:14.840007] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:14.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.754306] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] [COMPACTION DAG STATUS] (start_time=1755162421746029, end_time=1755162431754305, dag_status={}) [2025-08-14 09:07:15.778478] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:15.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:15.782255] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=26] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:07:15.785496] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=16] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:07:15.786497] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=17] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:07:15.802638] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=20] gc stale ls task succ [2025-08-14 09:07:15.808703] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:15.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:15.842587] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=47][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:15.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:15.849618] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:15.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:15.851273] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:07:15.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:15.881697] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=25] PNIO [ratelimit] time: 1755162435881695, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007619, add_bytes: 0 [2025-08-14 09:07:15.942633] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=67][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:15.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:15.944800] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] EASY SLOW: thread: 7f9954179300, lastrun: 1755162433.759726 cost: 2.185072 loop:26787, slowcnt: 21 [2025-08-14 09:07:15.949681] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:15.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:16.030020] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:16.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:16.042698] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:15.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:16.049749] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:15.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:16.142758] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:16.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:16.149812] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:16.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:16.242820] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:16.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:16.249876] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:16.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:16.271078] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:07:16.271107] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:07:16.271122] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:07:16.271138] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:07:16.271151] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:06:37.726, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:06:07.726], idx=471, next_allow_print_ts="09:07:07.726") [2025-08-14 09:07:16.271165] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:07:16.271174] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:07:16.271186] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:07:16.271203] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:07:16.271219] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:07:16.271243] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:07:16.271264] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:07:16.271277] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:07:16.271294] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:07:16.271309] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:07:16.271326] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:07:16.271336] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:07:16.271355] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:07:16.271517] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:07:16.271533] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:07:16.271640] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:16.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:16.301827] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:16.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:16.342889] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:16.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:16.349922] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:16.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:16.371783] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=21] gc stale ls task succ [2025-08-14 09:07:16.442951] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:16.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:16.449944] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:16.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:16.523102] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:16.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:16.543013] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:16.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:16.549966] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:16.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:16.643110] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:16.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:16.650366] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:16.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:16.721096] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:16.727608] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:16.743189] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:16.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:16.743269] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:16.744358] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:16.749380] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:16.750431] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:16.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:16.752379] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:16.752793] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=19] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:16.752832] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:16.752869] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:16.757412] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:16.759663] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:16.761081] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:16.766116] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:16.771712] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:07:16.771745] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:07:16.771760] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:07:16.771776] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:07:16.771792] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:06:37.726, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:06:07.726], idx=471, next_allow_print_ts="09:07:07.726") [2025-08-14 09:07:16.771815] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:07:16.771827] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:07:16.771841] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:07:16.771853] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:07:16.771868] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:07:16.771880] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:07:16.771895] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:07:16.771907] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:07:16.771919] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:07:16.771930] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:07:16.771940] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:07:16.771952] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:07:16.771973] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:07:16.772137] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:07:16.772150] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:07:16.774939] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:16.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:16.782323] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=24] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:07:16.785601] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=20] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:07:16.786570] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=26] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:07:16.802700] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=22] gc stale ls task succ [2025-08-14 09:07:16.805112] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:16.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:16.843219] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=57][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:16.764, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:16.845399] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=61][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fc8b0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcb90, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:1, time_interval:3.00s, expected_run_time:09:07:16.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:16.850493] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=52][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:16.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:16.852823] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:07:16.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:16.889311] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=27] PNIO [ratelimit] time: 1755162436889308, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007613, add_bytes: 0 [2025-08-14 09:07:16.943292] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=50][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:16.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:16.945477] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] EASY SLOW: thread: 7f9954179300, lastrun: 1755162433.759726 cost: 3.185750 loop:26787, slowcnt: 31 [2025-08-14 09:07:16.950504] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:16.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:17.026312] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:17.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:17.043312] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:16.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:17.050581] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:16.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:17.143376] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=49][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:17.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:17.150631] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:17.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:17.243415] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:17.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:17.250699] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:17.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:17.272314] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:07:17.272346] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4016] thread maybe hung!(point=[thread id=373, timeout ts=09:02:51.767, last click point="election_event_recorder.cpp:auto oceanbase::palf::election::EventRecorder::report_event_(oceanbase::palf::election::ElectionEventType, const common::ObString &)::(anonymous class)::operator()():75", last click ts=09:02:42.745], idx=373, next_allow_print_ts="09:03:21.767") [2025-08-14 09:07:17.272362] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=465, timeout ts=09:01:25.849, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:00:55.849], idx=465, next_allow_print_ts="09:01:55.849") [2025-08-14 09:07:17.272379] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] thread maybe hung!(point=[thread id=469, timeout ts=09:00:36.821, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:37.717], idx=469, next_allow_print_ts="09:01:06.821") [2025-08-14 09:07:17.272387] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] thread maybe hung!(point=[thread id=471, timeout ts=09:06:37.726, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:06:07.726], idx=471, next_allow_print_ts="09:07:07.726") [2025-08-14 09:07:17.272395] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4016] thread maybe hung!(point=[thread id=473, timeout ts=09:02:58.753, last click point="election_impl.cpp:int oceanbase::palf::election::ElectionImpl::broadcast_(const oceanbase::palf::election::ElectionPrepareRequestMsg &, const common::ObArray &) const:362", last click ts=09:02:48.753], idx=473, next_allow_print_ts="09:03:28.753") [2025-08-14 09:07:17.272411] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] thread maybe hung!(point=[thread id=491, timeout ts=09:03:01.743, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:31.743], idx=491, next_allow_print_ts="09:03:31.743") [2025-08-14 09:07:17.272427] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=555, timeout ts=08:59:39.709, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:09.709], idx=555, next_allow_print_ts="09:00:09.709") [2025-08-14 09:07:17.272439] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=578, timeout ts=09:03:50.791, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:20.791], idx=578, next_allow_print_ts="09:04:20.791") [2025-08-14 09:07:17.272463] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=651, timeout ts=09:03:49.851, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:03:19.851], idx=651, next_allow_print_ts="09:04:19.851") [2025-08-14 09:07:17.272473] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=655, timeout ts=08:59:10.742, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::read_and_convert_to_values_(const uint64_t, const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, T &...) [T = ]:378", last click ts=08:58:10.742], idx=655, next_allow_print_ts="08:59:40.742") [2025-08-14 09:07:17.272488] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=723, timeout ts=09:03:48.735, last click point="ob_lcl_batch_sender_thread.cpp:void oceanbase::share::detector::ObLCLBatchSenderThread::record_summary_info_and_logout_when_necessary_(int64_t, int64_t, int64_t):175", last click ts=09:03:38.735], idx=723, next_allow_print_ts="09:04:18.735") [2025-08-14 09:07:17.272499] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=740, timeout ts=08:59:54.724, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=08:59:24.724], idx=740, next_allow_print_ts="09:00:24.724") [2025-08-14 09:07:17.272511] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=767, timeout ts=09:00:57.755, last click point="ob_table_access_helper.h:static int oceanbase::common::ObTableAccessHelper::get_my_sql_result_(const char **, const int64_t, const oceanbase::common::ObString &, const oceanbase::common::ObString &, oceanbase::common::ObISQLClient &, const uint64_t, ObMySQLProxy::MySQLResult &, common::sqlclient::ObMySQLResult *&):500", last click ts=08:59:58.745], idx=767, next_allow_print_ts="09:01:27.755") [2025-08-14 09:07:17.272522] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=769, timeout ts=09:01:47.848, last click point="ob_failure_detector.cpp:void oceanbase::logservice::coordinator::ObFailureDetector::detect_data_disk_full_():477", last click ts=09:00:47.848], idx=769, next_allow_print_ts="09:02:17.848") [2025-08-14 09:07:17.272541] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4016] thread maybe hung!(point=[thread id=771, timeout ts=09:02:44.758, last click point="election_proposer.cpp:void oceanbase::palf::election::ElectionProposer::prepare(const oceanbase::common::ObRole):377", last click ts=09:02:35.790], idx=771, next_allow_print_ts="09:03:14.758") [2025-08-14 09:07:17.272553] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=791, timeout ts=09:01:48.580, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:01:18.580], idx=791, next_allow_print_ts="09:02:18.580") [2025-08-14 09:07:17.272568] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=868, timeout ts=09:02:58.749, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:02:28.749], idx=868, next_allow_print_ts="09:03:28.749") [2025-08-14 09:07:17.272726] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4016] thread maybe hung!(point=[thread id=15387, timeout ts=08:57:24.779, last click point="log_request_handler.cpp:int oceanbase::palf::LogRequestHandler::handle_request(const int64_t, const oceanbase::common::ObAddr &, const oceanbase::palf::election::ElectionAcceptRequestMsg &):457", last click ts=08:57:14.779], idx=3099, next_allow_print_ts="08:57:54.779") [2025-08-14 09:07:17.272739] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=15425, timeout ts=08:59:31.752, last click point="log_rpc_processor.h:int oceanbase::palf::LogRegisterParentReqP::process_impl_() [FLAG = false]:103", last click ts=08:59:21.752], idx=3137, next_allow_print_ts="09:00:01.752") [2025-08-14 09:07:17.277750] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:17.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:17.307896] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:17.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:17.343450] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:17.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:17.350768] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:17.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:17.371818] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=22] gc stale ls task succ [2025-08-14 09:07:17.443506] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:17.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:17.450832] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:17.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:17.529143] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:17.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:17.543566] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:17.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:17.550871] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:17.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:17.643633] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:17.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:17.650932] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:17.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:17.720983] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:15.774254] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:14.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:15.774254] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1944298, owner: ObLog, click_count: 3, time dist:[FORMAT_END=8, ALLOC_END=1944283, APPEND_END=6], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:15.775153] EDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=991021][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1944536, owner: ObLog, click_count: 3, time dist:[FORMAT_END=7, ALLOC_END=1944528, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:15.775248] EDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=933825][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1944957, owner: ObLog, click_count: 3, time dist:[FORMAT_END=11, ALLOC_END=1944946, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ed35c 0x97aead0 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:15.792702] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1950081, owner: ObLog, click_count: 3, time dist:[FORMAT_END=23, ALLOC_END=1950057, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:17.721531] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1944298][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:15.774|207=1.95s|210=0us|222=2us|total=1.95s) [2025-08-14 09:07:17.721557] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4389] timer task use too much time(end=1755162437721557, start=1755162435774230, delta=1947327) [2025-08-14 09:07:17.721573] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=2632254, expected_run_ts_us_=1755162435089316, ObClockGenerator::getRealClock()=1755162437721571, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:15.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:17.721588] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:15.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:17.721600] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=2517303, expected_run_ts_us_=1755162435204297, ObClockGenerator::getRealClock()=1755162437721600, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:15.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:17.721609] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:15.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:17.721629] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1712494, expected_run_ts_us_=1755162436009134, ObClockGenerator::getRealClock()=1755162437721628, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:16.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:17.721638] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:16.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:15.775137] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=991027][errcode=0] epoll handler too much time :992537 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:15.775153] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=991021][errcode=0] epoll handler too much time :992540 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:17.722668] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1944536] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:17.722681] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:15.775248] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=933825][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:14.840|207=935.24ms|210=0us|222=2us|total=935.25ms) [2025-08-14 09:07:17.723167] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1944957][errcode=-4389] timer task use too much time(end=1755162437723167, start=1755162434840004, delta=2883163) [2025-08-14 09:07:17.723179] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1929243, expected_run_ts_us_=1755162435793934, ObClockGenerator::getRealClock()=1755162437723178, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:15.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:17.723206] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:15.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:15.792702] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891807090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.745770] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1950081][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:15.792|207=1.95s|210=1us|222=2us|total=1.95s) [474]OB_MALLOC COST TOO MUCH TIME, cost_time=5000757, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=5000749], seq:[0] [2025-08-14 09:07:17.745791] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4389] timer task use too much time(end=1755162437745790, start=1755162435792699, delta=1953091) [2025-08-14 09:07:17.745801] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1958631, expected_run_ts_us_=1755162435787169, ObClockGenerator::getRealClock()=1755162437745800, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739aa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.745816] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739aa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.745826] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1954220, expected_run_ts_us_=1755162435791606, ObClockGenerator::getRealClock()=1755162437745826, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.791, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.745835] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.791, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.745845] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1954471, expected_run_ts_us_=1755162435791373, ObClockGenerator::getRealClock()=1755162437745845, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ecf10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.791, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.745853] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ecf10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.791, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.745862] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1953837, expected_run_ts_us_=1755162435792025, ObClockGenerator::getRealClock()=1755162437745862, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.745870] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fc5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.745879] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1946550, expected_run_ts_us_=1755162435799328, ObClockGenerator::getRealClock()=1755162437745878, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.799, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.745887] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.799, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.745896] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1944651, expected_run_ts_us_=1755162435801244, ObClockGenerator::getRealClock()=1755162437745896, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.745904] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e6a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.745912] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1942146, expected_run_ts_us_=1755162435803766, ObClockGenerator::getRealClock()=1755162437745912, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7351e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196f990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.803, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.745920] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7351e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196f990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.803, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.745929] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1941861, expected_run_ts_us_=1755162435804067, ObClockGenerator::getRealClock()=1755162437745929, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a22b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.745937] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a22b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ed510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.745945] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1938676, expected_run_ts_us_=1755162435807269, ObClockGenerator::getRealClock()=1755162437745945, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.745953] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.745962] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1919038, expected_run_ts_us_=1755162435826923, ObClockGenerator::getRealClock()=1755162437745962, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ebe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.826, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.745970] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ebe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.826, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.745979] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1921222, expected_run_ts_us_=1755162435824757, ObClockGenerator::getRealClock()=1755162437745979, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7350530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ea90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.824, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.745987] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7350530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196ea90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.824, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.745996] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1880001, expected_run_ts_us_=1755162435865994, ObClockGenerator::getRealClock()=1755162437745996, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fbbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.746011] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fbbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.746020] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1872539, expected_run_ts_us_=1755162435873480, ObClockGenerator::getRealClock()=1755162437746020, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72860b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.873, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.746028] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72860b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:15.873, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.746043] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1416841, expected_run_ts_us_=1755162436329201, ObClockGenerator::getRealClock()=1755162437746042, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7faa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.746051] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7faa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.746059] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1416318, expected_run_ts_us_=1755162436329741, ObClockGenerator::getRealClock()=1755162437746059, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72842b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.746067] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72842b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.746080] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1131748, expected_run_ts_us_=1755162436614331, ObClockGenerator::getRealClock()=1755162437746079, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eef90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.614, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.746088] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eef90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.614, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.746096] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1131276, expected_run_ts_us_=1755162436614819, ObClockGenerator::getRealClock()=1755162437746096, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eaf30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.614, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.746104] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eaf30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.614, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.746114] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=996748, expected_run_ts_us_=1755162436749366, ObClockGenerator::getRealClock()=1755162437746114, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.746123] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.746131] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1004880, expected_run_ts_us_=1755162436741251, ObClockGenerator::getRealClock()=1755162437746131, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.746139] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.746147] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=996197, expected_run_ts_us_=1755162436749950, ObClockGenerator::getRealClock()=1755162437746147, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919827b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.746155] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919827b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.746165] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=989803, expected_run_ts_us_=1755162436756361, ObClockGenerator::getRealClock()=1755162437746164, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.746173] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.746181] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=984212, expected_run_ts_us_=1755162436761969, ObClockGenerator::getRealClock()=1755162437746181, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72742b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.746190] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72742b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.746198] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=980937, expected_run_ts_us_=1755162436765261, ObClockGenerator::getRealClock()=1755162437746198, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919840b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.746206] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919840b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.746215] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=984588, expected_run_ts_us_=1755162436761626, ObClockGenerator::getRealClock()=1755162437746215, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180ef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.746223] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180ef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.749445] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=984449, expected_run_ts_us_=1755162436761782, ObClockGenerator::getRealClock()=1755162437749444, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180ecb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.749476] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180ecb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.749488] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=982343, expected_run_ts_us_=1755162436767145, ObClockGenerator::getRealClock()=1755162437749488, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.767, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.749497] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.767, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.749507] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=987545, expected_run_ts_us_=1755162436761961, ObClockGenerator::getRealClock()=1755162437749506, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.749516] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.749525] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=979406, expected_run_ts_us_=1755162436770119, ObClockGenerator::getRealClock()=1755162437749525, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919925b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196d890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.749534] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919925b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196d890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.749543] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=971129, expected_run_ts_us_=1755162436778414, ObClockGenerator::getRealClock()=1755162437749543, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.749551] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.749560] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=972497, expected_run_ts_us_=1755162436777062, ObClockGenerator::getRealClock()=1755162437749560, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.777, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.749568] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.777, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.749577] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=961487, expected_run_ts_us_=1755162436788089, ObClockGenerator::getRealClock()=1755162437749576, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728ca30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7291590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.749585] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728ca30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7291590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.749593] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=954081, expected_run_ts_us_=1755162436795512, ObClockGenerator::getRealClock()=1755162437749593, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739ee10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.749601] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739ee10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.795, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.749610] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=944945, expected_run_ts_us_=1755162436804664, ObClockGenerator::getRealClock()=1755162437749610, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ebd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.749618] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d3930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ebd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.749626] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=944914, expected_run_ts_us_=1755162436804712, ObClockGenerator::getRealClock()=1755162437749626, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739ec90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.762515] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:17.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:15.775137] EDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=991027][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1944404, owner: ObLog, click_count: 3, time dist:[FORMAT_END=8, ALLOC_END=1944395, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:16.733574] EDIAG [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=977648][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3029591, owner: ObLog, click_count: 5, time dist:[FORMAT_END=3029586, ObMalloc#start=1, ObMalloc#end=3, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1f9ebc06 0x97c9052 0x1f9819be 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [404]OB_MALLOC COST TOO MUCH TIME, cost_time=760865, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=760863], seq:[0] [474]OB_MALLOC COST TOO MUCH TIME, cost_time=5973436, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=5965433], seq:[0, 1] [2025-08-14 09:07:14.756492] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5973486, owner: ObLog, click_count: 5, time dist:[FORMAT_END=31, ObMalloc#start=0, ObMalloc#end=5973453, ALLOC_END=2, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:16.345071] EDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=3158][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 4404860, owner: ObLog, click_count: 3, time dist:[FORMAT_END=4404859, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x9757ced 0x97569df 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:16.775587] EDIAG [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [726][T1001_ObXAHbWor][T1001][Y0-0000000000000000-0-0] [lt=1002103][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3975378, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=8, ALLOC_BLOCK#end=4, FORMAT_END=3975365, ALLOC_END=0, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b64d8ba 0x1b64d154 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [404]OB_MALLOC COST TOO MUCH TIME, cost_time=1740218, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=760883], seq:[0, 1] [2025-08-14 09:07:19.974851] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=2894964][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1740266, owner: ObLog, click_count: 5, time dist:[FORMAT_END=16, ObMalloc#start=1, ObMalloc#end=1740247, ALLOC_END=2, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [836]OB_MALLOC COST TOO MUCH TIME, cost_time=991760, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=991748], seq:[0] [277]OB_MALLOC COST TOO MUCH TIME, cost_time=1951135, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1951134], seq:[0] [2025-08-14 09:07:18.735112] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=971120][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:17.762|207=972.59ms|210=0us|222=2us|total=972.63ms) [2025-08-14 09:07:18.735150] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4389] timer task use too much time(end=1755162438735148, start=1755162437762512, delta=972636) [2025-08-14 09:07:18.735161] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=976025, expected_run_ts_us_=1755162437759134, ObClockGenerator::getRealClock()=1755162438735160, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:17.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:17.749635] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739ec90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.823245] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:17.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:18.723340] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:18.735186] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:17.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:19.733023] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=996323][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:18.735|207=997.86ms|210=0us|222=2us|total=997.89ms) [2025-08-14 09:07:19.733054] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4389] timer task use too much time(end=1755162439733052, start=1755162438735159, delta=997893) [2025-08-14 09:07:19.733070] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1643752, expected_run_ts_us_=1755162438089316, ObClockGenerator::getRealClock()=1755162439733069, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:18.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:19.733090] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:18.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:18.738285] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=987151][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:17.749|207=988.65ms|210=1us|222=1us|total=988.68ms) [2025-08-14 09:07:18.771756] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=947066][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:17.823|207=948.50ms|210=0us|222=3us|total=948.54ms) [2025-08-14 09:07:18.771860] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=48443][errcode=0] epoll handler too much time :48529 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:19.735664] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=962450][errcode=-4389] timer task use too much time(end=1755162439735663, start=1755162437823243, delta=1912420) [2025-08-14 09:07:19.755294] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=19594][errcode=0] task scheduled too delay(scheduled_delay=961357, expected_run_ts_us_=1755162438793934, ObClockGenerator::getRealClock()=1755162439755292, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:18.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:19.755327] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:18.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:19.734229] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=994439][errcode=-4389] timer task use too much time(end=1755162439734229, start=1755162437749626, delta=1984603) [2025-08-14 09:07:19.735709] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=962370] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:19.733103] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1528805, expected_run_ts_us_=1755162438204297, ObClockGenerator::getRealClock()=1755162439733103, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:18.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:19.755362] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21099][errcode=0] task scheduled too delay(scheduled_delay=2938594, expected_run_ts_us_=1755162436816767, ObClockGenerator::getRealClock()=1755162439755362, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72874b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766526] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11144][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72874b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766549] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] task scheduled too delay(scheduled_delay=2943964, expected_run_ts_us_=1755162436822584, ObClockGenerator::getRealClock()=1755162439766549, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766560] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766569] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=2943760, expected_run_ts_us_=1755162436822809, ObClockGenerator::getRealClock()=1755162439766569, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739dc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766579] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739dc30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766589] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=2871415, expected_run_ts_us_=1755162436895174, ObClockGenerator::getRealClock()=1755162439766589, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a59b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.895, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766598] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a59b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:16.895, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766619] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=2040189, expected_run_ts_us_=1755162437726429, ObClockGenerator::getRealClock()=1755162439766618, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.726, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766627] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971a10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.726, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766638] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=2045488, expected_run_ts_us_=1755162437721150, ObClockGenerator::getRealClock()=1755162439766638, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739eb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.721, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766646] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739eb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.721, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766655] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=2029663, expected_run_ts_us_=1755162437736992, ObClockGenerator::getRealClock()=1755162439766655, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766663] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766672] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=2016230, expected_run_ts_us_=1755162437750442, ObClockGenerator::getRealClock()=1755162439766672, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.750, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766682] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.750, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766691] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=2011322, expected_run_ts_us_=1755162437755368, ObClockGenerator::getRealClock()=1755162439766691, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766699] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766708] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=2011974, expected_run_ts_us_=1755162437754733, ObClockGenerator::getRealClock()=1755162439766707, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d80b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766716] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d80b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766726] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=2011848, expected_run_ts_us_=1755162437754878, ObClockGenerator::getRealClock()=1755162439766726, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766734] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766743] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=2008646, expected_run_ts_us_=1755162437758096, ObClockGenerator::getRealClock()=1755162439766743, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891992330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196d710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766751] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891992330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196d710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766760] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=2014095, expected_run_ts_us_=1755162437752664, ObClockGenerator::getRealClock()=1755162439766759, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766769] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766778] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=2005688, expected_run_ts_us_=1755162437761089, ObClockGenerator::getRealClock()=1755162439766778, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766786] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983e30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766795] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=2006773, expected_run_ts_us_=1755162437760021, ObClockGenerator::getRealClock()=1755162439766795, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766805] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766813] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1994945, expected_run_ts_us_=1755162437771868, ObClockGenerator::getRealClock()=1755162439766813, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766821] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766830] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1990793, expected_run_ts_us_=1755162437776036, ObClockGenerator::getRealClock()=1755162439766830, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72751b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766838] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72751b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766848] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1990711, expected_run_ts_us_=1755162437776136, ObClockGenerator::getRealClock()=1755162439766848, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d31b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766856] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d31b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.776, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766864] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1992915, expected_run_ts_us_=1755162437773949, ObClockGenerator::getRealClock()=1755162439766864, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73959b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.773, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766872] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73959b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.773, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766888] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] task scheduled too delay(scheduled_delay=1979554, expected_run_ts_us_=1755162437787334, ObClockGenerator::getRealClock()=1755162439766888, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766905] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a4830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766913] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1979409, expected_run_ts_us_=1755162437787504, ObClockGenerator::getRealClock()=1755162439766913, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72856b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766922] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72856b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766930] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1978953, expected_run_ts_us_=1755162437787976, ObClockGenerator::getRealClock()=1755162439766930, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766938] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efb90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.787, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766948] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1976787, expected_run_ts_us_=1755162437790161, ObClockGenerator::getRealClock()=1755162439766948, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919920b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196d590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.790, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766956] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919920b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196d590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.790, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766965] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1926881, expected_run_ts_us_=1755162437840084, ObClockGenerator::getRealClock()=1755162439766965, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.840, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766973] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.840, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766982] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1920858, expected_run_ts_us_=1755162437846124, ObClockGenerator::getRealClock()=1755162439766982, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d54b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ecd90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.846, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.766993] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d54b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ecd90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.846, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767001] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1918948, expected_run_ts_us_=1755162437848053, ObClockGenerator::getRealClock()=1755162439767001, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ecc10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.848, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767009] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d5230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ecc10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:17.848, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767029] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1044852, expected_run_ts_us_=1755162438722177, ObClockGenerator::getRealClock()=1755162439767029, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.722, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767044] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.722, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767054] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1036173, expected_run_ts_us_=1755162438730881, ObClockGenerator::getRealClock()=1755162439767054, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767062] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab410, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767071] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1035544, expected_run_ts_us_=1755162438731526, ObClockGenerator::getRealClock()=1755162439767071, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891826cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891918190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767079] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891826cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891918190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767087] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1019749, expected_run_ts_us_=1755162438747338, ObClockGenerator::getRealClock()=1755162439767087, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767097] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.747, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767105] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1021012, expected_run_ts_us_=1755162438746092, ObClockGenerator::getRealClock()=1755162439767105, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919845b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767119] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919845b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.746, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767127] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1012609, expected_run_ts_us_=1755162438754518, ObClockGenerator::getRealClock()=1755162439767127, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767136] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767145] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1012422, expected_run_ts_us_=1755162438754723, ObClockGenerator::getRealClock()=1755162439767145, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767154] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ea090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767162] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1011759, expected_run_ts_us_=1755162438755403, ObClockGenerator::getRealClock()=1755162439767162, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767170] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.755, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767179] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=1007802, expected_run_ts_us_=1755162438759376, ObClockGenerator::getRealClock()=1755162439767178, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767188] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728a910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.759, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767196] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1009610, expected_run_ts_us_=1755162438757586, ObClockGenerator::getRealClock()=1755162439767196, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eacb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767204] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eacb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.757, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767213] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=999222, expected_run_ts_us_=1755162438767990, ObClockGenerator::getRealClock()=1755162439767213, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.767, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767221] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.767, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.755381] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19639][errcode=0] epoll handler too much time :19684 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:19.755392] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=22254][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:18.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:19.767619] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12198][errcode=0] task scheduled too delay(scheduled_delay=1047064, expected_run_ts_us_=1755162438720554, ObClockGenerator::getRealClock()=1755162439767619, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc190, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc470, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:10.00s, expected_run_time:09:07:18.720, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:19.767634] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc190, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:301, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_92, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc470, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:10.00s, expected_run_time:09:07:18.720, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:19.767645] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1012157, expected_run_ts_us_=1755162438755487, ObClockGenerator::getRealClock()=1755162439767645, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:18.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:19.767656] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:18.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:19.767681] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:19.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:16.733574] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=977648] [IO STATUS CALLBACK](tenant_id=1, io_status=runner[0]: tid=515, queue_count=0) [2025-08-14 09:07:19.767801] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=987831, expected_run_ts_us_=1755162438779400, ObClockGenerator::getRealClock()=1755162439767801, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767811] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767821] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=989488, expected_run_ts_us_=1755162438778332, ObClockGenerator::getRealClock()=1755162439767821, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767829] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767838] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=989256, expected_run_ts_us_=1755162438778581, ObClockGenerator::getRealClock()=1755162439767837, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767848] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.778, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767856] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=997702, expected_run_ts_us_=1755162438770154, ObClockGenerator::getRealClock()=1755162439767856, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767864] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767873] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=984221, expected_run_ts_us_=1755162438783651, ObClockGenerator::getRealClock()=1755162439767873, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72716b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a4910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767881] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72716b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a4910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767898] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] task scheduled too delay(scheduled_delay=979871, expected_run_ts_us_=1755162438788027, ObClockGenerator::getRealClock()=1755162439767898, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767911] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767920] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=979652, expected_run_ts_us_=1755162438788267, ObClockGenerator::getRealClock()=1755162439767920, ret=0, ret="OB_SUCCESS", *this={this:0x7f98918100b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767928] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918100b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.788, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767936] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=986724, expected_run_ts_us_=1755162438781212, ObClockGenerator::getRealClock()=1755162439767936, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767946] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7395730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.781, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767955] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=974160, expected_run_ts_us_=1755162438793794, ObClockGenerator::getRealClock()=1755162439767955, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.793, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767963] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.793, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767971] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=975571, expected_run_ts_us_=1755162438792400, ObClockGenerator::getRealClock()=1755162439767971, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180fe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767979] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180fe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.792, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767991] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=962243, expected_run_ts_us_=1755162438805747, ObClockGenerator::getRealClock()=1755162439767990, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.805, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.767999] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7392a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.805, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.768007] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=955452, expected_run_ts_us_=1755162438812555, ObClockGenerator::getRealClock()=1755162439768007, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73927b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.812, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.768015] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73927b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.812, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.768024] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=952558, expected_run_ts_us_=1755162438815466, ObClockGenerator::getRealClock()=1755162439768024, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7399090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.815, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.768034] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a99b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7399090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.815, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.768042] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=951744, expected_run_ts_us_=1755162438816298, ObClockGenerator::getRealClock()=1755162439768042, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.768051] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398f10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.816, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.768059] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=940730, expected_run_ts_us_=1755162438827329, ObClockGenerator::getRealClock()=1755162439768059, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73954b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.827, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.768069] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73954b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.827, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.768078] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=894393, expected_run_ts_us_=1755162438873684, ObClockGenerator::getRealClock()=1755162439768078, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.873, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.768086] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:18.873, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.768099] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=519926, expected_run_ts_us_=1755162439248172, ObClockGenerator::getRealClock()=1755162439768099, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.248, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.768107] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.248, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.768124] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7270030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919a3b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.735, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.768132] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ecb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.739, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.768141] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7353730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7396990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.768149] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ec830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7efd10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.736, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.768157] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199ea30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.768166] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919afbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.754, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.768177] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919854b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.768185] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891985230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891972010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.768193] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891984fb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.758, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.768202] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d45b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.752, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.848898] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fcc90, caller:mds_tenant_service.cpp:mtl_start:117, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_145, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcf70, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:19.848931] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fc8b0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcb90, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:1, time_interval:3.00s, expected_run_time:09:07:19.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:19.776741] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72747b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:20.724553] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=946373][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:19.776|207=947.81ms|210=0us|222=2us|total=947.85ms) [2025-08-14 09:07:20.724591] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4389] timer task use too much time(end=1755162440724590, start=1755162439776739, delta=947851) [2025-08-14 09:07:19.855357] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:19.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:20.019180] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:20.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:20.724331] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11963] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:20.735451] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=715182][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:20.191|207=716.27ms|210=0us|222=2us|total=716.32ms) [2025-08-14 09:07:20.736929] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1475][errcode=-4389] timer task use too much time(end=1755162440736929, start=1755162440019178, delta=717751) [2025-08-14 09:07:20.736942] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=727807, expected_run_ts_us_=1755162440009134, ObClockGenerator::getRealClock()=1755162440736942, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:20.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:20.736968] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:20.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:20.736983] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=647666, expected_run_ts_us_=1755162440089316, ObClockGenerator::getRealClock()=1755162440736983, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:20.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:20.724606] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=958195, expected_run_ts_us_=1755162439766408, ObClockGenerator::getRealClock()=1755162440724604, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:14.756492] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:14.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:20.739042] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=5973486][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:14.756|207=5.98s|210=0us|222=2us|total=5.98s) [2025-08-14 09:07:20.739060] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4389] timer task use too much time(end=1755162440739060, start=1755162434756491, delta=5982569) [2025-08-14 09:07:20.739073] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=5755541, expected_run_ts_us_=1755162434983531, ObClockGenerator::getRealClock()=1755162440739073, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:14.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:20.736714] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1944404] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:16.345071] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=3158][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a7058, timer:0x7f994b1dfb70, task:0x7f98ebbf7c30, task_type:"N9oceanbase8observer26ObTableLoadResourceManager21ObRefreshAndCheckTaskE", scheduled_time:1755161774744853, pushed_time:0, delay:30000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=661600216, threashold=1000000, tenant_id=1002) [2025-08-14 09:07:20.737243] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12616][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:20.756656] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19377][errcode=0] task scheduled too delay(scheduled_delay=981035, expected_run_ts_us_=1755162439775620, ObClockGenerator::getRealClock()=1755162440756655, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:20.756689] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fcab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:20.736911] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12559][errcode=0] epoll handler too much time :12595 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:20.736993] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:20.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:16.775587] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [726][T1001_ObXAHbWor][T1001][Y0-0000000000000000-0-0] [lt=1002103] XA scheduler heartbeat task statistics(avg_time=139850) [2025-08-14 09:07:20.739287] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2555] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:20.756701] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=981045, expected_run_ts_us_=1755162439775655, ObClockGenerator::getRealClock()=1755162440756701, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:21.716754] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=958598][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.775, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:21.716807] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=49][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:20.756|207=960.10ms|210=0us|222=2us|total=960.12ms) [2025-08-14 09:07:21.716823] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4389] timer task use too much time(end=1755162441716822, start=1755162440756700, delta=960122) [2025-08-14 09:07:21.716834] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1944932, expected_run_ts_us_=1755162439771901, ObClockGenerator::getRealClock()=1755162441716834, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:19.974851] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=2894964][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 59.230230 loop:26707, slowcnt: 541 [2025-08-14 09:07:20.756929] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=19899][errcode=0] task scheduled too delay(scheduled_delay=552630, expected_run_ts_us_=1755162440204297, ObClockGenerator::getRealClock()=1755162440756928, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:20.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:20.739088] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:14.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:20.763986] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24659][errcode=0] epoll handler too much time :24704 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:21.717786] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=959404][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:20.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:20.724630] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=867956][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:19.855|207=869.27ms|210=0us|222=1us|total=1.87s) [2025-08-14 09:07:21.722590] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=4792][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:20.756|207=965.66ms|210=1us|222=1us|total=965.69ms) [2025-08-14 09:07:21.722620] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4389] timer task use too much time(end=1755162441722619, start=1755162440756927, delta=965692) [2025-08-14 09:07:21.722622] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=996483][errcode=-4389] timer task use too much time(end=1755162441722621, start=1755162439855354, delta=1867267) [2025-08-14 09:07:21.722638] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=0] task scheduled too delay(scheduled_delay=967149, expected_run_ts_us_=1755162440755487, ObClockGenerator::getRealClock()=1755162441722637, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:20.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:21.716846] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d7930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.771, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:13.725643] EDIAG [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=62][errcode=0] LOGGER COST TOO MUCH TIME, cost: 8012690, owner: ObLog, click_count: 3, time dist:[FORMAT_END=8012686, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1bdd73e2 0x1bdd6e36 0x986bc6f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:18.734099] EDIAG [RPC] handle_resp (ob_poc_rpc_proxy.cpp:192) [319][pnio1][T0][Y0-0000000000000000-0-0] [lt=990177][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3011997, owner: ObLog, click_count: 5, time dist:[FORMAT_END=3011990, ObMalloc#start=1, ObMalloc#end=2, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x27a861af 0x97bcba5 0x97bc4ca 0x27ad7f11 0x97b81a7 0x27a8a6d6 0x97b7ba0 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:13.725519] EDIAG [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=100][errcode=0] LOGGER COST TOO MUCH TIME, cost: 8012749, owner: ObLog, click_count: 5, time dist:[FORMAT_END=8012735, ObMalloc#start=2, ObMalloc#end=5, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1bdd73e2 0x1bdd6e36 0x986bc6f 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [561]OB_MALLOC COST TOO MUCH TIME, cost_time=5969697, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=4952859, ALLOC_CHUNK_END=1016829], seq:[0, 1] [2025-08-14 09:07:19.903983] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1847176, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1847170, ALLOC_END=1, APPEND_END=5], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:13.725519] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [737][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=100] finish update min start scn(max_decided_scn={val:1755161719775179002, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1755161287778691002, v:0}, keep_alive_scn_:{val:1755161287778691002, v:0}, update_ts_:1755162433724900}) [2025-08-14 09:07:13.725643] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1062) [575][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=62] finish update min start scn(max_decided_scn={val:1755161547738516001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1755161493768447002, v:0}, keep_alive_scn_:{val:1755161493768447002, v:0}, update_ts_:1755162433724900}) [2025-08-14 09:07:14.785149] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [411][sql_nio4][T0][Y0-0000000000000000-0-0] [lt=1993726] [sql nio session](*s={this:0x7f98f72c3030, session_id:3221590336, trace_id:YB42AC4D0007-00063C49634A0941-0-0, sql_handling_stage:3, sql_initiative_shutdown:false, reader:{fd:255}, err:5, last_decode_time:1755162354747599, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1422, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:07:14.779960] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [412][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=1985344] [sql nio session](*s={this:0x7f98f733a030, session_id:3221590255, trace_id:YB42AC4D0007-00063C49651A3232-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:257}, err:0, last_decode_time:1755161952736711, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1716, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:07:18.734099] WDIAG [RPC] handle_resp (ob_poc_rpc_proxy.cpp:192) [319][pnio1][T0][Y0-0000000000000000-0-0] [lt=990177][errcode=0] async_cb handler cost too much time(total_time=9946813, decode_time=-1755162426773272, process_time=0, session_destroy_time=1755162436720085, ret=0, pcode=1815) [2025-08-14 09:07:19.903983] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=23] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:07:21.722654] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:20.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:21.753998] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=31287][errcode=0] task scheduled too delay(scheduled_delay=994860, expected_run_ts_us_=1755162440759134, ObClockGenerator::getRealClock()=1755162441753996, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:20.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:21.736846] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=956325] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:21.722761] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=5901][errcode=0] task scheduled too delay(scheduled_delay=1949789, expected_run_ts_us_=1755162439772971, ObClockGenerator::getRealClock()=1755162441722760, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:21.722644] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=0] task scheduled too delay(scheduled_delay=928707, expected_run_ts_us_=1755162440793934, ObClockGenerator::getRealClock()=1755162441722642, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:20.793, task_priority:1, with_handle_protected:1}) [277]OB_MALLOC COST TOO MUCH TIME, cost_time=1989788, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1951151], seq:[0, 1] [2025-08-14 09:07:19.919116] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1839581, owner: ObLog, click_count: 2, time dist:[FORMAT_END=1834040, FREE_END=5541], seq:[0, 1], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:20.735506] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:288) [884][T1002_ArcFetche][T1002][YB42AC4D0007-00063C495ED99C16-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1024487, owner: ObLog, click_count: 2, time dist:[FORMAT_END=21336, FREE_END=1003151], seq:[0, 1], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113b2889 0x979b621 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:17.490623] EDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 4265027, owner: ObLog, click_count: 2, time dist:[FORMAT_END=3261405, FREE_END=1003622], seq:[0, 1], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x9757ced 0x97569df 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [400]OB_MALLOC COST TOO MUCH TIME, cost_time=3812220, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=48, WASH_BLOCK_END=3812172], seq:[0, 1, 2] [686]OB_MALLOC COST TOO MUCH TIME, cost_time=5977550, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=35, WASH_BLOCK_END=5977515], seq:[0, 1, 2] [376]OB_MALLOC COST TOO MUCH TIME, cost_time=3028276, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=49, WASH_BLOCK_END=3028227], seq:[0, 1, 2] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=5981254, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=33, WASH_BLOCK_END=5981221], seq:[0, 1, 2] [386]OB_MALLOC COST TOO MUCH TIME, cost_time=3039998, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=49, WASH_BLOCK_END=3039949], seq:[0, 1, 2] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=4015329, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=48, WASH_BLOCK_END=4015281], seq:[0, 1, 2] [402]OB_MALLOC COST TOO MUCH TIME, cost_time=2787873, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=43, WASH_BLOCK_END=2787830], seq:[0, 1, 2] [2025-08-14 09:07:14.785102] EDIAG [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] LOGGER COST TOO MUCH TIME, cost: 6965942, owner: ObLog, click_count: 2, time dist:[FORMAT_END=6956174, FREE_END=9768], seq:[0, 1], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a5bbf2 0x94da542 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [394]OB_MALLOC COST TOO MUCH TIME, cost_time=5980488, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=36, WASH_BLOCK_END=5974371, ALLOC_BLOCK#start=6044, ALLOC_BLOCK#end=37], seq:[0, 1, 2, 3, 4] [2025-08-14 09:07:14.785149] EDIAG [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [411][sql_nio4][T0][Y0-0000000000000000-0-0] [lt=1993726][errcode=0] LOGGER COST TOO MUCH TIME, cost: 6954911, owner: ObLog, click_count: 5, time dist:[FORMAT_END=6954901, ObMalloc#start=1, ObMalloc#end=5, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a5bbf2 0x94da542 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:20.737279] EDIAG [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1029125, owner: ObLog, click_count: 2, time dist:[FORMAT_END=1021688, FREE_END=7437], seq:[0, 1], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x12377491 0x1236e8e9 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:20.758420] EDIAG [RS.COMPACTION] try_idle (ob_major_merge_scheduler.cpp:226) [1057][T1_MergeSchedul][T1][YB42AC4D0007-00063C4961599CC3-0-0] [lt=28945][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1008046, owner: ObLog, click_count: 2, time dist:[FORMAT_END=1007968, FREE_END=78], seq:[0, 1], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1125d502 0x1125cf81 0x1125a4c3 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:14.779960] EDIAG [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [412][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=1985344][errcode=0] LOGGER COST TOO MUCH TIME, cost: 6960149, owner: ObLog, click_count: 3, time dist:[FORMAT_END=6960144, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a5bbf2 0x94da542 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:10.786570] EDIAG [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 10958311, owner: ObLog, click_count: 5, time dist:[FORMAT_END=8968498, ObMalloc#start=1, ObMalloc#end=1989806, ALLOC_END=5, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x989c577 0x989bc96 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [385]OB_MALLOC COST TOO MUCH TIME, cost_time=4017242, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=34, WASH_BLOCK_END=4017208], seq:[0, 1, 2] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=4017406, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=56, WASH_BLOCK_END=4017350], seq:[0, 1, 2] [2025-08-14 09:07:17.751816] EDIAG [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4017443, owner: ObLog, click_count: 5, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=4017414, ALLOC_END=19, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a1f867 0x27a135c3 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [807]OB_MALLOC COST TOO MUCH TIME, cost_time=3036249, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=35, WASH_BLOCK_END=3036212], seq:[0, 1, 2] [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=3036198, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=31, WASH_BLOCK_END=3036167], seq:[0, 1, 2] [733]OB_MALLOC COST TOO MUCH TIME, cost_time=2038078, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=35, WASH_BLOCK_END=2038042], seq:[0, 1, 2] [2025-08-14 09:07:16.772819] EDIAG [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=1006721][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4995070, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1958816, ObMalloc#start=0, ObMalloc#end=3036230, ALLOC_END=24, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1f832f91 0x97c36fe 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [393]OB_MALLOC COST TOO MUCH TIME, cost_time=4017333, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=33, WASH_BLOCK_END=4017300], seq:[0, 1, 2] [2025-08-14 09:07:17.751989] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4017493, owner: ObLog, click_count: 5, time dist:[FORMAT_END=5, ObMalloc#start=0, ObMalloc#end=4017468, ALLOC_END=19, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [660]OB_MALLOC COST TOO MUCH TIME, cost_time=3995478, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=38, WASH_BLOCK_END=3995440], seq:[0, 1, 2] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=4020298, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=48, WASH_BLOCK_END=4020250], seq:[0, 1, 2] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=4024219, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=33, WASH_BLOCK_END=4024186], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=4025877, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=33, WASH_BLOCK_END=4025843], seq:[0, 1, 2] [2025-08-14 09:07:15.772460] EDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4038] LOGGER COST TOO MUCH TIME, cost: 5980106, owner: ObLog, click_count: 5, time dist:[FORMAT_END=4955614, ObMalloc#start=2, ObMalloc#end=117, ALLOC_END=1024372, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1b4fd0db 0x1b4fc327 0x1b5014e0 0x1b5009d1 0x1b50387c 0x1b8905de 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [321]OB_MALLOC COST TOO MUCH TIME, cost_time=4818724, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=43, WASH_BLOCK_END=3863048, ALLOC_BLOCK#start=955482, ALLOC_BLOCK#end=151], seq:[0, 1, 2, 3, 4] [2025-08-14 09:07:15.761694] EDIAG [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=135][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5989747, owner: ObLog, click_count: 3, time dist:[FORMAT_END=5979866, ALLOC_END=9880, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a5bbf2 0x94da542 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [474]OB_MALLOC COST TOO MUCH TIME, cost_time=1012506, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1012496], seq:[0] [2025-08-14 09:07:20.764460] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:233) [709][T1001_RFLWorker][T1001][YB42AC4D0007-00063C495BE99C16-0-0] [lt=620958] ObRemoteFetchWorker is running(thread_index=0) [2025-08-14 09:07:19.766095] WDIAG [OCCAM] operator() (ob_occam_timer.h:81) [835][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4016] task executed too delay(delay_time=1996624, *p_task_={this:0x7f99319fe030, caller:ob_table_lock_service.cpp:start:276, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE6$_1273, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99319fe310, time_wheel:0x7f99535f3990, thread_pool:0x7f99535f3170, is_running:1, time_interval:10.00s, expected_run_time:09:07:16.426, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:21.731136] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19946] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:21.755826] WDIAG ev_invoke_pending (ev.c:2136) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24651][errcode=0] epoll handler too much time :24711 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:15.761694] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=135] [sql nio session](*s={this:0x7f98f733f030, session_id:3221589417, trace_id:YB42AC4D0007-00063C496359B268-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:252}, err:0, last_decode_time:1755161813752329, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1778, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:07:10.786570] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] [thread pool stat](pool_stat={pool:0x564d6e8922c0, last_check_time:1755162430786568, last_idle_time:102477764956, last_thread_count:12}, *pool={name:TimerWK, this:0x564d6e8922c0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:10, threads_idle_time:102477764956, tenant_id:500}, qsize=10, current_idle_time=102477764956) [2025-08-14 09:07:15.772460] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:217) [381][TsMgr][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162432758728], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:07:21.754060] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17185][errcode=0] epoll handler too much time :17227 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:21.754078] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=31269][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:21.761949] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7854][errcode=0] task scheduled too delay(scheduled_delay=1976088, expected_run_ts_us_=1755162439785859, ObClockGenerator::getRealClock()=1755162441761948, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:21.761971] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7276330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:21.761983] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1976822, expected_run_ts_us_=1755162439785160, ObClockGenerator::getRealClock()=1755162441761982, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:21.761992] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d4330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:21.762002] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=1957923, expected_run_ts_us_=1755162439804079, ObClockGenerator::getRealClock()=1755162441762002, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919836b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:21.762012] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919836b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:21.762026] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1931625, expected_run_ts_us_=1755162439830400, ObClockGenerator::getRealClock()=1755162441762026, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:21.762038] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.830, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:21.762048] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1931023, expected_run_ts_us_=1755162439831025, ObClockGenerator::getRealClock()=1755162441762048, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.831, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:21.754035] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:20.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:21.754433] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31737][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:20.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:17.751928] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:17.751816] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=21] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:14.738553] INFO [SERVER] execute_adapter_maintenance (ob_plugin_vector_index_scheduler.cpp:477) [807][T1002_TimerWK3_TTLTabletMgr][T1002][YB42AC4D0007-00063C495F8A1988-0-0] [lt=967357] finish generate tenant tablet tasks(ret=0, ret="OB_SUCCESS", tenant_id=1002) [2025-08-14 09:07:16.772819] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=1006721] current monitor number(seq_=-1) [2025-08-14 09:07:17.751989] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:21.775574] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4017493][errcode=0] epoll handler too much time :4023592 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:21.775591] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:21.775603] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:21.762058] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eb590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.831, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:22.722671] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=959154][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:21.762|207=960.62ms|210=1us|222=2us|total=960.65ms) [2025-08-14 09:07:22.722704] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4389] timer task use too much time(end=1755162442722703, start=1755162441762047, delta=960656) [2025-08-14 09:07:22.722726] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] task scheduled too delay(scheduled_delay=2860591, expected_run_ts_us_=1755162439862133, ObClockGenerator::getRealClock()=1755162442722725, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.862, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:21.762079] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8023][errcode=0] task scheduled too delay(scheduled_delay=672762, expected_run_ts_us_=1755162441089316, ObClockGenerator::getRealClock()=1755162441762079, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:21.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:17.896929] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=26] PNIO [ratelimit] time: 1755162437896926, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007618, add_bytes: 0 [2025-08-14 09:07:21.822709] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=11232][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:21.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:22.726996] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=902916][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:21.822|207=904.29ms|210=0us|222=1us|total=904.31ms) [2025-08-14 09:07:22.735583] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=8572][errcode=-4389] timer task use too much time(end=1755162442735582, start=1755162441822706, delta=912876) [2025-08-14 09:07:22.726314] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:22.737819] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7862] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:22.722761] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=959229][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:21.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:22.751771] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=28959][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:21.762|207=989.69ms|210=0us|222=2us|total=989.72ms) [2025-08-14 09:07:22.751806] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-4389] timer task use too much time(end=1755162442751805, start=1755162441762078, delta=989727) [2025-08-14 09:07:22.722744] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:19.862, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:09.765075] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=4023312] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623321168397, flush_io_finished_ret_:0, flush_io_finished_round_:26435, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:4, normal_idle_loop_cnt_:4, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [861]OB_MALLOC COST TOO MUCH TIME, cost_time=3014724, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=30, WASH_BLOCK_END=3014689, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=4], seq:[0, 1, 2, 3, 4] [2025-08-14 09:07:14.832272] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=1989035][errcode=-4018] LOGGER COST TOO MUCH TIME, cost: 7909435, owner: ObLog, click_count: 5, time dist:[FORMAT_END=4894653, ObMalloc#start=1, ObMalloc#end=3014753, ALLOC_END=1, APPEND_END=26], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3b4b 0x113d34aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [391]OB_MALLOC COST TOO MUCH TIME, cost_time=4991556, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=32, WASH_BLOCK_END=4991524], seq:[0, 1, 2] [2025-08-14 09:07:17.754595] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4991626, owner: ObLog, click_count: 5, time dist:[FORMAT_END=7, ObMalloc#start=1, ObMalloc#end=4991569, ALLOC_END=46, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [383]OB_MALLOC COST TOO MUCH TIME, cost_time=4992888, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=51, WASH_BLOCK_END=4992837], seq:[0, 1, 2] [2025-08-14 09:07:17.753420] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4992928, owner: ObLog, click_count: 5, time dist:[FORMAT_END=10, ObMalloc#start=1, ObMalloc#end=4992896, ALLOC_END=18, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [399]OB_MALLOC COST TOO MUCH TIME, cost_time=5018943, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=48, WASH_BLOCK_END=5018894], seq:[0, 1, 2] [2025-08-14 09:07:17.727463] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5019007, owner: ObLog, click_count: 5, time dist:[FORMAT_END=20, ObMalloc#start=1, ObMalloc#end=5018963, ALLOC_END=20, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [730]OB_MALLOC COST TOO MUCH TIME, cost_time=4375645, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=43, WASH_BLOCK_END=4375602], seq:[0, 1, 2] [568]OB_MALLOC COST TOO MUCH TIME, cost_time=3976407, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=49, WASH_BLOCK_END=3976356, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1], seq:[0, 1, 2, 3, 4] [357]OB_MALLOC COST TOO MUCH TIME, cost_time=3016816, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=147, WASH_BLOCK_END=3016669], seq:[0, 1, 2] [397]OB_MALLOC COST TOO MUCH TIME, cost_time=4980719, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=33, WASH_BLOCK_END=4980686], seq:[0, 1, 2] [2025-08-14 09:07:17.765991] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4980760, owner: ObLog, click_count: 5, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=4980727, ALLOC_END=20, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:17.896929] EDIAG eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4818823, owner: ObLog, click_count: 5, time dist:[FORMAT_END=16, ObMalloc#start=0, ObMalloc#end=4818742, ALLOC_END=65, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x27a8a8c8 0x97b7e1a 0x27a8f14d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [389]OB_MALLOC COST TOO MUCH TIME, cost_time=4988384, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=43, WASH_BLOCK_END=4988341], seq:[0, 1, 2] [2025-08-14 09:07:14.832272] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=1989035][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:07:22.752312] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29495][errcode=0] task scheduled too delay(scheduled_delay=2014252, expected_run_ts_us_=1755162440738058, ObClockGenerator::getRealClock()=1755162442752311, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:22.753710] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=1393][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:22.753742] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] task scheduled too delay(scheduled_delay=2022305, expected_run_ts_us_=1755162440731436, ObClockGenerator::getRealClock()=1755162442753742, ret=0, ret="OB_SUCCESS", *this={this:0x7f98918114b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.754595] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:09.765011] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=4022569] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:43623320180547, flush_io_finished_ret_:0, flush_io_finished_round_:26472, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:3, normal_idle_loop_cnt_:3, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-08-14 09:07:22.753801] WDIAG ev_invoke_pending (ev.c:2136) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4991626][errcode=0] epoll handler too much time :4999215 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:22.753823] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:22.753839] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:17.753420] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:22.753955] WDIAG ev_invoke_pending (ev.c:2136) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4992928][errcode=0] epoll handler too much time :5000544 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:22.753979] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:22.753992] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:17.727463] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:22.754067] WDIAG ev_invoke_pending (ev.c:2136) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5019007][errcode=0] epoll handler too much time :5026625 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:22.754091] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:22.754104] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:22.751828] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=996338, expected_run_ts_us_=1755162441755487, ObClockGenerator::getRealClock()=1755162442751826, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:21.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:17.758419] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4988428, owner: ObLog, click_count: 5, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=4988392, ALLOC_END=25, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:14.736699] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=2962799] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:07:17.765991] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:22.754311] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4980760][errcode=0] epoll handler too much time :4988328 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:22.754327] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:22.754340] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:17.758419] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:22.754414] WDIAG ev_invoke_pending (ev.c:2136) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4988428][errcode=0] epoll handler too much time :4996004 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:22.754426] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:22.754447] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [477]OB_MALLOC COST TOO MUCH TIME, cost_time=5006972, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=45, WASH_BLOCK_END=5006927], seq:[0, 1, 2] [656]OB_MALLOC COST TOO MUCH TIME, cost_time=4893708, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=50, WASH_BLOCK_END=4893653, ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=3], seq:[0, 1, 2, 3, 4] [2025-08-14 09:07:17.853391] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4893750, owner: ObLog, click_count: 5, time dist:[FORMAT_END=26, ObMalloc#start=1, ObMalloc#end=4893722, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [579]OB_MALLOC COST TOO MUCH TIME, cost_time=991457, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=41, WASH_BLOCK_END=991416], seq:[0, 1, 2] [395]OB_MALLOC COST TOO MUCH TIME, cost_time=4983329, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=41, WASH_BLOCK_END=4983287], seq:[0, 1, 2] [400]OB_MALLOC COST TOO MUCH TIME, cost_time=987858, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=987818], seq:[0, 1, 2] [2025-08-14 09:07:17.763781] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4983369, owner: ObLog, click_count: 7, time dist:[FORMAT_END=9, ObMalloc#start=1, ObMalloc#end=4983336, ObMalloc#start=21, ObMalloc#end=2, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [686]OB_MALLOC COST TOO MUCH TIME, cost_time=987826, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=34, WASH_BLOCK_END=987792], seq:[0, 1, 2] [376]OB_MALLOC COST TOO MUCH TIME, cost_time=987765, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=33, WASH_BLOCK_END=987732], seq:[0, 1, 2] [2025-08-14 09:07:11.760115] EDIAG [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=22660][errcode=0] LOGGER COST TOO MUCH TIME, cost: 10978060, owner: ObLog, click_count: 6, time dist:[FORMAT_END=4012625, ObMalloc#start=1, ObMalloc#end=5977560, ObMalloc#start=21, ObMalloc#end=987834, FREE_END=19], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1af3d850 0x1af3c294 0x20071394 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [856]OB_MALLOC COST TOO MUCH TIME, cost_time=987701, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=33, WASH_BLOCK_END=987668], seq:[0, 1, 2] [386]OB_MALLOC COST TOO MUCH TIME, cost_time=987679, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=75, WASH_BLOCK_END=987604], seq:[0, 1, 2] [776]OB_MALLOC COST TOO MUCH TIME, cost_time=987670, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=167, WASH_BLOCK_END=987503], seq:[0, 1, 2] [402]OB_MALLOC COST TOO MUCH TIME, cost_time=987444, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=61, WASH_BLOCK_END=987383], seq:[0, 1, 2] [1275]OB_MALLOC COST TOO MUCH TIME, cost_time=985862, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=47, WASH_BLOCK_END=985815], seq:[0, 1, 2] [394]OB_MALLOC COST TOO MUCH TIME, cost_time=985710, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=36, WASH_BLOCK_END=985674], seq:[0, 1, 2] [733]OB_MALLOC COST TOO MUCH TIME, cost_time=978021, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=34, WASH_BLOCK_END=977987], seq:[0, 1, 2] [729]OB_MALLOC COST TOO MUCH TIME, cost_time=3019703, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=47, WASH_BLOCK_END=3019656], seq:[0, 1, 2] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=4965065, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=55, WASH_BLOCK_END=4965010], seq:[0, 1, 2] [770]OB_MALLOC COST TOO MUCH TIME, cost_time=4996354, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=38, WASH_BLOCK_END=4996316], seq:[0, 1, 2] [2025-08-14 09:07:17.751107] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4996411, owner: ObLog, click_count: 5, time dist:[FORMAT_END=29, ObMalloc#start=1, ObMalloc#end=4996362, ALLOC_END=19, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [396]OB_MALLOC COST TOO MUCH TIME, cost_time=5003229, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=5003189], seq:[0, 1, 2] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=978040, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=46, WASH_BLOCK_END=977994], seq:[0, 1, 2] [384]OB_MALLOC COST TOO MUCH TIME, cost_time=977992, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=38, WASH_BLOCK_END=977953], seq:[0, 1, 2] [372]OB_MALLOC COST TOO MUCH TIME, cost_time=4974070, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=34, WASH_BLOCK_END=4974036], seq:[0, 1, 2] [392]OB_MALLOC COST TOO MUCH TIME, cost_time=978123, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=130, WASH_BLOCK_END=977993], seq:[0, 1, 2] [777]OB_MALLOC COST TOO MUCH TIME, cost_time=4995366, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=36, WASH_BLOCK_END=4995311, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=18], seq:[0, 1, 2, 3, 4] [663]OB_MALLOC COST TOO MUCH TIME, cost_time=4954376, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=39, WASH_BLOCK_END=4954337], seq:[0, 1, 2] [2025-08-14 09:07:13.759736] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] LOGGER COST TOO MUCH TIME, cost: 8982032, owner: ObLog, click_count: 6, time dist:[FORMAT_END=2015371, ObMalloc#start=1, ObMalloc#end=5980497, ObMalloc#start=22, ObMalloc#end=985719, FREE_END=422], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [574]OB_MALLOC COST TOO MUCH TIME, cost_time=4961195, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=56, WASH_BLOCK_END=4961139], seq:[0, 1, 2] [736]OB_MALLOC COST TOO MUCH TIME, cost_time=4961868, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=34, WASH_BLOCK_END=4961833], seq:[0, 1, 2] [2025-08-14 09:07:17.774083] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=45][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4973853, owner: ObLog, click_count: 8, time dist:[FORMAT_END=27, ObMalloc#start=2, ObMalloc#end=3995489, ObMalloc#start=20, ObMalloc#end=978048, ObMalloc#start=131, ObMalloc#end=26, FREE_END=110], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [387]OB_MALLOC COST TOO MUCH TIME, cost_time=5004196, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=40, WASH_BLOCK_END=5004156], seq:[0, 1, 2] [2025-08-14 09:07:17.749259] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4998725, owner: ObLog, click_count: 6, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=4020306, ObMalloc#start=19, ObMalloc#end=978131, FREE_END=256], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:18.973497] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [402][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3776476, owner: ObLog, click_count: 8, time dist:[FORMAT_END=19, ObMalloc#start=1, ObMalloc#end=2788164, ObMalloc#start=22, ObMalloc#end=987452, ObMalloc#start=329, ObMalloc#end=79, FREE_END=410], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:14.738553] EDIAG [SERVER] execute_adapter_maintenance (ob_plugin_vector_index_scheduler.cpp:477) [807][T1002_TimerWK3_TTLTabletMgr][T1002][YB42AC4D0007-00063C495F8A1988-0-0] [lt=967357][errcode=0] LOGGER COST TOO MUCH TIME, cost: 7026212, owner: ObLog, click_count: 5, time dist:[FORMAT_END=3989934, ObMalloc#start=1, ObMalloc#end=3036256, ALLOC_END=20, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1fb9131c 0x1fb8f4ec 0x1fb98f17 0x1fb8617a 0x96ec6c8 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:13.831542] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=990469][errcode=-4018] LOGGER COST TOO MUCH TIME, cost: 8910551, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3b4b 0x113d34aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:17.743701] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5004246, owner: ObLog, click_count: 5, time dist:[FORMAT_END=12, ObMalloc#start=1, ObMalloc#end=5004204, ALLOC_END=28, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [303]OB_MALLOC COST TOO MUCH TIME, cost_time=1006030, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=82, WASH_BLOCK_END=1005948], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=990797, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=37, WASH_BLOCK_END=990760], seq:[0, 1, 2] [404]OB_MALLOC COST TOO MUCH TIME, cost_time=949007, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=41, WASH_BLOCK_END=948942, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=23], seq:[0, 1, 2, 3, 4] [2025-08-14 09:07:17.743794] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5016983, owner: ObLog, click_count: 8, time dist:[FORMAT_END=42, ObMalloc#start=1, ObMalloc#end=4025886, ObMalloc#start=20, ObMalloc#end=990805, ObMalloc#start=109, ObMalloc#end=24, FREE_END=96], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [882]OB_MALLOC COST TOO MUCH TIME, cost_time=1005487, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1005485], seq:[0] [2025-08-14 09:07:09.765075] EDIAG [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [832][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=4023312][errcode=0] LOGGER COST TOO MUCH TIME, cost: 12967787, owner: ObLog, click_count: 5, time dist:[FORMAT_END=12967777, ObMalloc#start=1, ObMalloc#end=5, ALLOC_END=0, APPEND_END=4], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1d17bb52 0x1d17ada0 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [1275]OB_MALLOC COST TOO MUCH TIME, cost_time=960371, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=960369], seq:[0] [2025-08-14 09:07:17.853391] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:07:17.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:22.754567] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=4893750][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:17.853|207=4.90s|210=0us|222=2us|total=4.90s) [2025-08-14 09:07:22.754592] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=-4389] timer task use too much time(end=1755162442754592, start=1755162437853389, delta=4901203) [2025-08-14 09:07:17.763781] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:22.754709] WDIAG ev_invoke_pending (ev.c:2136) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4983369][errcode=0] epoll handler too much time :4990938 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:22.754725] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:22.754737] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:17.782416] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=28] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:07:17.751107] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:17.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:22.755096] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=4996411][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:17.751|207=5.00s|210=0us|222=2us|total=5.00s) [2025-08-14 09:07:22.755111] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] timer task use too much time(end=1755162442755110, start=1755162437751104, delta=5004006) [2025-08-14 09:07:17.744337] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:17.773611] WDIAG [OCCAM] operator() (ob_occam_time_guard.h:100) [372][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] thread maybe hung!(point=[thread id=338, timeout ts=09:05:56.779, last click point="ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()():73", last click ts=09:05:26.779], idx=338, next_allow_print_ts="09:06:26.779") [2025-08-14 09:07:17.786692] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=25] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:07:17.786052] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-08-14 09:07:17.743701] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:17.802765] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=19] gc stale ls task succ [2025-08-14 09:07:18.371856] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=33] gc stale ls task succ [2025-08-14 09:07:22.754189] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=2356][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:21.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:22.760015] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=5809][errcode=0] task scheduled too delay(scheduled_delay=750879, expected_run_ts_us_=1755162442009134, ObClockGenerator::getRealClock()=1755162442760014, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:22.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:19.733036] EDIAG [COMMON] generate_mod_stat_task (memory_dump.cpp:213) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3030741, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1002337, ObMalloc#start=2, ObMalloc#end=300, ObMalloc#start=1021797, ObMalloc#end=1006041, FREE_END=264], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf98b221 0xf98aea2 0x96b1d30 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:09.765011] EDIAG [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [552][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=4022569][errcode=0] LOGGER COST TOO MUCH TIME, cost: 12969156, owner: ObLog, click_count: 3, time dist:[FORMAT_END=12967838, ALLOC_END=1314, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1d17bb52 0x1d17ada0 0x1d17aaaa 0x1d182182 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:17.724538] EDIAG [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4017277][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5034050, owner: ObLog, click_count: 3, time dist:[FORMAT_END=5034049, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x99721a1 0x997a72e 0x2006ed17 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:17.782416] EDIAG do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4965112, owner: ObLog, click_count: 5, time dist:[FORMAT_END=17, ObMalloc#start=1, ObMalloc#end=4965073, ALLOC_END=20, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x202c530b 0x202c6888 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [882]OB_MALLOC COST TOO MUCH TIME, cost_time=1969719, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=1005503], seq:[0, 1] [2025-08-14 09:07:14.760775] EDIAG [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=22841][errcode=0] LOGGER COST TOO MUCH TIME, cost: 7982072, owner: ObLog, click_count: 6, time dist:[FORMAT_END=3965866, ObMalloc#start=1, ObMalloc#end=3028285, ObMalloc#start=18, ObMalloc#end=987773, FREE_END=129], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1fce4e91 0x1fce455d 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:14.736699] EDIAG [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=2962799][errcode=0] LOGGER COST TOO MUCH TIME, cost: 8005436, owner: ObLog, click_count: 5, time dist:[FORMAT_END=4988577, ObMalloc#start=2, ObMalloc#end=3016823, ALLOC_END=19, APPEND_END=15], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x18bdb084 0x18bda6b1 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:17.739892] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [477][T1_StandbySchem][T1][YB42AC4D0007-00063C495B6A040C-0-0] [lt=25][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5007048, owner: ObLog, click_count: 5, time dist:[FORMAT_END=24, ObMalloc#start=2, ObMalloc#end=5006981, ALLOC_END=34, APPEND_END=7], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [400]OB_MALLOC COST TOO MUCH TIME, cost_time=980098, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=980091], seq:[0] [400]OB_MALLOC COST TOO MUCH TIME, cost_time=980113, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=980108], seq:[0, 1] [2025-08-14 09:07:11.754730] EDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=0] LOGGER COST TOO MUCH TIME, cost: 11960654, owner: ObLog, click_count: 5, time dist:[FORMAT_END=7963915, ObMalloc#start=2, ObMalloc#end=3019711, ALLOC_END=977025, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0xfa06846 0xfa05f62 0xfa05c84 0xfa05a50 0x1c020b34 0x97a4183 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:17.752389] EDIAG [RS] skip_check_member_list_change_ (ob_ls_recovery_stat_handler.cpp:109) [777][T1002_TeRec][T1002][YB42AC4D0007-00063C495DDA00C9-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5006160, owner: ObLog, click_count: 5, time dist:[FORMAT_END=8, ObMalloc#start=1, ObMalloc#end=4995375, ALLOC_END=10776, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x1067fb2c 0x1067ecc2 0x1067e03d 0x1002f2a3 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:22.760041] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:22.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:22.754604] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=0] task scheduled too delay(scheduled_delay=2960692, expected_run_ts_us_=1755162439793911, ObClockGenerator::getRealClock()=1755162442754604, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd070, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fd350, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:5.00s, expected_run_time:09:07:19.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:22.761284] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=6667][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd070, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fd350, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:5.00s, expected_run_time:09:07:19.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:22.753758] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98918114b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0d90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.731, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.724538] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4017277] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:22.726984] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=5989747] [sql nio session](*s={this:0x7f98f733e030, session_id:3221589980, trace_id:YB42AC4D0007-00063C496359B261-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:251}, err:0, last_decode_time:1755161748778070, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1681, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:07:21.817841] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=1740266][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 61.073220 loop:26707, slowcnt: 542 [2025-08-14 09:07:22.761325] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7551][errcode=0] task scheduled too delay(scheduled_delay=2026872, expected_run_ts_us_=1755162440734452, ObClockGenerator::getRealClock()=1755162442761324, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728aa90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:22.768459] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=7121][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728aa90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:22.768473] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=2037655, expected_run_ts_us_=1755162440730817, ObClockGenerator::getRealClock()=1755162442768473, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:22.760056] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=555758, expected_run_ts_us_=1755162442204297, ObClockGenerator::getRealClock()=1755162442760056, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:22.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:22.854667] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:07:22.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:22.855187] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:22.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:22.868409] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=949052][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 62.123788 loop:26707, slowcnt: 543 [2025-08-14 09:07:22.955211] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:22.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:23.055264] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:22.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:23.255363] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:23.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:23.455444] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=52][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:23.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:23.655541] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:23.555, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:22.768052] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=41005] [sql nio session](*s={this:0x7f98f7332030, session_id:3221590132, trace_id:YB42AC4D0007-00063C496359B265-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:266}, err:0, last_decode_time:1755161768783802, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:2125, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:07:22.768483] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.717750] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=947828][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:22.768|207=949.27ms|210=0us|222=2us|total=949.30ms) [2025-08-14 09:07:23.717777] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4389] timer task use too much time(end=1755162443717776, start=1755162442768472, delta=949304) [2025-08-14 09:07:23.717788] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] task scheduled too delay(scheduled_delay=2985385, expected_run_ts_us_=1755162440732401, ObClockGenerator::getRealClock()=1755162443717787, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:22.768636] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=8565][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:22.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:22.835645] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:22.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:11.754730] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=1755161119797572, error_count_for_change_leader_=1, last_error_tstamp_for_change_leader_=1755162138745740) [2025-08-14 09:07:23.726844] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9432] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:18.789575] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [882][T1002_ArcSender][T1002][YB42AC4D0007-00063C495EC99C16-0-0] [lt=2834551] ObArchiveSender is running(thread_index=6) [2025-08-14 09:07:23.717922] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=947848][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:22.760|207=957.86ms|210=0us|222=2us|total=957.88ms) [2025-08-14 09:07:23.734481] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=16532][errcode=-4389] timer task use too much time(end=1755162443734480, start=1755162442760055, delta=974425) [2025-08-14 09:07:23.734522] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=729034, expected_run_ts_us_=1755162443005487, ObClockGenerator::getRealClock()=1755162443734522, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:23.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:23.734545] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:23.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:23.717169] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=947681] [sql nio session](*s={this:0x7f98f733b030, session_id:3221590977, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:10, sql_initiative_shutdown:false, reader:{fd:211}, err:5, last_decode_time:1755162219768794, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:07:23.733312] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=896305][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:22.835|207=897.66ms|210=0us|222=2us|total=897.70ms) [2025-08-14 09:07:23.717804] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.732, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:17.947266] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] EASY SLOW: thread: 7f99543f7300, lastrun: 1755162437.754585 cost: 0.192679 loop:26780, slowcnt: 1 [2025-08-14 09:07:22.767913] WDIAG ev_invoke_pending (ev.c:2136) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5004246][errcode=0] epoll handler too much time :5024227 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:23.738005] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=968606] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:23.738019] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:17.802765] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=19][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4949538, , BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:17.793483] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [663][T1001_StandbySc][T1001][YB42AC4D0007-00063C495C7A03F0-0-0] [lt=31][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4954419, owner: ObLog, click_count: 5, time dist:[FORMAT_END=15, ObMalloc#start=1, ObMalloc#end=4954384, ALLOC_END=19, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:17.744208] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [776][T1002_StandbySc][T1002][YB42AC4D0007-00063C495DFA0402-0-0] [lt=23][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5003678, owner: ObLog, click_count: 6, time dist:[FORMAT_END=14, ObMalloc#start=2, ObMalloc#end=4015350, ObMalloc#start=24, ObMalloc#end=987678, FREE_END=610], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:17.786052] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [736][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4961913, owner: ObLog, click_count: 5, time dist:[FORMAT_END=17, ObMalloc#start=0, ObMalloc#end=4961876, ALLOC_END=19, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:17.786692] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [574][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4961236, owner: ObLog, click_count: 5, time dist:[FORMAT_END=14, ObMalloc#start=1, ObMalloc#end=4961202, ALLOC_END=19, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:14.842059] EDIAG [RS] get_valid_sts_after (ob_tenant_info_loader.cpp:454) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 7900898, owner: ObLog, click_count: 6, time dist:[FORMAT_END=931812, ObMalloc#start=1, ObMalloc#end=5981262, ObMalloc#start=19, ObMalloc#end=987708, FREE_END=96], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x1001d942 0x996d843 0x996d15a 0x996cfd8 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [1275]OB_MALLOC COST TOO MUCH TIME, cost_time=988423, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=960402], seq:[0, 1] [2025-08-14 09:07:20.756591] EDIAG [RS.COMPACTION] try_idle (ob_major_merge_scheduler.cpp:226) [1275][T1001_MergeSche][T1001][YB42AC4D0007-00063C4963E99CBE-0-0] [lt=28993][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2983750, owner: ObLog, click_count: 7, time dist:[FORMAT_END=1009406, ObMalloc#start=1, ObMalloc#end=985870, ObMalloc#start=27, ObMalloc#end=988441, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1125d502 0x1125cf81 0x1125a4c3 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:23.740001] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:23.733591] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=1029125] snapshot_traffic_info once [2025-08-14 09:07:23.744901] INFO [COMMON] handle (memory_dump.cpp:544) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=3030741] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-08-14 09:07:23.745116] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [564][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=979574] XA scheduler heartbeat task statistics(avg_time=488518) [2025-08-14 09:07:23.745286] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5034050] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:23.745324] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=39] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:23.745335] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:18.371856] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=33][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4381310, owner: ObLog, click_count: 7, time dist:[FORMAT_END=14, ObMalloc#start=2, ObMalloc#end=4375653, ObMalloc#start=19, ObMalloc#end=1, ALLOC_END=5620, APPEND_END=1], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:23.745966] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=22] update_check_range(min_check_version=2540, max_check_version=2540, global_version=2653) [2025-08-14 09:07:23.734696] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1008) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=17491][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x7f98f7335030, session_id:3221590537, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:265}, err:5, last_decode_time:1755162392718915, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1366, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:07:21.755573] EDIAG [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=4326][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1976075, owner: ObLog, click_count: 8, time dist:[FORMAT_END=1975137, ObMalloc#start=1, ObMalloc#end=260, ObMalloc#start=34, ObMalloc#end=39, FREE_END=604, ALLOC_BLOCK#start=4, ALLOC_BLOCK#end=24], seq:[0, 1, 2, 3, 4, 5, 6, 7], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x1af3d850 0x1af3c294 0x20071394 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:23.734716] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1401][errcode=-4389] timer task use too much time(end=1755162443734716, start=1755162442835643, delta=899073) [2025-08-14 09:07:23.734730] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=16896][errcode=0] task scheduled too delay(scheduled_delay=3001292, expected_run_ts_us_=1755162440733437, ObClockGenerator::getRealClock()=1755162443734729, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747326] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12572][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fb430, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.733, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747375] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=45][errcode=0] task scheduled too delay(scheduled_delay=3005747, expected_run_ts_us_=1755162440741627, ObClockGenerator::getRealClock()=1755162443747374, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747404] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747426] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] task scheduled too delay(scheduled_delay=3003873, expected_run_ts_us_=1755162440743552, ObClockGenerator::getRealClock()=1755162443747426, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747439] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891983bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747453] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=3004297, expected_run_ts_us_=1755162440743155, ObClockGenerator::getRealClock()=1755162443747452, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747466] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891811230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0c10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747488] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] task scheduled too delay(scheduled_delay=2990589, expected_run_ts_us_=1755162440756899, ObClockGenerator::getRealClock()=1755162443747488, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fc830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747510] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fc830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747524] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=2991335, expected_run_ts_us_=1755162440756188, ObClockGenerator::getRealClock()=1755162443747523, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747538] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a5730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747551] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=2991232, expected_run_ts_us_=1755162440756319, ObClockGenerator::getRealClock()=1755162443747551, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a54b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747568] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a54b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7282a90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747582] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=2987034, expected_run_ts_us_=1755162440760547, ObClockGenerator::getRealClock()=1755162443747581, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72edc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747595] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72edc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747608] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=2982390, expected_run_ts_us_=1755162440765218, ObClockGenerator::getRealClock()=1755162443747608, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72edb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747622] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a2cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72edb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747637] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=2981514, expected_run_ts_us_=1755162440766122, ObClockGenerator::getRealClock()=1755162443747637, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747651] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a85b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398490, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.766, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747665] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=2979031, expected_run_ts_us_=1755162440768633, ObClockGenerator::getRealClock()=1755162443747664, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747677] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a8330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7398310, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.768, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747691] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=2986693, expected_run_ts_us_=1755162440760998, ObClockGenerator::getRealClock()=1755162443747691, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747707] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.760, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747720] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=2976885, expected_run_ts_us_=1755162440770834, ObClockGenerator::getRealClock()=1755162443747719, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747731] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199f6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ac910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.770, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747743] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=2968404, expected_run_ts_us_=1755162440779339, ObClockGenerator::getRealClock()=1755162443747743, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728ac10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747755] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739d230, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728ac10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747770] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=2944864, expected_run_ts_us_=1755162440802905, ObClockGenerator::getRealClock()=1755162443747770, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.802, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747783] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7275930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.802, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747795] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=2946320, expected_run_ts_us_=1755162440801474, ObClockGenerator::getRealClock()=1755162443747794, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747806] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982cb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.801, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747818] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=2943507, expected_run_ts_us_=1755162440804310, ObClockGenerator::getRealClock()=1755162443747818, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728f9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747853] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728f9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7292010, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747865] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=2929325, expected_run_ts_us_=1755162440818540, ObClockGenerator::getRealClock()=1755162443747865, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eae10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747876] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d2030, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72eae10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.818, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747888] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=2924981, expected_run_ts_us_=1755162440822907, ObClockGenerator::getRealClock()=1755162443747888, ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747910] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180e7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f290, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.822, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747924] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=2882566, expected_run_ts_us_=1755162440865358, ObClockGenerator::getRealClock()=1755162443747924, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747936] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7287730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289e90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747948] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=2881175, expected_run_ts_us_=1755162440866773, ObClockGenerator::getRealClock()=1755162443747948, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.866, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747968] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7285bb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288f90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:20.866, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.747988] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=2418889, expected_run_ts_us_=1755162441329099, ObClockGenerator::getRealClock()=1755162443747988, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748000] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fa2b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748019] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=2022178, expected_run_ts_us_=1755162441725840, ObClockGenerator::getRealClock()=1755162443748019, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72756b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748031] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72756b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.725, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748043] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=2023910, expected_run_ts_us_=1755162441724133, ObClockGenerator::getRealClock()=1755162443748043, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.724, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748057] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a3930, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.724, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748070] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=2018697, expected_run_ts_us_=1755162441729372, ObClockGenerator::getRealClock()=1755162443748069, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a36b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748081] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a36b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7283c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.729, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748102] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] task scheduled too delay(scheduled_delay=2009852, expected_run_ts_us_=1755162441738249, ObClockGenerator::getRealClock()=1755162443748101, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919bfbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aa690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748113] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919bfbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aa690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748128] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=2009221, expected_run_ts_us_=1755162441738906, ObClockGenerator::getRealClock()=1755162443748128, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919911b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196cc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748140] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919911b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196cc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748152] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=2009860, expected_run_ts_us_=1755162441738292, ObClockGenerator::getRealClock()=1755162443748152, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d40b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748164] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d40b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ec190, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748176] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=2013767, expected_run_ts_us_=1755162441734408, ObClockGenerator::getRealClock()=1755162443748176, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748189] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c5b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.734, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748201] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=2010071, expected_run_ts_us_=1755162441738130, ObClockGenerator::getRealClock()=1755162443748201, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748213] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739c330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728b510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748225] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=2009939, expected_run_ts_us_=1755162441738286, ObClockGenerator::getRealClock()=1755162443748225, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748244] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728ad90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748258] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=2009738, expected_run_ts_us_=1755162441738519, ObClockGenerator::getRealClock()=1755162443748258, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748269] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739cd30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f728af10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.738, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748282] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=2003757, expected_run_ts_us_=1755162441744524, ObClockGenerator::getRealClock()=1755162443748281, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.744, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748293] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919af1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b9210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.744, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748305] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=2006216, expected_run_ts_us_=1755162441742088, ObClockGenerator::getRealClock()=1755162443748305, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aaf90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748318] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c0ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aaf90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.742, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748330] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=2008029, expected_run_ts_us_=1755162441740301, ObClockGenerator::getRealClock()=1755162443748330, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aba10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748341] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c1c30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919aba10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.740, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748353] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=2002894, expected_run_ts_us_=1755162441745459, ObClockGenerator::getRealClock()=1755162443748353, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748365] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a0830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.745, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748380] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=2000337, expected_run_ts_us_=1755162441748042, ObClockGenerator::getRealClock()=1755162443748380, ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196cb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748391] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891990f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196cb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.748, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748403] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=2005232, expected_run_ts_us_=1755162441743171, ObClockGenerator::getRealClock()=1755162443748403, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748415] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f739a7b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196b810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.743, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748427] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1996479, expected_run_ts_us_=1755162441751947, ObClockGenerator::getRealClock()=1755162443748427, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.751, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748440] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a05b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ad210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.751, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748452] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1984910, expected_run_ts_us_=1755162441763541, ObClockGenerator::getRealClock()=1755162443748452, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748463] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73fd730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891825b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.763, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748475] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1969358, expected_run_ts_us_=1755162441779116, ObClockGenerator::getRealClock()=1755162443748475, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919acf10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748487] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a00b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919acf10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.779, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748511] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] task scheduled too delay(scheduled_delay=1975581, expected_run_ts_us_=1755162441772930, ObClockGenerator::getRealClock()=1755162443748511, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748523] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7ed9b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f0790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.772, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748536] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1965014, expected_run_ts_us_=1755162441783521, ObClockGenerator::getRealClock()=1755162443748536, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919abe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748547] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199e530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919abe90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.783, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748559] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1963694, expected_run_ts_us_=1755162441784865, ObClockGenerator::getRealClock()=1755162443748559, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748573] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7274f30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b5090, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.784, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748584] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] task scheduled too delay(scheduled_delay=1962821, expected_run_ts_us_=1755162441785763, ObClockGenerator::getRealClock()=1755162443748584, ret=0, ret="OB_SUCCESS", *this={this:0x7f989199fe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919acd90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748597] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989199fe30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919acd90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.785, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748608] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1965856, expected_run_ts_us_=1755162441782752, ObClockGenerator::getRealClock()=1755162443748608, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e80b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748620] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e80b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891973b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.782, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748633] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=1959103, expected_run_ts_us_=1755162441789530, ObClockGenerator::getRealClock()=1755162443748633, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a31b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ede10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748644] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73a31b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f72ede10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.789, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748656] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1942456, expected_run_ts_us_=1755162441806200, ObClockGenerator::getRealClock()=1755162443748656, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.806, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748668] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919c19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ab890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.806, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748679] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1941235, expected_run_ts_us_=1755162441807444, ObClockGenerator::getRealClock()=1755162443748679, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748692] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d8830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f989196a610, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.807, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748704] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1944324, expected_run_ts_us_=1755162441804380, ObClockGenerator::getRealClock()=1755162443748704, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748716] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fcfb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.804, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748727] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1948288, expected_run_ts_us_=1755162441800439, ObClockGenerator::getRealClock()=1755162443748727, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.800, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748739] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288510, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.800, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748753] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=1943408, expected_run_ts_us_=1755162441805344, ObClockGenerator::getRealClock()=1755162443748753, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728e330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7290690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.805, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748764] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728e330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7290690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.805, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748775] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1945049, expected_run_ts_us_=1755162441803726, ObClockGenerator::getRealClock()=1755162443748775, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.803, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748787] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7394830, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7f1b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.803, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748801] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=1924184, expected_run_ts_us_=1755162441824616, ObClockGenerator::getRealClock()=1755162443748800, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.824, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748812] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72e9730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739e990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.824, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748824] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1916490, expected_run_ts_us_=1755162441832333, ObClockGenerator::getRealClock()=1755162443748824, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.832, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748835] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f73d76b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891969b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.832, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748848] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=1883917, expected_run_ts_us_=1755162441864930, ObClockGenerator::getRealClock()=1755162443748847, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ade10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.864, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748861] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a19b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919ade10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.864, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748873] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1883783, expected_run_ts_us_=1755162441865089, ObClockGenerator::getRealClock()=1755162443748873, ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748885] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989191d4b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891921990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748897] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1883474, expected_run_ts_us_=1755162441865422, ObClockGenerator::getRealClock()=1755162443748896, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919adc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748908] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a1730, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919adc90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748921] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=1883380, expected_run_ts_us_=1755162441865541, ObClockGenerator::getRealClock()=1755162443748921, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919adb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748933] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919a14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919adb10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748944] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1870833, expected_run_ts_us_=1755162441878111, ObClockGenerator::getRealClock()=1755162443748944, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72865b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.878, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748955] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72865b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288990, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.878, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748976] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] task scheduled too delay(scheduled_delay=1849136, expected_run_ts_us_=1755162441899839, ObClockGenerator::getRealClock()=1755162443748975, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728de30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7290b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.899, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.748989] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f728de30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7290b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.899, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.749001] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1846145, expected_run_ts_us_=1755162441902855, ObClockGenerator::getRealClock()=1755162443749001, ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.902, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.749013] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb6b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ee210, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:21.902, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.749028] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=1637371, expected_run_ts_us_=1755162442111656, ObClockGenerator::getRealClock()=1755162443749028, ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:22.111, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.749040] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7284530, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289d10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:22.111, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.749062] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] task scheduled too delay(scheduled_delay=1018340, expected_run_ts_us_=1755162442730722, ObClockGenerator::getRealClock()=1755162443749062, ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:22.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.749074] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b0d30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968390, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:22.730, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:23.749086] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:23.734557] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=725423, expected_run_ts_us_=1755162443009134, ObClockGenerator::getRealClock()=1755162443734557, ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:23.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:23.749381] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=14800][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:19.886293] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1872341, owner: ObLog, click_count: 4, time dist:[FORMAT_END=1866829, FREE_END=5512, ALLOC_BLOCK#start=5, ALLOC_BLOCK#end=2], seq:[0, 1, 2, 3], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:21.721741] EDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=981163][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2028981, owner: ObLog, click_count: 7, time dist:[ALLOC_BLOCK#start=6, ALLOC_BLOCK#end=1012525, FORMAT_END=1016438, ObMalloc#start=1, ObMalloc#end=5, ALLOC_END=0, APPEND_END=6], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ed35c 0x97aead0 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:17.744337] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5003265, owner: ObLog, click_count: 5, time dist:[FORMAT_END=8, ObMalloc#start=0, ObMalloc#end=5003237, ALLOC_END=20, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:18.720990] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4027831, owner: ObLog, click_count: 6, time dist:[FORMAT_END=15, ObMalloc#start=1, ObMalloc#end=3040021, ObMalloc#start=27, ObMalloc#end=987687, FREE_END=80], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:15.755392] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] LOGGER COST TOO MUCH TIME, cost: 6989680, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1986993, ObMalloc#start=1, ObMalloc#end=4024226, ObMalloc#start=18, ObMalloc#end=978009, FREE_END=433], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:17.751928] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4017289, owner: ObLog, click_count: 5, time dist:[FORMAT_END=7, ObMalloc#start=0, ObMalloc#end=4017258, ALLOC_END=24, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:18.789575] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:239) [882][T1002_ArcSender][T1002][YB42AC4D0007-00063C495EC99C16-0-0] [lt=2834551][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4937286, owner: ObLog, click_count: 7, time dist:[FORMAT_END=1962678, ObMalloc#start=2, ObMalloc#end=121, ObMalloc#start=1004741, ObMalloc#end=1969738, ALLOC_END=6, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x113d3c61 0x113d348b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:17.947266] EDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [400][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5780345, owner: ObLog, click_count: 9, time dist:[FORMAT_END=17, ObMalloc#start=1, ObMalloc#end=3812239, ObMalloc#start=27, ObMalloc#end=987868, ObMalloc#start=71, ObMalloc#end=980120, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x27ade52a 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:21.721741] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=981163][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:20.739|207=982.66ms|210=1us|222=1us|total=3.01s) [2025-08-14 09:07:23.753822] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=2028981][errcode=-4389] timer task use too much time(end=1755162443753820, start=1755162440739072, delta=3014748) [2025-08-14 09:07:23.753892] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:19.765144] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [836][T1002_OBJLockGC][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] cost too much time:ob_occam_timer.h:int oceanbase::common::occam::ObOccamTimerTask::commit_task_to_thread_pool_(), (this=|threshold=100.00ms|start at 09:07:17.769|289=2.00s|total=5.98s) [2025-08-14 09:07:23.755568] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5003265][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:23.755608] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:21.754476] EDIAG [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [279][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=2010145][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2962743, owner: ObLog, click_count: 6, time dist:[FORMAT_END=2962234, ObMalloc#start=1, ObMalloc#end=105, ObMalloc#start=36, ObMalloc#end=38, FREE_END=329], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x95ed9fd 0x95ed3ab 0x95ed0b4 0x95ece9b 0x260b4cc1 0x260b9960 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=1006002, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=1006001], seq:[0] [2025-08-14 09:07:23.745342] EDIAG [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2968488, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1017212, ALLOC_END=1951275, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x99721a1 0x997a72e 0x2006ed17 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:24.964426] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1787777, owner: ObLog, click_count: 3, time dist:[FORMAT_END=30, ALLOC_END=1787746, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:23.747769] EDIAG [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=1976075][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3005319, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1986825, ALLOC_END=1018493, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x99721a1 0x997a72e 0x2006ed17 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=2999134, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=57, WASH_BLOCK_END=83, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=1006022], seq:[0, 1, 2, 3, 4] [2025-08-14 09:07:22.727400] EDIAG dump_info (ob_timer_service.cpp:728) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=4404860][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4026999, owner: ObLog, click_count: 3, time dist:[FORMAT_END=3998653, ALLOC_END=28346, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x2618816f 0x9756b3d 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:23.754384] EDIAG [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=10978060][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3003173, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1980210, ALLOC_END=1022962, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x99721a1 0x997a72e 0x2006ed17 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:16.445160] EDIAG try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] LOGGER COST TOO MUCH TIME, cost: 10308030, owner: ObLog, click_count: 6, time dist:[FORMAT_END=10307564, ObMalloc#start=1, ObMalloc#end=65, ObMalloc#start=23, ObMalloc#end=42, FREE_END=335], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0x26499505 0x97074a3 0x26498fa9 0x7f9973e081cf 0x7f9973a39dd3 [385]OB_MALLOC COST TOO MUCH TIME, cost_time=3980996, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=3980988], seq:[0] [385]OB_MALLOC COST TOO MUCH TIME, cost_time=3981020, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=3981016], seq:[0, 1] [2025-08-14 09:07:23.757024] EDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4017289][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3981043, owner: ObLog, click_count: 5, time dist:[FORMAT_END=7, ObMalloc#start=0, ObMalloc#end=3981030, ALLOC_END=5, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0x27a0e162 0x9975d8e 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:25.738314] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=4381310][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2004212, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2004207, ObMalloc#start=1, ObMalloc#end=4, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:25.738316] EDIAG [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=4949538][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2004343, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2004343, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x9a22832 0xf986a86 0x97cae56 0xf9865f2 0x113f9d76 0x113f3896 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:26.718206] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [776][T1002_StandbySc][T1002][YB42AC4D0007-00063C495DFA0403-0-0] [lt=5003678][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1026160, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1026158, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:26.718387] EDIAG [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2968488][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1027109, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1027099, ALLOC_END=10, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x99721a1 0x997a72e 0x2006ed17 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=977587, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=65536, owner: ALLOC_BLOCK, click_count: 16, time dist:[ALLOC_CHUNK_END=977502, ALLOC_CHUNK_END=18, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=5, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4, ALLOC_CHUNK_END=4], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] [2025-08-14 09:07:26.718242] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [477][T1_StandbySchem][T1][YB42AC4D0007-00063C495B6A040D-0-0] [lt=5007048][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1038242, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1038241, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:26.718222] EDIAG [RS] do_work (ob_standby_schema_refresh_trigger.cpp:68) [663][T1001_StandbySc][T1001][YB42AC4D0007-00063C495C7A03F1-0-0] [lt=4954419][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1038307, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1038306, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x10038632 0x10037213 0x1002aafa 0x266875b4 0x2668e4e7 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:23.755625] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:23.755664] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4027831][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:23.755682] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:23.755696] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:23.755754] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:23.756429] WDIAG ev_invoke_pending (ev.c:2136) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6989680][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:23.756449] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:23.756463] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:23.757281] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=219] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:23.758582] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=265] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:23.769590] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=300][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:23.777143] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=294][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:23.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:23.807398] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:23.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:23.812340] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [741][T1001_MultiVers][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:23.855793] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:23.858617] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:23.868956] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:23.969709] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:23.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:23.995372] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:23.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:24.028779] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:24.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:24.058736] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:23.955, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:24.169829] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:24.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:24.246869] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:24.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:24.258806] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:24.155, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:24.270128] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:24.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:24.300329] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:24.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:24.369903] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:24.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:24.521888] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:24.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:24.570035] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:24.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:24.738873] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:24.256930] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:24.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:24.746221] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=488546][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:24.256|207=489.29ms|210=0us|222=2us|total=489.33ms) [2025-08-14 09:07:24.746259] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=37][errcode=-4389] timer task use too much time(end=1755162444746259, start=1755162444256928, delta=489331) [2025-08-14 09:07:24.746276] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] task scheduled too delay(scheduled_delay=262743, expected_run_ts_us_=1755162444483531, ObClockGenerator::getRealClock()=1755162444746275, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:24.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:24.746293] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:24.483, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:24.755734] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=41] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:24.894992] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=1872341][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2862282, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2862280, ALLOC_END=2, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:24.911585] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=1847176][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2845825, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2845825, ALLOC_END=0, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:24.921497] EDIAG [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=1839581][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2836947, owner: ObLog, click_count: 3, time dist:[FORMAT_END=2836946, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1dbe46cb 0x1db87f76 0x94d5563 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:16.743439] EDIAG [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=0] LOGGER COST TOO MUCH TIME, cost: 11005313, owner: ObLog, click_count: 3, time dist:[FORMAT_END=11005305, ALLOC_END=7, APPEND_END=1], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1b0b25c8 0x98936c3 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:23.738559] EDIAG [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3029591][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4021563, owner: ObLog, click_count: 3, time dist:[FORMAT_END=4021562, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x1f9ebc06 0x97c9052 0x1f9819be 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:26.757712] EDIAG [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1007440, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1007439, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x99721a1 0x997a72e 0x2006ed17 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:24.756504] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:24.759286] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:24.763367] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:24.763642] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:24.458926] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:24.355, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:24.764224] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=304832][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:24.458|207=305.30ms|210=0us|222=2us|total=305.33ms) [2025-08-14 09:07:24.764261] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4389] timer task use too much time(end=1755162444764260, start=1755162444458922, delta=305338) [2025-08-14 09:07:24.770180] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:24.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:24.779905] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:24.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:24.800037] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:24.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:24.856470] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd070, caller:ob_leader_coordinator.cpp:mtl_start:97, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_69, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fd350, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:5.00s, expected_run_time:09:07:24.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:24.856497] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:07:24.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:24.869536] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 64.124915 loop:26707, slowcnt: 563 [2025-08-14 09:07:24.970275] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:24.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:25.021450] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:25.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:25.170387] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:25.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:25.272880] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:25.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:25.303045] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:25.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:25.370503] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:25.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:25.524268] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:25.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:25.570630] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:25.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:25.752338] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:25.755556] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:25.757315] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:25.758379] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:25.770710] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:25.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:25.779756] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:25.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:25.809930] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:25.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:25.970792] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:25.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:26.021148] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:26.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:26.170921] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:26.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:26.272486] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:26.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:26.302708] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:26.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:26.371041] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:26.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:26.524024] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:26.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:26.571264] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=40][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:26.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:23.745342] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:26.739604] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:24.964426] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:24.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:26.754947] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1787777][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:24.964|207=1.79s|210=1us|222=1us|total=1.79s) [2025-08-14 09:07:26.754968] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task use too much time(end=1755162446754967, start=1755162444964423, delta=1790544) [2025-08-14 09:07:23.747769] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=1976075] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:26.757678] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=3005319] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:26.757691] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:26.757698] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:26.757705] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:25.757424] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:26.758430] WDIAG ev_invoke_pending (ev.c:2136) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=999492][errcode=0] epoll handler too much time :1001012 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:26.758456] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:25.757619] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:26.758470] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=999338][errcode=0] epoll handler too much time :1000857 count=1 malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2025-08-14 09:07:25.857005] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:07:25.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:26.759296] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=900922][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:25.857|207=902.29ms|210=0us|222=3us|total=902.31ms) [2025-08-14 09:07:26.759317] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] timer task use too much time(end=1755162446759316, start=1755162445857003, delta=902313) [2025-08-14 09:07:25.870086] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 65.125464 loop:26707, slowcnt: 573 [2025-08-14 09:07:26.759384] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:26.759434] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:23.727253] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=4995070] current monitor number(seq_=-1) [2025-08-14 09:07:22.727400] INFO dump_info (ob_timer_service.cpp:728) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=4404860] dump info [summary](this=0x7f99230b0af0, this={this:0x7f99230b0af0, tenant_id_:1002, priority_task_queue_.size():36, running_task_set_.size():31, uncanceled_task_set_.size():0, worker_thread_pool_.get_queue_num():15}) [2025-08-14 09:07:23.754384] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=10978060] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:24.756481] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:26.769517] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:26.770079] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:26.760, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:26.770899] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=267] PNIO [ratelimit] time: 1755162446770898, bytes: 0, bw: 0.000000 MB/s, add_ts: 1001373, add_bytes: 0 [2025-08-14 09:07:26.771659] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:26.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:26.800283] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=61][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:26.792, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:26.855045] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:26.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:26.971785] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=17][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:26.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:27.022118] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:27.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:27.171889] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:27.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:27.273799] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:27.372024] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=37][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:27.740368] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:23.757024] WDIAG ev_invoke_pending (ev.c:2136) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4017289][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:27.744126] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3981043] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:27.744140] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:26.758489] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:27.745503] WDIAG ev_invoke_pending (ev.c:2136) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=985522][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:25.738314] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=4381310] gc stale ls task succ [2025-08-14 09:07:27.745592] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=2004212] gc stale ls task succ [2025-08-14 09:07:25.738316] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=4949538] gc stale ls task succ [2025-08-14 09:07:27.745714] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=2004343] gc stale ls task succ [2025-08-14 09:07:26.762129] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=3003173] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:27.746626] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=983008] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:27.746641] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:27.746648] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:26.718387] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2968488] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:27.746654] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:26.955110] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:26.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:27.751020] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=794701][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:27.751050] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4389] timer task use too much time(end=1755162447751049, start=1755162446955107, delta=795942) [2025-08-14 09:07:27.755321] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=900] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:27.757203] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:27.757283] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:27.757882] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4227] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:26.859348] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:07:26.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:27.759409] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=898692][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:27.759445] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=-4389] timer task use too much time(end=1755162447759445, start=1755162446859346, delta=900099) [2025-08-14 09:07:26.959462] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=887913][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 66.214841 loop:26707, slowcnt: 575 [2025-08-14 09:07:27.759472] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:27.759497] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:27.759570] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=2804] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:27.759627] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=213] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:27.759689] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:24.894992] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [833][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=1872341] report RowHolderMapper summary info(count=2, bkt_cnt=248) [2025-08-14 09:07:24.911585] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [553][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=1847176] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:07:15.775173] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [561][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=18] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=16, total_destructed_detector=16, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f994b3ee2b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-08-14 09:07:24.921497] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [718][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=1839581] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-08-14 09:07:16.743439] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [864][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-08-14 09:07:27.747301] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=643] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:27.765415] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=18084] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:27.765426] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:23.738559] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3029591] [IO STATUS CALLBACK](tenant_id=1001, io_status=runner[0]: tid=696, queue_count=0) [2025-08-14 09:07:27.747076] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1027109] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:27.766648] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=19542] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:27.766656] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:26.757712] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:27.766694] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=1007440] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:27.766700] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:27.766705] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:23.727253] EDIAG [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=4995070][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3028506, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1f832f91 0x97c36fe 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:24.756481] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2002639, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:27.770839] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2002639][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:27.770882] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:27.770897] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:27.772627] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=14] PNIO [ratelimit] time: 1755162447772624, bytes: 0, bw: 0.000000 MB/s, add_ts: 1001726, add_bytes: 0 [2025-08-14 09:07:24.765857] EDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=417543][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3002433, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0xf9ed35c 0x97aead0 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:24.765857] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=417543][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:24.347|207=418.18ms|210=0us|222=2us|total=3.43s) [2025-08-14 09:07:27.772853] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=3002433][errcode=-4389] timer task use too much time(end=1755162447772852, start=1755162444347674, delta=3425178) [2025-08-14 09:07:27.772868] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:24.998165] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2770636, , BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:24.998165] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:24.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:27.773014] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=2770636][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:27.773026] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] timer task use too much time(end=1755162447773026, start=1755162444998163, delta=2774863) [2025-08-14 09:07:24.737634] EDIAG [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1009709, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975e47 0x27a133d6 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:27.776464] WDIAG ev_invoke_pending (ev.c:2136) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1009709][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:27.776491] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:27.776506] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:24.777209] EDIAG [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=401][errcode=0] LOGGER COST TOO MUCH TIME, cost: 2996135, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x9975947 0x997588d 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:24.777209] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=401] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:27.777905] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2996135][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:27.777924] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:27.777938] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:27.853804] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [861][T1002_ArcSender][T1002][YB42AC4D0007-00063C495E499C16-0-0] [lt=7909435][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:07:27.866287] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:233) [733][T1001_ArcSender][T1001][YB42AC4D0007-00063C495CB99C16-0-0] [lt=8910551][errcode=-4018] try free send task failed(ret=-4018) [2025-08-14 09:07:27.959626] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=798801][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:28.724196] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=252] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:19.765088] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=2000008] current log_task status(palf_id=1001, self="172.77.0.7:2882", log_id=8571657, log_task={header:{begin_lsn:{lsn:7323686009}, end_lsn:{lsn:7323686131}, log_id:8571657, min_scn:{val:1755161749741193000, v:0}, max_scn:{val:1755161749741193000, v:0}, data_len:66, proposal_id:49, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:7323686009}, data_checksum:2222604892, accum_checksum:1531236791, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1755161749745909, freeze_ts:1755161749745914, submit_ts:1755161749745940, flushed_ts:1755161785758706, gen_to_freeze cost time:5, gen_to_submit cost time:31, submit_to_flush cost time:36012766}, ack_info_list=[{member:{server:"172.77.0.7:2882", timestamp:1, flag:0}, last_ack_time_us:1755161853783286, last_flushed_end_lsn:{lsn:7323686131}}]) [2025-08-14 09:07:19.765088] EDIAG [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [822][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=2000008][errcode=0] LOGGER COST TOO MUCH TIME, cost: 8947382, owner: ObLog, click_count: 7, time dist:[FORMAT_END=8947364, ObMalloc#start=2, ObMalloc#end=6, ALLOC_END=0, APPEND_END=10, ALLOC_BLOCK#start=7, ALLOC_BLOCK#end=2], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0xfb85bcd 0xfb858a5 0x993f72e 0xfbc46b8 0x993e2cf 0x993df9d 0xfabac9b 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:22.850912] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fc8b0, caller:mds_tenant_service.cpp:mtl_start:107, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_143, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fcb90, time_wheel:0x7f99375d9990, thread_pool:0x7f99375d9170, is_running:1, time_interval:3.00s, expected_run_time:09:07:22.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:22.850912] EDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5872418, , BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0xf9fb7ca 0x97aedae 0x97ad969 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:28.732564] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=5872418][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:28.732591] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [652][T1001_MdsT][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=-4389] timer task use too much time(end=1755162448732590, start=1755162442850910, delta=5881680) [2025-08-14 09:07:28.736277] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=4] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:07:28.736319] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=40] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:07:28.736365] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:07:28.736379] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:07:28.739257] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:28.745168] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:28.745656] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=5] gc stale ls task succ [2025-08-14 09:07:28.745777] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=4] gc stale ls task succ [2025-08-14 09:07:28.746244] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:07:28.746271] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=26] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:07:28.755591] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:28.755770] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=343] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:28.755809] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:28.755850] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2772] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:28.755882] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:28.756273] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:28.756435] INFO handle (memory_dump.cpp:671) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=22] statistics: tenant_cnt: 6, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 168 270 0 1 2 110 1 0 1 3 7 0 0 1 4 1 1 0 1 5 85 6 0 1 6 13 0 0 1 7 22 2 0 1 8 53 1 0 1 12 17 3 0 1 13 10 1 0 1 14 1 1 0 1 15 1 1 0 1 16 8 7 0 1 18 4 3 0 1 20 2 1 0 1 22 1 1 0 1 24 1 0 0 1 25 6 0 0 500 0 561 274 0 500 7 194 5 0 500 8 41 1 0 500 9 2 1 0 500 10 2 1 0 500 16 57 4 0 500 17 60 7 0 500 22 38 64 0 500 23 27 10 0 508 0 2 7 0 508 8 5 1 0 509 0 2 7 0 509 8 5 1 0 1001 0 147 252 0 1001 2 26 1 0 1001 3 5 0 0 1001 4 1 1 0 1001 5 85 6 0 1001 6 11 0 0 1001 7 30 2 0 1001 8 36 1 0 1001 12 17 3 0 1001 13 3 1 0 1001 14 1 1 0 1001 16 7 4 0 1001 18 4 3 0 1001 20 4 1 0 1001 25 8 0 0 1002 0 263 278 0 1002 2 198 1 0 1002 3 4 0 0 1002 4 1 1 0 1002 5 101 6 0 1002 6 14 0 0 1002 7 6 2 0 1002 8 73 1 0 1002 12 18 3 0 1002 13 7 1 0 1002 14 1 2 0 1002 16 10 7 0 1002 18 9 3 0 1002 20 2 1 0 1002 21 1 0 0 1002 24 1 0 0 1002 25 7 0 0 cost_time: 4062199 [2025-08-14 09:07:28.757295] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:28.757464] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:28.757477] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:28.757599] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:28.771908] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:28.775902] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=22] PNIO [ratelimit] time: 1755162448775900, bytes: 0, bw: 0.000000 MB/s, add_ts: 1003276, add_bytes: 0 [2025-08-14 09:07:28.777393] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:28.778977] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:23.716968] EDIAG [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10958311][errcode=0] LOGGER COST TOO MUCH TIME, cost: 5993851, owner: ObLog, click_count: 5, time dist:[FORMAT_END=5993836, ObMalloc#start=2, ObMalloc#end=6, ALLOC_END=1, APPEND_END=6], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x989c577 0x989bc96 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:23.716968] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10958311] [thread pool stat](pool_stat={pool:0x7f99667cbcf0, last_check_time:1755162443716966, last_idle_time:35052863610, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x7f99667cbcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:35058879414, tenant_id:500}, qsize=-1, current_idle_time=35052863610) [2025-08-14 09:07:28.737036] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:108) [369][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=974434][errcode=-4015] clock out of order(cur_ts=1755162447766708, cur_ts_=1755162447742841, delta=23867) [2025-08-14 09:07:29.724153] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=46] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:29.720737] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [357][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=8005436] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1333) [2025-08-14 09:07:27.768874] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5980106][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.77.0.7:2882", msg={tenant_id:1, srr:[mts=1755162432758728], range_size:1, sender:"172.77.0.7:2882"}) [2025-08-14 09:07:27.768874] EDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:556) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5980106][errcode=-4038] LOGGER COST TOO MUCH TIME, cost: 1962340, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1962326, ObMalloc#start=1, ObMalloc#end=4, ALLOC_END=1, APPEND_END=8], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1b5043ed 0x1b501579 0x1b5009d1 0x1b50387c 0x1b8905de 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:27.745611] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=4026999] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a5998, timer:0x7f9935f8df20, task:0x7f99200f5a78, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr15MediumCheckTaskE", scheduled_time:1755161721762022, pushed_time:0, delay:20000000}) [2025-08-14 09:07:27.745611] EDIAG dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=4026999][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1988223, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1988209, ObMalloc#start=1, ObMalloc#end=4, ALLOC_END=1, APPEND_END=8], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x26188091 0x9756b3d 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:29.736884] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=1988223] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a6418, timer:0x7f994b1dfb70, task:0x7f9920110068, task_type:"N9oceanbase5share22ObIndexUsageReportTaskE", scheduled_time:1755161727099859, pushed_time:0, delay:900000000}) [2025-08-14 09:07:29.736902] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=18] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a6bf8, timer:0x7f994b1dfb70, task:0x7f99201e29f8, task_type:"N9oceanbase5table21ObTableGroupCommitMgr19ObTableGroupOpsTaskE", scheduled_time:1755161732717636, pushed_time:0, delay:1000000}) [2025-08-14 09:07:29.736914] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=12] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a6878, timer:0x7f994b1dfb70, task:0x7f9920110178, task_type:"N9oceanbase5share27ObIndexUsageRefreshConfTaskE", scheduled_time:1755161733727027, pushed_time:0, delay:2000000}) [2025-08-14 09:07:29.736920] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=6] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a6958, timer:0x7f994b1dfb70, task:0x7f9920110810, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1755161734741162, pushed_time:0, delay:5000000}) [2025-08-14 09:07:29.736931] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=11] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a6a38, timer:0x7f994b1dfb70, task:0x7f9920110af8, task_type:"N9oceanbase5share23ObStorageIOUsageRepoter23TenantIOUsageReportTaskE", scheduled_time:1755161734803774, pushed_time:0, delay:5000000}) [2025-08-14 09:07:29.736941] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=10] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a6e98, timer:0x7f994b1dfb70, task:0x7f99201075a8, task_type:"N9oceanbase3omt11ObTenantSrs27TenantSrsUpdatePeriodicTaskE", scheduled_time:1755161734810627, pushed_time:0, delay:0}) [2025-08-14 09:07:29.736954] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=13] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a64f8, timer:0x7f994b1dfb70, task:0x7f992010e1b0, task_type:"N9oceanbase5table20ObTableObjectPoolMgr26ObTableSessEliminationTaskE", scheduled_time:1755161735735271, pushed_time:0, delay:5000000}) [2025-08-14 09:07:29.736960] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=6] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a65d8, timer:0x7f994b1dfb70, task:0x7f992010e1c8, task_type:"N9oceanbase5table20ObTableObjectPoolMgr27ObTableSessSysVarUpdateTaskE", scheduled_time:1755161735797729, pushed_time:0, delay:5000000}) [2025-08-14 09:07:29.736980] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=20] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a66b8, timer:0x7f994b1dfb70, task:0x7f992010e1e0, task_type:"N9oceanbase5table20ObTableObjectPoolMgr28ObTableReqResEliminationTaskE", scheduled_time:1755161735811808, pushed_time:0, delay:5000000}) [2025-08-14 09:07:29.736986] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=6] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a4ff8, timer:0x7f994b1dfb70, task:0x7f9922ff3e50, task_type:"N9oceanbase3sql3dtl23ObDTLIntermResultGCTaskE", scheduled_time:1755161742809616, pushed_time:0, delay:10000000}) [2025-08-14 09:07:29.736996] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=10] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a5b58, timer:0x7f99535c3f30, task:0x7f9922ffee90, task_type:"N9oceanbase7storage21ObTenantFreezeInfoMgr24UpdateLSResvSnapshotTaskE", scheduled_time:1755161744768148, pushed_time:0, delay:10000000}) [2025-08-14 09:07:29.737008] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=12] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a7838, timer:0x7f994b1dfb70, task:0x7f9920110820, task_type:"N9oceanbase10rootserver16ObMViewMdsOpTaskE", scheduled_time:1755161749755625, pushed_time:0, delay:5000000}) [2025-08-14 09:07:29.737014] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=6] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a6db8, timer:0x7f994b1dfb70, task:0x7f99201e33a0, task_type:"N9oceanbase5table20ObTableClientInfoMgr27ObTableClientInfoRetireTaskE", scheduled_time:1755161750768689, pushed_time:0, delay:60000000}) [2025-08-14 09:07:29.737023] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=9] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a4d58, timer:0x7f994b1dfb70, task:0x7f991f004030, task_type:"N9oceanbase8observer20ObTableQueryASyncMgrE", scheduled_time:1755161750817104, pushed_time:0, delay:0}) [2025-08-14 09:07:29.737029] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=6] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a7678, timer:0x7f994b1dfb70, task:0x7f9920110450, task_type:"N9oceanbase10rootserver21ObMLogMaintenanceTaskE", scheduled_time:1755161754754755, pushed_time:0, delay:0}) [2025-08-14 09:07:29.737038] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=9] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a7758, timer:0x7f994b1dfb70, task:0x7f9920110530, task_type:"N9oceanbase10rootserver22ObMViewMaintenanceTaskE", scheduled_time:1755161754754764, pushed_time:0, delay:0}) [2025-08-14 09:07:29.737051] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=13] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a7bb8, timer:0x7f994b1dfb70, task:0x7f9920110628, task_type:"N9oceanbase10rootserver34ObMViewRefreshStatsMaintenanceTaskE", scheduled_time:1755161754754765, pushed_time:0, delay:0}) [2025-08-14 09:07:29.737056] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=5] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a7058, timer:0x7f994b1dfb70, task:0x7f98ebbf7c30, task_type:"N9oceanbase8observer26ObTableLoadResourceManager21ObRefreshAndCheckTaskE", scheduled_time:1755161774744853, pushed_time:0, delay:30000000}) [2025-08-14 09:07:29.737065] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=9] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a6b18, timer:0x7f994b1dfb70, task:0x7f9920110718, task_type:"N9oceanbase10rootserver25ObMViewPushRefreshScnTaskE", scheduled_time:1755161774754765, pushed_time:0, delay:30000000}) [2025-08-14 09:07:29.737078] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=12] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a73d8, timer:0x7f994b1dfb70, task:0x7f9920110748, task_type:"N9oceanbase10rootserver22ObReplicaSafeCheckTaskE", scheduled_time:1755161774754766, pushed_time:0, delay:0}) [2025-08-14 09:07:29.737084] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=6] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a6cd8, timer:0x7f994b1dfb70, task:0x7f99201107e0, task_type:"N9oceanbase10rootserver24ObCollectMvMergeInfoTaskE", scheduled_time:1755161774755618, pushed_time:0, delay:30000000}) [2025-08-14 09:07:29.737094] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=10] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a74b8, timer:0x7f994b1dfb70, task:0x7f9920110730, task_type:"N9oceanbase10rootserver23ObMViewPushSnapshotTaskE", scheduled_time:1755161804754766, pushed_time:0, delay:60000000}) [2025-08-14 09:07:29.737106] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=12] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a7218, timer:0x7f994b1dfb70, task:0x7f99201107f8, task_type:"N9oceanbase10rootserver24ObMViewCleanSnapshotTaskE", scheduled_time:1755161804755624, pushed_time:0, delay:60000000}) [2025-08-14 09:07:29.737112] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=6] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a4e38, timer:0x7f99535f3df0, task:0x7f9922ff6bf0, task_type:"N9oceanbase7obmysql15ObEliminateTaskE", scheduled_time:1755161931284195, pushed_time:0, delay:500000}) [2025-08-14 09:07:29.737122] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=10] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a5df8, timer:0x7f99200ff180, task:0x7f99200ff1c0, task_type:"N9oceanbase8observer18ObTableLoadService15ObHeartBeatTaskE", scheduled_time:1755162386750869, pushed_time:0, delay:10000000}) [2025-08-14 09:07:29.737131] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=9] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a6098, timer:0x7f99200ff180, task:0x7f99200ff208, task_type:"N9oceanbase8observer18ObTableLoadService25ObClientTaskAutoAbortTaskE", scheduled_time:1755162386758966, pushed_time:0, delay:1000000}) [2025-08-14 09:07:29.737144] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=13] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a4498, timer:0x7f9966a63f10, task:0x7f9922e05010, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr11TableGCTaskE", scheduled_time:1755162392785176, pushed_time:0, delay:20000}) [2025-08-14 09:07:29.737149] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=5] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a6178, timer:0x7f99200ff180, task:0x7f99200ff220, task_type:"N9oceanbase8observer18ObTableLoadService21ObClientTaskPurgeTaskE", scheduled_time:1755162393751914, pushed_time:0, delay:1000000}) [2025-08-14 09:07:29.737155] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=6] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a5ed8, timer:0x7f99200ff180, task:0x7f99200ff1d8, task_type:"N9oceanbase8observer18ObTableLoadService8ObGCTaskE", scheduled_time:1755162408748371, pushed_time:0, delay:30000000}) [2025-08-14 09:07:29.737160] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=5] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a5d18, timer:0x7f99200ff180, task:0x7f99200ff1a8, task_type:"N9oceanbase8observer18ObTableLoadService17ObCheckTenantTaskE", scheduled_time:1755162411764702, pushed_time:0, delay:1000000}) [2025-08-14 09:07:29.737169] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=9] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a4578, timer:0x7f9966a63f10, task:0x7f9922e05028, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr17RefreshConfigTaskE", scheduled_time:1755162421758060, pushed_time:0, delay:10000000}) [2025-08-14 09:07:29.737181] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=12] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a4c78, timer:0x7f99329e1b10, task:0x7f98e4e091f8, task_type:"N9oceanbase5table21ObTTLAllTaskSchedulerE", scheduled_time:1755162423747124, pushed_time:0, delay:15000000}) [2025-08-14 09:07:29.737186] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=5] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a72f8, timer:0x7f9922d738c0, task:0x7f99231fe2b8, task_type:"N9oceanbase8observer32ObTenantTabletMetaTableCheckTaskE", scheduled_time:1755162627881458, pushed_time:0, delay:0}) [2025-08-14 09:07:29.737196] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=10] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a57d8, timer:0x7f9935f89f20, task:0x7f99200f5a68, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr24TabletUpdaterRefreshTaskE", scheduled_time:1755162650773194, pushed_time:0, delay:300000000}) [2025-08-14 09:07:29.737209] INFO dump_info (ob_timer_service.cpp:731) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=12] dump queue token(this=0x7f99230b0af0, token={this:0x7f991b7a6258, timer:0x7f994b1dfb70, task:0x7f9920106f88, task_type:"N9oceanbase6common28ObOptStatMonitorFlushAllTaskE", scheduled_time:1755222026435491, pushed_time:0, delay:86400000000}) [2025-08-14 09:07:29.739416] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:29.744161] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:29.745117] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:29.745856] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=16] gc stale ls task succ [2025-08-14 09:07:29.746083] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=17] gc stale ls task succ [2025-08-14 09:07:29.748774] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:29.755763] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:29.756267] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:29.757571] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:29.761396] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:29.761443] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:29.761471] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:29.761497] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:29.761523] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:29.763139] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:29.763184] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=26] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:29.763214] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:29.764404] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:15.775173] EDIAG [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [561][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] LOGGER COST TOO MUCH TIME, cost: 11969183, owner: ObLog, click_count: 5, time dist:[ALLOC_BLOCK#start=3, ALLOC_BLOCK#end=5969713, FORMAT_END=5999465, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x987408f 0x9873856 0x987344f 0x9873355 0x973b640 0x973b132 0x973a32d 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:27.745492] WDIAG [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:304) [376][Blacklist][T0][Y0-0000000000000000-0-0] [lt=7982072][errcode=-4389] blacklist_loop cost too much time(cost_time=2995989) [401]OB_MALLOC COST TOO MUCH TIME, cost_time=964101, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=964097], seq:[0] [332]OB_MALLOC COST TOO MUCH TIME, cost_time=964364, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=964361], seq:[0] [279]OB_MALLOC COST TOO MUCH TIME, cost_time=964440, tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, size=32896, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=964437], seq:[0] [303]OB_MALLOC COST TOO MUCH TIME, cost_time=965369, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=965360], seq:[0] [808]OB_MALLOC COST TOO MUCH TIME, cost_time=965357, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=965346, ALLOC_CHUNK_END=5], seq:[0, 1] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=965431, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=965419, ALLOC_CHUNK_END=4], seq:[0, 1] [807]OB_MALLOC COST TOO MUCH TIME, cost_time=964721, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=964711, ALLOC_CHUNK_END=4], seq:[0, 1] [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=963982, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=963971, ALLOC_CHUNK_END=4], seq:[0, 1] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=959533, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=959524, ALLOC_CHUNK_END=5], seq:[0, 1] [660]OB_MALLOC COST TOO MUCH TIME, cost_time=959023, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=959014, ALLOC_CHUNK_END=4], seq:[0, 1] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=959188, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=959178, ALLOC_CHUNK_END=4], seq:[0, 1] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=958237, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=958227, ALLOC_CHUNK_END=5], seq:[0, 1] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=955762, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=955751, ALLOC_CHUNK_END=5], seq:[0, 1] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=955247, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=955237, ALLOC_CHUNK_END=5], seq:[0, 1] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=950757, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=950749, ALLOC_CHUNK_END=3], seq:[0, 1] [833]OB_MALLOC COST TOO MUCH TIME, cost_time=835996, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=835988, ALLOC_CHUNK_END=4], seq:[0, 1] [553]OB_MALLOC COST TOO MUCH TIME, cost_time=815785, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=815776, ALLOC_CHUNK_END=4], seq:[0, 1] [718]OB_MALLOC COST TOO MUCH TIME, cost_time=806516, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=806507, ALLOC_CHUNK_END=5], seq:[0, 1] [402]OB_MALLOC COST TOO MUCH TIME, cost_time=770102, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 2, time dist:[ALLOC_CHUNK_END=770086, ALLOC_CHUNK_END=12], seq:[0, 1] [856]OB_MALLOC COST TOO MUCH TIME, cost_time=970734, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=965441, WASH_BLOCK_END=5293], seq:[0, 1, 2] [807]OB_MALLOC COST TOO MUCH TIME, cost_time=970062, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=964733, WASH_BLOCK_END=5329], seq:[0, 1, 2] [1027]OB_MALLOC COST TOO MUCH TIME, cost_time=969295, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=963989, WASH_BLOCK_END=5306], seq:[0, 1, 2] [396]OB_MALLOC COST TOO MUCH TIME, cost_time=964879, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=959550, WASH_BLOCK_END=5329], seq:[0, 1, 2] [658]OB_MALLOC COST TOO MUCH TIME, cost_time=960894, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=958247, WASH_BLOCK_END=2646], seq:[0, 1, 2] [382]OB_MALLOC COST TOO MUCH TIME, cost_time=958429, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=955770, WASH_BLOCK_END=2659], seq:[0, 1, 2] [393]OB_MALLOC COST TOO MUCH TIME, cost_time=957946, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=955257, WASH_BLOCK_END=2689], seq:[0, 1, 2] [321]OB_MALLOC COST TOO MUCH TIME, cost_time=953458, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=950766, WASH_BLOCK_END=2692], seq:[0, 1, 2] [833]OB_MALLOC COST TOO MUCH TIME, cost_time=838712, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=836003, WASH_BLOCK_END=2709], seq:[0, 1, 2] [553]OB_MALLOC COST TOO MUCH TIME, cost_time=818522, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=815795, WASH_BLOCK_END=2727], seq:[0, 1, 2] [718]OB_MALLOC COST TOO MUCH TIME, cost_time=809276, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=806523, WASH_BLOCK_END=2753], seq:[0, 1, 2] [402]OB_MALLOC COST TOO MUCH TIME, cost_time=772873, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=770109, WASH_BLOCK_END=2764], seq:[0, 1, 2] [472]OB_MALLOC COST TOO MUCH TIME, cost_time=962459, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 3, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=959207, WASH_BLOCK_END=3252], seq:[0, 1, 2] [401]OB_MALLOC COST TOO MUCH TIME, cost_time=1978794, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=66, WASH_BLOCK_END=976241, ALLOC_BLOCK#start=30403, ALLOC_BLOCK#end=964126], seq:[0, 1, 2, 3, 4] [2025-08-14 09:07:28.757500] EDIAG [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1978820, owner: ObLog, click_count: 5, time dist:[FORMAT_END=7, ObMalloc#start=0, ObMalloc#end=1978810, ALLOC_END=2, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x27a1f867 0x27a135c3 0x9975aa6 0x27add893 0x997556f 0x27aeea59 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [332]OB_MALLOC COST TOO MUCH TIME, cost_time=972786, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=964385], seq:[0, 1] [2025-08-14 09:07:28.747071] EDIAG do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=4965112][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1989945, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1017141, ObMalloc#start=0, ObMalloc#end=972802, ALLOC_END=1, APPEND_END=1], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x202c530b 0x202c6888 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:28.770849] EDIAG [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=3028506][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1970597, owner: ObLog, click_count: 6, time dist:[FORMAT_END=995041, ObMalloc#start=1, ObMalloc#end=969303, ObMalloc#start=17, ObMalloc#end=6136, FREE_END=99], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1f832f91 0x97c36fe 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [660]OB_MALLOC COST TOO MUCH TIME, cost_time=970874, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=959031, WASH_BLOCK_END=5858, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=5984], seq:[0, 1, 2, 3, 4] [2025-08-14 09:07:27.716089] EDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][YB42AC4D0007-00063C495F8A1988-0-0] [lt=7026212][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 3024139, owner: ObLog, click_count: 7, time dist:[FORMAT_END=2047395, ObMalloc#start=0, ObMalloc#end=970069, ObMalloc#start=18, ObMalloc#end=6144, ALLOC_END=512, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9821580 0x9820ba6 0x9820ad8 0x98208f6 0x96ecc48 0x96c9b87 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:29.719938] EDIAG [SHARE] run1 (ob_dynamic_thread_pool.cpp:615) [277][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=5993851][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1023995, owner: ObLog, click_count: 6, time dist:[FORMAT_END=1022694, ObMalloc#start=1, ObMalloc#end=701, ObMalloc#start=62, ObMalloc#end=32, FREE_END=505], seq:[0, 1, 2, 3, 4, 5], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x989c577 0x989bc96 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [276]OB_MALLOC COST TOO MUCH TIME, cost_time=978388, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079936, owner: ALLOC_BLOCK, click_count: 1, time dist:[ALLOC_CHUNK_END=978373], seq:[0] [2025-08-14 09:07:28.757500] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=7] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:30.739353] WDIAG ev_invoke_pending (ev.c:2136) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=1978820][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:30.739372] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=10] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:28.747071] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=4965112] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:07:30.744511] WDIAG ev_invoke_pending (ev.c:2136) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=964066][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:30.744561] WDIAG ev_invoke_pending (ev.c:2136) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=971231][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:26.752380] EDIAG [SHARE] refresh (ob_task_define.cpp:396) [280][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3987213, , BACKTRACE: 0x26a1806d 0x971dfed 0x971d547 0x971d486 0x971d2ad 0x1fd6cd17 0x96c37e0 0x1fda2168 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [303]OB_MALLOC COST TOO MUCH TIME, cost_time=978883, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=965377], seq:[0, 1] [2025-08-14 09:07:28.756481] EDIAG [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1986658, owner: ObLog, click_count: 5, time dist:[FORMAT_END=1007764, ObMalloc#start=1, ObMalloc#end=978891, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x99daaa6 0x99da47c 0xf9845d6 0xf9b9f47 0xf9b9e23 0x9710595 0xf98de83 0x96b2c9c 0x96b1c88 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [808]OB_MALLOC COST TOO MUCH TIME, cost_time=988896, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 5, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=965374, WASH_BLOCK_END=5370, ALLOC_BLOCK#start=1, ALLOC_BLOCK#end=18147], seq:[0, 1, 2, 3, 4] [2025-08-14 09:07:29.737221] EDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 1017468, owner: ObLog, click_count: 5, time dist:[FORMAT_END=28547, ObMalloc#start=1, ObMalloc#end=988919, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x9757ced 0x97569df 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:25.844605] EDIAG [ARCHIVE] do_thread_task_ (ob_archive_timer.cpp:125) [862][T1002_ArcTimer][T1002][YB42AC4D0007-00063C495E599C16-0-0] [lt=1912062][errcode=0] LOGGER COST TOO MUCH TIME, cost: 4904660, owner: ObLog, click_count: 9, time dist:[ALLOC_BLOCK#start=913180, ALLOC_BLOCK#end=89, ALLOC_BLOCK#start=998947, ALLOC_BLOCK#end=82, ALLOC_BLOCK#start=12846, ALLOC_BLOCK#end=1, FORMAT_END=2979514, ALLOC_END=1, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6, 7, 8], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0x11406c8f 0x114063aa 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:27.751076] EDIAG [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7900898][errcode=0] LOGGER COST TOO MUCH TIME, cost: 3001109, owner: ObLog, click_count: 7, time dist:[FORMAT_END=2011721, ObMalloc#start=0, ObMalloc#end=970750, ObMalloc#start=29, ObMalloc#end=18607, ALLOC_END=2, APPEND_END=0], seq:[0, 1, 2, 3, 4, 5, 6], BACKTRACE: 0x26a1806d 0x9649525 0x9648af4 0x9648a32 0x9648851 0x1b52e1b7 0x996d441 0x996cfd8 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:27.716089] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][YB42AC4D0007-00063C495F8A1988-0-0] [lt=7026212][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a4b98, timer:0x7f99535c9ed0, task:0x7f99004c9710, task_type:"N9oceanbase5share32ObPluginVectorIndexLoadSchedulerE", scheduled_time:1755162401727767, pushed_time:1755162401727844, delay:10000000}, delay_in_thread_pool=25021576, threashold=500000) [2025-08-14 09:07:30.745109] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:26.752380] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [280][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:117, tid:660}]) [2025-08-14 09:07:30.745356] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] timer task use too much time(end=1755162450745355, start=1755162449772403, delta=972952) [2025-08-14 09:07:28.756481] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=17] [MEMORY] tenant: 1, limit: 2147483648 hold: 1013174272 cache_hold: 56623104 cache_used: 56623104 cache_item_count: 27 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 444993536 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 230686720 limit= 9223372036854775807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 1163264 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 483328 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 51617792 limit= 9223372036854775807 [MEMORY] ctx_id= WORK_AREA hold_bytes= 1458176 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 14204928 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 111149056 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35651584 limit= 429496720 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 8003584 limit= 9223372036854775807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 147456 limit= 9223372036854775807 [MEMORY] ctx_id= PS_CACHE_CTX_ID hold_bytes= 65536 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 14827520 limit= 9223372036854775807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 3481600 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 3964928 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 212992 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_NORMAL_CTX_ID hold_bytes= 884736 limit= 9223372036854775807 [2025-08-14 09:07:30.754917] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:30.755419] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:30.755481] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:29.737221] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6b18, timer:0x7f994b1dfb70, task:0x7f9920110718, task_type:"N9oceanbase10rootserver25ObMViewPushRefreshScnTaskE", scheduled_time:1755161774754765, pushed_time:0, delay:30000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=674982455, threashold=1000000, tenant_id=1002) [2025-08-14 09:07:30.756258] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=1017468][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a73d8, timer:0x7f994b1dfb70, task:0x7f9920110748, task_type:"N9oceanbase10rootserver22ObReplicaSafeCheckTaskE", scheduled_time:1755161774754766, pushed_time:0, delay:0}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=676001491, threashold=1000000, tenant_id=1002) [2025-08-14 09:07:30.756273] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6cd8, timer:0x7f994b1dfb70, task:0x7f99201107e0, task_type:"N9oceanbase10rootserver24ObCollectMvMergeInfoTaskE", scheduled_time:1755161774755618, pushed_time:0, delay:30000000}, running_task_token={this:0x7f991b7a6338, timer:0x7f994b1dfb70, task:0x7f9920106fa0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1755161727089071, pushed_time:1755161732809627, delay:900000000}, total_delay=676000655, threashold=1000000, tenant_id=1002) [2025-08-14 09:07:30.756283] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a5df8, timer:0x7f99200ff180, task:0x7f99200ff1c0, task_type:"N9oceanbase8observer18ObTableLoadService15ObHeartBeatTaskE", scheduled_time:1755162386750869, pushed_time:0, delay:10000000}, running_task_token={this:0x7f991b7a5fb8, timer:0x7f99200ff180, task:0x7f99200ff1f0, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162378750218, pushed_time:1755162410766348, delay:1000000}, total_delay=64005414, threashold=1000000, tenant_id=1002) [2025-08-14 09:07:30.756291] WDIAG pop_task (ob_timer_service.cpp:565) [808][T1002_TimerSvr][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f991b7a6098, timer:0x7f99200ff180, task:0x7f99200ff208, task_type:"N9oceanbase8observer18ObTableLoadService25ObClientTaskAutoAbortTaskE", scheduled_time:1755162386758966, pushed_time:0, delay:1000000}, running_task_token={this:0x7f991b7a5fb8, timer:0x7f99200ff180, task:0x7f99200ff1f0, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1755162378750218, pushed_time:1755162410766348, delay:1000000}, total_delay=63997325, threashold=1000000, tenant_id=1002) [2025-08-14 09:07:30.744209] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=7519] finish to resize log external storage handler(current_ts=1755162450735598, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-08-14 09:07:30.756340] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [528][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=12100] snapshot_traffic_info once [2025-08-14 09:07:25.844605] INFO [ARCHIVE] do_thread_task_ (ob_archive_timer.cpp:125) [862][T1002_ArcTimer][T1002][YB42AC4D0007-00063C495E599C16-0-0] [lt=1912062] archive round not in doing status, just skip(key={incarnation:-1, dest_id:-1, round:-1}, state={status:"INVALID"}) [2025-08-14 09:07:27.751076] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7900898] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:07:30.756745] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=3001109] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:07:30.756762] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:30.757090] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:30.757131] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [726][T1001_ObXAHbWor][T1001][Y0-0000000000000000-0-0] [lt=3975378] XA scheduler heartbeat task statistics(avg_time=1) [2025-08-14 09:07:30.758410] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:30.758466] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:30.758487] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:30.758652] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:30.749341] WDIAG alarm_if_necessary (ob_timer_service.cpp:143) [807][T1002_TimerWK3][T1002][YB42AC4D0007-00063C495F8A1988-0-0] [lt=3024139][errcode=-4389] timer task's elapsed_time is greater than period(token={this:0x7f991b7a4b98, timer:0x7f99535c9ed0, task:0x7f99004c9710, task_type:"N9oceanbase5share32ObPluginVectorIndexLoadSchedulerE", scheduled_time:1755162401727767, pushed_time:1755162401727844, delay:10000000}, elapsed_time=20966668, period=10000000) [2025-08-14 09:07:30.732020] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:233) [529][T1_RFLWorker][T1][YB42AC4D0007-00063C495B199C16-0-0] [lt=2915] ObRemoteFetchWorker is running(thread_index=0) [2025-08-14 09:07:29.729489] EDIAG [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11960654][errcode=0] LOGGER COST TOO MUCH TIME, cost: 1029586, owner: ObLog, click_count: 3, time dist:[FORMAT_END=1029580, ALLOC_END=1, APPEND_END=4], seq:[0, 1, 2], BACKTRACE: 0x26a1806d 0x982b779 0x982af25 0x982ae74 0x982ad06 0xfebfd41 0x94edd90 0x94edb97 0x97a8fbd 0x97a8cc7 0x97a88d2 0x97a7481 0x97a6a43 0x97a42c5 0x97a1536 0x2668daa5 0x2668bbb2 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:29.867425] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:30.760190] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] timer task use too much time(end=1755162450760189, start=1755162449867422, delta=892767) [2025-08-14 09:07:30.732026] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:288) [884][T1002_ArcFetche][T1002][YB42AC4D0007-00063C495ED99C16-0-0] [lt=1024487] ObArchiveFetcher is running(thread_index=1) [2025-08-14 09:07:29.729489] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:740) [729][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11960654] get_max_applied_scn(scn={val:1755161719775179002, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:45, palf_committed_end_lsn_:{lsn:8461098908}, palf_committed_end_scn_:{val:1755161719775179002, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1755161719775179002, v:0}}) [2025-08-14 09:07:30.739140] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=11897] kill and revert session(conn.sessid_=3221590537, proxy_sessid=0, ret=0) [2025-08-14 09:07:30.764450] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:30.764473] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:30.764530] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:30.769948] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=24] [COMPACTION DAG STATUS] (start_time=1755162431754305, end_time=1755162450769946, dag_status={}) [2025-08-14 09:07:30.769979] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=28] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:30.769995] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:30.770005] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:30.770014] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:30.770022] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:30.770035] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:30.770043] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:30.770053] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:30.770062] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2878) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-08-14 09:07:30.770073] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3697) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-08-14 09:07:30.770082] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4445) [500][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-08-14 09:07:30.770960] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:30.770985] WDIAG ev_invoke_pending (ev.c:2136) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15000][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [276]OB_MALLOC COST TOO MUCH TIME, cost_time=1003755, tenant_id=500, label=Logger, ctx_id=10, prio=0, size=2079744, owner: ObMalloc, click_count: 2, time dist:[ALLOC_BLOCK#start=0, ALLOC_BLOCK#end=978396], seq:[0, 1] [2025-08-14 09:07:26.966040] EDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=4265027][errcode=-4389] LOGGER COST TOO MUCH TIME, cost: 3799244, owner: ObLog, click_count: 5, time dist:[FORMAT_END=2795474, ObMalloc#start=0, ObMalloc#end=1003765, ALLOC_END=1, APPEND_END=3], seq:[0, 1, 2, 3, 4], BACKTRACE: 0x26a1806d 0x95ef847 0x95eecf4 0x95eec2b 0x95eea40 0x9757ced 0x97569df 0x2668daa5 0x2668bd3e 0x7f9973e081cf 0x7f9973a39dd3 [2025-08-14 09:07:26.966040] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=4265027][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc4658, timer:0x7f9970f40f80, task:0x564d6d07a468, task_type:"N9oceanbase8observer23ObSchemaReleaseTimeTaskE", scheduled_time:1755161716573087, pushed_time:0, delay:0}, running_task_token={this:0x7f9970fc5fb8, timer:0x7f9970f40f80, task:0x564d727182f0, task_type:"N9oceanbase5share16ObAshRefreshTaskE", scheduled_time:1755161611717613, pushed_time:1755161614757742, delay:120000000}, total_delay=730392951, threashold=1000000, tenant_id=500) [2025-08-14 09:07:30.771050] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=3799244][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc4578, timer:0x7f9970f40f80, task:0x564d6d07a498, task_type:"N9oceanbase8observer26ObRemoteMasterRsUpdateTaskE", scheduled_time:1755162028742204, pushed_time:0, delay:600000000}, running_task_token={this:0x7f9970fc5fb8, timer:0x7f9970f40f80, task:0x564d727182f0, task_type:"N9oceanbase5share16ObAshRefreshTaskE", scheduled_time:1755161611717613, pushed_time:1755161614757742, delay:120000000}, total_delay=422028845, threashold=1000000, tenant_id=500) [2025-08-14 09:07:30.771068] WDIAG pop_task (ob_timer_service.cpp:565) [276][TimerSvr][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] timer task too much delay because the same timer has another running task(token={this:0x7f9970fc5618, timer:0x564d6efbde60, task:0x564d6efbde20, task_type:"N9oceanbase12blocksstable14ObBlockManager13MarkBlockTaskE", scheduled_time:1755162196749035, pushed_time:0, delay:30000000}, running_task_token={this:0x7f9970fc5538, timer:0x564d6efbde60, task:0x564d6efbde38, task_type:"N9oceanbase12blocksstable14ObBlockManager19InspectBadBlockTaskE", scheduled_time:1755162049755883, pushed_time:1755162166774569, delay:1000000}, total_delay=254022033, threashold=1000000, tenant_id=500) [2025-08-14 09:07:30.771647] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:30.771856] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=15059] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"172.77.0.7:2882", switch_to_leader_ts:-1}) [2025-08-14 09:07:30.771873] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:30.772888] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:30.773033] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [800][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] [COMPACTION DAG STATUS] (start_time=1755162441722576, end_time=1755162450773032, dag_status={}) [2025-08-14 09:07:30.773064] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [686][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] [COMPACTION DAG STATUS] (start_time=1755162431760114, end_time=1755162450773064, dag_status={}) [2025-08-14 09:07:30.773866] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4021563] [IO STATUS CALLBACK](tenant_id=1002, io_status=runner[0]: tid=810, queue_count=0 | runner[1]: tid=1066, queue_count=0 | runner[2]: tid=1067, queue_count=0) [2025-08-14 09:07:30.773917] INFO [COMMON] print_io_status (ob_io_manager.cpp:2768) [416][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS CALLBACK](tenant_id=500, io_status=runner[0]: tid=415, queue_count=0) [2025-08-14 09:07:30.778597] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [393][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:30.778829] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [382][RpcIO][T0][Y0-0000000000000000-0-0] [lt=564] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:30.791136] INFO eloop_run (eloop.c:145) [321][pnio2][T0][Y0-0000000000000000-0-0] [lt=959660] PNIO [ratelimit] time: 1755162450791134, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007617, add_bytes: 0 [2025-08-14 09:07:30.845151] INFO try_report (ob_tc_stat.cpp:35) [419][][T0][Y0-0000000000000000-0-0] [lt=4936] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1001m0:13,x,0 t1001m0gd:1,x,0 t1001m1:13,x,0 t1001m1gd:1,x,0 t1001m2:13,x,x t1001m2gd:1,x,0 t1002m0:13,x,0 t1002m0gd:1,x,0 t1002m1:13,x,0 t1002m1gd:1,x,0 t1002m2:13,x,x t1002m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: |q0: |q1: [2025-08-14 09:07:30.845644] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=413][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-08-14 09:07:30.845892] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [280][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=3987213][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:273, tid:374}]) [2025-08-14 09:07:30.846183] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=1425][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:30.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:30.856837] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:30.860275] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [656][T1001_CoordTR][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd830, caller:ob_failure_detector.cpp:mtl_start:77, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f99571fdb10, time_wheel:0x7f99375bf990, thread_pool:0x7f99375bf170, is_running:1, time_interval:1.00s, expected_run_time:09:07:30.793, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:30.875496] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7fbbb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7280e10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:30.865, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:30.885590] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=46][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72860b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288c90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:30.873, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:30.945695] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:30.864, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:30.946249] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:30.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:30.956896] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:30.964572] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:30.855, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:30.965499] WDIAG easy_baseth_pool_monitor_func (easy_baseth_pool.c:265) [404][EasyBasethPoolM][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] EASY SLOW: thread: 7f9953df7300, lastrun: 1755162380.744619 cost: 70.220878 loop:26707, slowcnt: 607 [2025-08-14 09:07:30.990471] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=22][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:30.983, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.013435] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:31.548, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.013471] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:31.913, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.024327] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:31.100, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.045734] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:30.964, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.046313] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:30.993, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.056958] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:31.093927] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:31.893, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.145780] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:31.648, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.146376] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:31.939, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.157021] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:31.164676] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:31.552, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.214641] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:31.204, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.245845] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:31.164, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.246439] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:31.193, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.257084] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:31.264916] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:31.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.264950] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:31.259, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.275939] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:31.260, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.306127] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f996f40bbd0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f996f40beb0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:31.292, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.338177] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=43][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7faa30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7281890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:31.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:31.338225] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=45][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72842b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7289b90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:31.329, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:31.345872] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:31.264, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.346504] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:31.293, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.357148] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:31.364816] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:31.255, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.445908] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:31.364, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.446569] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:31.393, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.457211] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:31.516443] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:31.505, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.516481] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:31.509, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.527509] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [660][T1001_ElectTime][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fdc10, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571fdef0, time_wheel:0x7f995359d990, thread_pool:0x7f995359d170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:31.510, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.545972] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:31.464, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.546600] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:31.493, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.557274] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:31.564953] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:31.455, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.596833] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc570, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fc850, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:31.589, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.629787] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eb1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7eef90, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:31.614, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:31.629823] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f994b7eaf30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f994b7ef110, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:31.614, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:31.646036] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:31.564, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.646662] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:31.593, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.657369] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:31.717782] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd4f0, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fd7d0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:31.704, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.724424] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [398][BatchIO][T0][Y0-0000000000000000-0-0] [lt=362] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:31.738972] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [387][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5557] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:31.740086] INFO do_work (ob_rl_mgr.cpp:708) [332][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=1989945] swc wakeup.(stat_period_=1000000, ready=false) [2025-08-14 09:07:30.760683] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [381][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1962340] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1755161547741086], gts:1755161547738516000, latest_srr:[mts=1755162432758728]}) [2025-08-14 09:07:31.241989] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f87f0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f8ad0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:31.233, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.741488] WDIAG [OCCAM] ~ObOccamTimeGuard (ob_occam_time_guard.h:271) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=498741][errcode=0] cost too much time:ob_occam_timer.h:virtual void oceanbase::common::occam::ObOccamTimerTask::runTimerTask(), (this=|threshold=100.00ms|start at 09:07:31.241|207=499.50ms|210=0us|222=1us|total=499.55ms) [2025-08-14 09:07:31.741539] WDIAG [STORAGE.TRANS] scan (ob_time_wheel.cpp:283) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=51][errcode=-4389] timer task use too much time(end=1755162451741539, start=1755162451241987, delta=499552) [2025-08-14 09:07:31.741553] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] task scheduled too delay(scheduled_delay=496918, expected_run_ts_us_=1755162451244633, ObClockGenerator::getRealClock()=1755162451741552, ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:31.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.741586] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=33][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:31.244, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.741649] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=1986658] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 444993536 used= 389804064 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 70 washed_blocks= 287 washed_size= 70221824 [MEMORY] request_cached_chunk_cnt= 6 [MEMORY] hold= 41963520 used= 41943040 count= 1 avg_used= 41943040 block_cnt= 1 chunk_cnt= 1 mod=LogGroupBuffer [MEMORY] hold= 41963520 used= 41943072 count= 1 avg_used= 41943072 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 29130752 used= 29116416 count= 14 avg_used= 2079744 block_cnt= 14 chunk_cnt= 14 mod=CACHE_MAP_NODE [MEMORY] hold= 25640960 used= 25394048 count= 37 avg_used= 686325 block_cnt= 37 chunk_cnt= 27 mod=MysqlRequesReco [MEMORY] hold= 20615168 used= 20545536 count= 4 avg_used= 5136384 block_cnt= 4 chunk_cnt= 4 mod=ServerObjecPool [MEMORY] hold= 16908288 used= 16777728 count= 16 avg_used= 1048608 block_cnt= 16 chunk_cnt= 16 mod=DiagnosticInfos [MEMORY] hold= 16797696 used= 16777336 count= 1 avg_used= 16777336 block_cnt= 1 chunk_cnt= 1 mod=MrgMemPoolChk [MEMORY] hold= 16083712 used= 15926225 count= 112 avg_used= 142198 block_cnt= 38 chunk_cnt= 9 mod=OmtTenant [MEMORY] hold= 10510336 used= 10491392 count= 1 avg_used= 10491392 block_cnt= 1 chunk_cnt= 1 mod=ASHListBuffer [MEMORY] hold= 9457664 used= 9437784 count= 1 avg_used= 9437784 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 8777728 used= 8760064 count= 2 avg_used= 4380032 block_cnt= 2 chunk_cnt= 2 mod=FreeTbltStream [MEMORY] hold= 8540160 used= 8519680 count= 1 avg_used= 8519680 block_cnt= 1 chunk_cnt= 1 mod=ArcFetchQueue [MEMORY] hold= 8540160 used= 8519680 count= 1 avg_used= 8519680 block_cnt= 1 chunk_cnt= 1 mod=RCSrv [MEMORY] hold= 6856704 used= 6788160 count= 9 avg_used= 754240 block_cnt= 9 chunk_cnt= 9 mod=ResourceGroup [MEMORY] hold= 6438912 used= 5701632 count= 90 avg_used= 63351 block_cnt= 90 chunk_cnt= 30 mod=[T]char [MEMORY] hold= 5537792 used= 4850976 count= 169 avg_used= 28704 block_cnt= 169 chunk_cnt= 31 mod=DiagnosticInfo [MEMORY] hold= 5386240 used= 5368320 count= 1 avg_used= 5368320 block_cnt= 1 chunk_cnt= 1 mod=RespTimeColl [MEMORY] hold= 4943872 used= 4915456 count= 2 avg_used= 2457728 block_cnt= 2 chunk_cnt= 2 mod=HashBuckDmId [MEMORY] hold= 4739072 used= 4718712 count= 1 avg_used= 4718712 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmChe [MEMORY] hold= 4472832 used= 2529072 count= 546 avg_used= 4632 block_cnt= 132 chunk_cnt= 55 mod=WorkerMap [MEMORY] hold= 4235264 used= 4194304 count= 2 avg_used= 2097152 block_cnt= 2 chunk_cnt= 2 mod=SHARE_BLOCK [MEMORY] hold= 3865600 used= 3705600 count= 800 avg_used= 4632 block_cnt= 800 chunk_cnt= 13 mod=CkptDgnMem [MEMORY] hold= 3865600 used= 3705600 count= 800 avg_used= 4632 block_cnt= 800 chunk_cnt= 13 mod=CkptDgnMemCU [MEMORY] hold= 3772416 used= 3751944 count= 1 avg_used= 3751944 block_cnt= 1 chunk_cnt= 1 mod=IteratorStorage [MEMORY] hold= 3723264 used= 3698704 count= 2 avg_used= 1849352 block_cnt= 2 chunk_cnt= 2 mod=LogDIOAligned [MEMORY] hold= 3162112 used= 3145808 count= 2 avg_used= 1572904 block_cnt= 2 chunk_cnt= 2 mod=TenantDLMgr [MEMORY] hold= 3145728 used= 2228224 count= 128 avg_used= 17408 block_cnt= 128 chunk_cnt= 6 mod=SqlDtlQueue [MEMORY] hold= 3047424 used= 3013468 count= 10 avg_used= 301346 block_cnt= 10 chunk_cnt= 2 mod=LDIOSetup [MEMORY] hold= 3031040 used= 3013952 count= 1 avg_used= 3013952 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2850816 used= 2832384 count= 12 avg_used= 236032 block_cnt= 12 chunk_cnt= 10 mod=DiagnosticSum [MEMORY] hold= 2379776 used= 2359608 count= 1 avg_used= 2359608 block_cnt= 1 chunk_cnt= 1 mod=FastFrezCkr [MEMORY] hold= 2379776 used= 2359608 count= 1 avg_used= 2359608 block_cnt= 1 chunk_cnt= 1 mod=MediumTabletMap [MEMORY] hold= 2379776 used= 2359608 count= 1 avg_used= 2359608 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMonMap [MEMORY] hold= 2375680 used= 2359536 count= 2 avg_used= 1179768 block_cnt= 2 chunk_cnt= 2 mod=HashBuckLCSta [MEMORY] hold= 2248704 used= 2228224 count= 1 avg_used= 2228224 block_cnt= 1 chunk_cnt= 1 mod=LogIOCb [MEMORY] hold= 2198720 used= 416700 count= 25277 avg_used= 16 block_cnt= 272 chunk_cnt= 3 mod=Number [MEMORY] hold= 2197056 used= 2174312 count= 77 avg_used= 28237 block_cnt= 77 chunk_cnt= 2 mod=TabletMap [MEMORY] hold= 2064384 used= 1835008 count= 28 avg_used= 65536 block_cnt= 28 chunk_cnt= 11 mod=MajorMergeMgr [MEMORY] hold= 1843200 used= 1770696 count= 9 avg_used= 196744 block_cnt= 9 chunk_cnt= 5 mod=DagMap [MEMORY] hold= 1571328 used= 1566720 count= 24 avg_used= 65280 block_cnt= 24 chunk_cnt= 9 mod=ResultSet [MEMORY] hold= 1294336 used= 1280384 count= 2 avg_used= 640192 block_cnt= 2 chunk_cnt= 2 mod=TransTimeWheel [MEMORY] hold= 1294336 used= 1280384 count= 2 avg_used= 640192 block_cnt= 2 chunk_cnt= 1 mod=XATimeWheel [MEMORY] hold= 1187840 used= 1179768 count= 1 avg_used= 1179768 block_cnt= 1 chunk_cnt= 1 mod=RewriteRuleMap [MEMORY] hold= 1187840 used= 1179768 count= 1 avg_used= 1179768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsCache [MEMORY] hold= 1187840 used= 1179768 count= 1 avg_used= 1179768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPlanCac [MEMORY] hold= 1187840 used= 1179768 count= 1 avg_used= 1179768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsInfo [MEMORY] hold= 1179648 used= 1142528 count= 24 avg_used= 47605 block_cnt= 16 chunk_cnt= 11 mod=CommSysVarFac [MEMORY] hold= 1130496 used= 1126688 count= 6 avg_used= 187781 block_cnt= 6 chunk_cnt= 6 mod=IoControl [MEMORY] hold= 1114112 used= 1079296 count= 131 avg_used= 8238 block_cnt= 80 chunk_cnt= 44 mod=ParamCharsType [MEMORY] hold= 958464 used= 950272 count= 1 avg_used= 950272 block_cnt= 1 chunk_cnt= 1 mod=IOWorkerLQ [MEMORY] hold= 933888 used= 931072 count= 1 avg_used= 931072 block_cnt= 1 chunk_cnt= 1 mod=TransService [MEMORY] hold= 933888 used= 931072 count= 1 avg_used= 931072 block_cnt= 1 chunk_cnt= 1 mod=ArcSenderQueue [MEMORY] hold= 829632 used= 821248 count= 2 avg_used= 410624 block_cnt= 2 chunk_cnt= 2 mod=LSMap [MEMORY] hold= 802816 used= 800000 count= 1 avg_used= 800000 block_cnt= 1 chunk_cnt= 1 mod=SqlFltSpanRec [MEMORY] hold= 798112 used= 789624 count= 3 avg_used= 263208 block_cnt= 2 chunk_cnt= 2 mod=PartitTableTask [MEMORY] hold= 786688 used= 524352 count= 33 avg_used= 15889 block_cnt= 33 chunk_cnt= 4 mod=LogAlloc [MEMORY] hold= 786432 used= 386304 count= 48 avg_used= 8048 block_cnt= 48 chunk_cnt= 20 mod=[T]ObPerfEventR [MEMORY] hold= 679872 used= 658688 count= 83 avg_used= 7936 block_cnt= 10 chunk_cnt= 7 mod=TltMemtablMgr [MEMORY] hold= 671744 used= 663552 count= 1 avg_used= 663552 block_cnt= 1 chunk_cnt= 1 mod=ApplySrv [MEMORY] hold= 666880 used= 404416 count= 89 avg_used= 4544 block_cnt= 66 chunk_cnt= 32 mod=[T]ObTraceEvent [MEMORY] hold= 663552 used= 659200 count= 1 avg_used= 659200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 663552 used= 655360 count= 1 avg_used= 655360 block_cnt= 1 chunk_cnt= 1 mod=FetchLog [MEMORY] hold= 655360 used= 626816 count= 4 avg_used= 156704 block_cnt= 4 chunk_cnt= 4 mod=SQLSessionInfo [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=MdsT [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=OBJLockGC [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=CoordTF [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=DupTbLease [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=ElectTimer [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=MultiVersionGC [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=DetectorTimer [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=CoordTR [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=FrzTrigger [MEMORY] hold= 598016 used= 590232 count= 1 avg_used= 590232 block_cnt= 1 chunk_cnt= 1 mod=DagNetIdMap [MEMORY] hold= 516096 used= 507904 count= 1 avg_used= 507904 block_cnt= 1 chunk_cnt= 1 mod=CommitVersions [MEMORY] hold= 507904 used= 491520 count= 2 avg_used= 245760 block_cnt= 2 chunk_cnt= 1 mod=SqlExecutor [MEMORY] hold= 483328 used= 477216 count= 1 avg_used= 477216 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMon [MEMORY] hold= 475136 used= 458816 count= 2 avg_used= 229408 block_cnt= 2 chunk_cnt= 2 mod=ts_token [MEMORY] hold= 442368 used= 393216 count= 6 avg_used= 65536 block_cnt= 6 chunk_cnt= 6 mod=MinorMergeMgr [MEMORY] hold= 409600 used= 401408 count= 1 avg_used= 401408 block_cnt= 1 chunk_cnt= 1 mod=ReplaySrv [MEMORY] hold= 401408 used= 393504 count= 1 avg_used= 393504 block_cnt= 1 chunk_cnt= 1 mod=ClogGe [MEMORY] hold= 385024 used= 375520 count= 2 avg_used= 187760 block_cnt= 2 chunk_cnt= 2 mod=bf_queue [MEMORY] hold= 376832 used= 223376 count= 23 avg_used= 9712 block_cnt= 23 chunk_cnt= 17 mod=[T]ObDSActionAr [MEMORY] hold= 311104 used= 303904 count= 6 avg_used= 50650 block_cnt= 6 chunk_cnt= 3 mod=PoolFreeList [MEMORY] hold= 303104 used= 294936 count= 1 avg_used= 294936 block_cnt= 1 chunk_cnt= 1 mod=DmlStatHashMap [MEMORY] hold= 303104 used= 294936 count= 1 avg_used= 294936 block_cnt= 1 chunk_cnt= 1 mod=RSCompStMap [MEMORY] hold= 303104 used= 294936 count= 1 avg_used= 294936 block_cnt= 1 chunk_cnt= 1 mod=RSCompactMap [MEMORY] hold= 303104 used= 294936 count= 1 avg_used= 294936 block_cnt= 1 chunk_cnt= 1 mod=ColUsagHashMap [MEMORY] hold= 286720 used= 280192 count= 1 avg_used= 280192 block_cnt= 1 chunk_cnt= 1 mod=MetaMemMgr [MEMORY] hold= 278528 used= 270468 count= 2 avg_used= 135234 block_cnt= 2 chunk_cnt= 2 mod=CsSstableReader [MEMORY] hold= 247120 used= 3318 count= 3061 avg_used= 1 block_cnt= 31 chunk_cnt= 1 mod=CharsetUtil [MEMORY] hold= 212992 used= 211072 count= 1 avg_used= 211072 block_cnt= 1 chunk_cnt= 1 mod=TimerWK [MEMORY] hold= 204800 used= 196616 count= 1 avg_used= 196616 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 204800 used= 196616 count= 1 avg_used= 196616 block_cnt= 1 chunk_cnt= 1 mod=TenDLBucket [MEMORY] hold= 204800 used= 196616 count= 1 avg_used= 196616 block_cnt= 1 chunk_cnt= 1 mod=T3MBucket [MEMORY] hold= 204800 used= 196744 count= 1 avg_used= 196744 block_cnt= 1 chunk_cnt= 1 mod=DagNetMap [MEMORY] hold= 188416 used= 182528 count= 23 avg_used= 7936 block_cnt= 12 chunk_cnt= 8 mod=SqlSession [MEMORY] hold= 155648 used= 150784 count= 19 avg_used= 7936 block_cnt= 19 chunk_cnt= 15 mod=LinearHashMapNo [MEMORY] hold= 147456 used= 139264 count= 1 avg_used= 139264 block_cnt= 1 chunk_cnt= 1 mod=RFLTaskQueue [MEMORY] hold= 131072 used= 130560 count= 2 avg_used= 65280 block_cnt= 2 chunk_cnt= 2 mod=MemTblObj [MEMORY] hold= 122880 used= 110608 count= 2 avg_used= 55304 block_cnt= 2 chunk_cnt= 2 mod=TabletIDSetBkt [MEMORY] hold= 114688 used= 111104 count= 14 avg_used= 7936 block_cnt= 13 chunk_cnt= 10 mod=QueryAllocator [MEMORY] hold= 114688 used= 111096 count= 1 avg_used= 111096 block_cnt= 1 chunk_cnt= 1 mod=DbmsSched_Job [MEMORY] hold= 114688 used= 111096 count= 1 avg_used= 111096 block_cnt= 1 chunk_cnt= 1 mod=AsyncFreezeTabl [MEMORY] hold= 114688 used= 106496 count= 1 avg_used= 106496 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlMgr [MEMORY] hold= 102512 used= 27016 count= 370 avg_used= 73 block_cnt= 74 chunk_cnt= 31 mod=Coro [MEMORY] hold= 98304 used= 95520 count= 1 avg_used= 95520 block_cnt= 1 chunk_cnt= 1 mod=LSSvr [MEMORY] hold= 98304 used= 91968 count= 1 avg_used= 91968 block_cnt= 1 chunk_cnt= 1 mod=tenant_mf_mgr [MEMORY] hold= 88000 used= 85888 count= 11 avg_used= 7808 block_cnt= 11 chunk_cnt= 1 mod=TabletIDSetNode [MEMORY] hold= 81920 used= 73896 count= 1 avg_used= 73896 block_cnt= 1 chunk_cnt= 1 mod=RSCompPairCache [MEMORY] hold= 81824 used= 73472 count= 2 avg_used= 36736 block_cnt= 2 chunk_cnt= 2 mod=ts_cancel_set [MEMORY] hold= 81824 used= 73472 count= 2 avg_used= 36736 block_cnt= 2 chunk_cnt= 2 mod=ts_queue [MEMORY] hold= 81824 used= 73472 count= 2 avg_used= 36736 block_cnt= 2 chunk_cnt= 2 mod=ts_run_set [MEMORY] hold= 81536 used= 78080 count= 10 avg_used= 7808 block_cnt= 10 chunk_cnt= 8 mod=HashNodeLCSta [MEMORY] hold= 73728 used= 72736 count= 1 avg_used= 72736 block_cnt= 1 chunk_cnt= 1 mod=LogSharedQueueT [MEMORY] hold= 73728 used= 72736 count= 1 avg_used= 72736 block_cnt= 1 chunk_cnt= 1 mod=LogSubmitLogQue [MEMORY] hold= 73728 used= 66176 count= 1 avg_used= 66176 block_cnt= 1 chunk_cnt= 1 mod=Rpc [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=DEVICE_MANAGER [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=TxCtxMemObj [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=FlushLog [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=FlushMeta [MEMORY] hold= 65536 used= 63488 count= 8 avg_used= 7936 block_cnt= 8 chunk_cnt= 7 mod=SqlSessiVarMap [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=TxDataMemObj [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=LockMemObj [MEMORY] hold= 65536 used= 63488 count= 8 avg_used= 7936 block_cnt= 8 chunk_cnt= 6 mod=PlanVaIdx [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=StorMetaCacheIO [MEMORY] hold= 57344 used= 51328 count= 1 avg_used= 51328 block_cnt= 1 chunk_cnt= 1 mod=palf_env [MEMORY] hold= 56448 used= 48384 count= 14 avg_used= 3456 block_cnt= 11 chunk_cnt= 9 mod=CACHE_INST [MEMORY] hold= 52672 used= 34944 count= 52 avg_used= 672 block_cnt= 36 chunk_cnt= 17 mod=[T]ObWarningBuf [MEMORY] hold= 49152 used= 47616 count= 2 avg_used= 23808 block_cnt= 2 chunk_cnt= 1 mod=IOUsageInfo [MEMORY] hold= 49152 used= 45184 count= 1 avg_used= 45184 block_cnt= 1 chunk_cnt= 1 mod=IO_MGR [MEMORY] hold= 48960 used= 44840 count= 2 avg_used= 22420 block_cnt= 2 chunk_cnt= 2 mod=DynamicFactor [MEMORY] hold= 48112 used= 5016 count= 209 avg_used= 24 block_cnt= 74 chunk_cnt= 29 mod=[T]MemoryContex [MEMORY] hold= 46400 used= 32160 count= 67 avg_used= 480 block_cnt= 25 chunk_cnt= 8 mod=OMT_Worker [MEMORY] hold= 42096 used= 28792 count= 59 avg_used= 488 block_cnt= 30 chunk_cnt= 12 mod=[T]ObSQLMockedT [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=TLD_TblCtxMap [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletPtrMap [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CliTaskMap [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=DagWarnHisBkt [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoBkt [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=Autoincrement [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CTBriefMap [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=HB_SERVICE [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=HTblRkMapBkt [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=HashBucCliInFo [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=TableAQueryBkt [MEMORY] hold= 40960 used= 39680 count= 5 avg_used= 7936 block_cnt= 5 chunk_cnt= 5 mod=LSIter [MEMORY] hold= 40896 used= 39680 count= 5 avg_used= 7936 block_cnt= 5 chunk_cnt= 5 mod=MdsTableMgr [MEMORY] hold= 40768 used= 39040 count= 5 avg_used= 7808 block_cnt= 5 chunk_cnt= 5 mod=HashNodePlanCac [MEMORY] hold= 40640 used= 39680 count= 5 avg_used= 7936 block_cnt= 5 chunk_cnt= 1 mod=ObDMMDL [MEMORY] hold= 36096 used= 33216 count= 3 avg_used= 11072 block_cnt= 3 chunk_cnt= 3 mod=DI_CONTAINER [MEMORY] hold= 32768 used= 27960 count= 3 avg_used= 9320 block_cnt= 3 chunk_cnt= 3 mod=PlGdbHelper [MEMORY] hold= 32768 used= 24688 count= 2 avg_used= 12344 block_cnt= 2 chunk_cnt= 2 mod=TLD_AssignedMgr [MEMORY] hold= 32768 used= 15872 count= 4 avg_used= 3968 block_cnt= 4 chunk_cnt= 4 mod=CkpDgn [MEMORY] hold= 32704 used= 25191 count= 2 avg_used= 12595 block_cnt= 2 chunk_cnt= 2 mod=CGReadInfoMgr [MEMORY] hold= 24576 used= 20136 count= 2 avg_used= 10068 block_cnt= 2 chunk_cnt= 2 mod=TabletStats [MEMORY] hold= 24576 used= 16392 count= 1 avg_used= 16392 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 24576 used= 19776 count= 1 avg_used= 19776 block_cnt= 1 chunk_cnt= 1 mod=LogApplyStatus [MEMORY] hold= 17024 used= 8448 count= 3 avg_used= 2816 block_cnt= 2 chunk_cnt= 2 mod=BaseLogWriter [MEMORY] hold= 16384 used= 8448 count= 1 avg_used= 8448 block_cnt= 1 chunk_cnt= 1 mod=PalfEnv [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=MaBlkLoadMap [MEMORY] hold= 16384 used= 15584 count= 2 avg_used= 7792 block_cnt= 2 chunk_cnt= 2 mod=CkptDgnCUNode [MEMORY] hold= 16384 used= 9792 count= 1 avg_used= 9792 block_cnt= 1 chunk_cnt= 1 mod=TSQLSessionMgr [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=TLD_ResourceMgr [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=MedCheckSet [MEMORY] hold= 16384 used= 13896 count= 1 avg_used= 13896 block_cnt= 1 chunk_cnt= 1 mod=task_cache [MEMORY] hold= 16384 used= 10248 count= 1 avg_used= 10248 block_cnt= 1 chunk_cnt= 1 mod=TabletSvrBucket [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=backupTaskSched [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletMap [MEMORY] hold= 16384 used= 14016 count= 1 avg_used= 14016 block_cnt= 1 chunk_cnt= 1 mod=LogReplayStatus [MEMORY] hold= 16384 used= 8200 count= 1 avg_used= 8200 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 12864 used= 12288 count= 3 avg_used= 4096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapCn [MEMORY] hold= 12864 used= 12288 count= 3 avg_used= 4096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapDi [MEMORY] hold= 11008 used= 10624 count= 2 avg_used= 5312 block_cnt= 1 chunk_cnt= 1 mod=Scheduler [MEMORY] hold= 9456 used= 9064 count= 2 avg_used= 4532 block_cnt= 2 chunk_cnt= 2 mod=GROUP_INDEX_MAP [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=HandleSubmit [MEMORY] hold= 8192 used= 7792 count= 1 avg_used= 7792 block_cnt= 1 chunk_cnt= 1 mod=CkptDgnMemNode [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=RSCompPairs [MEMORY] hold= 8192 used= 7632 count= 1 avg_used= 7632 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoNode [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=FillCache [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=GetMediumList [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=ls_id [MEMORY] hold= 8192 used= 7712 count= 1 avg_used= 7712 block_cnt= 1 chunk_cnt= 1 mod=DmlStatNode [MEMORY] hold= 8192 used= 7792 count= 1 avg_used= 7792 block_cnt= 1 chunk_cnt= 1 mod=ColUsagNode [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=GetKeptShotInfo [MEMORY] hold= 8192 used= 7352 count= 1 avg_used= 7352 block_cnt= 1 chunk_cnt= 1 mod=mrg_prog_cker [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=MediumList [MEMORY] hold= 8192 used= 7632 count= 1 avg_used= 7632 block_cnt= 1 chunk_cnt= 1 mod=CheckMapNode [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=LsElectRefInfo [MEMORY] hold= 8192 used= 7728 count= 1 avg_used= 7728 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsCache [MEMORY] hold= 8192 used= 7808 count= 1 avg_used= 7808 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsInfo [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=MatchLsnMap [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=LCLSender [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=HTableLockMap [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=CPUUSage [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=APPLY_STATUS [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=leakChecker [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=ShareBlksMap [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=REPLAY_STATUS [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=MdsDebugMap [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=DASIDCache [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=LSInfo [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=PurgeThrottle [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileMap [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=LockWaitMgr [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileBlkMgr [MEMORY] hold= 6880 used= 5600 count= 7 avg_used= 800 block_cnt= 7 chunk_cnt= 6 mod=DICollector [MEMORY] hold= 5888 used= 3392 count= 8 avg_used= 424 block_cnt= 8 chunk_cnt= 7 mod=SequenceIdMap [MEMORY] hold= 5632 used= 4096 count= 8 avg_used= 512 block_cnt= 7 chunk_cnt= 6 mod=SqlSessiQuerSql [MEMORY] hold= 5216 used= 1976 count= 13 avg_used= 152 block_cnt= 2 chunk_cnt= 2 mod=ObTxLSLogCb [MEMORY] hold= 5120 used= 3392 count= 8 avg_used= 424 block_cnt= 8 chunk_cnt= 7 mod=PackStateMap [MEMORY] hold= 5120 used= 3392 count= 8 avg_used= 424 block_cnt= 8 chunk_cnt= 7 mod=SequenceMap [MEMORY] hold= 5120 used= 3392 count= 8 avg_used= 424 block_cnt= 8 chunk_cnt= 7 mod=ContextsMap [MEMORY] hold= 5056 used= 4656 count= 2 avg_used= 2328 block_cnt= 2 chunk_cnt= 2 mod=MViewService [MEMORY] hold= 4832 used= 4632 count= 1 avg_used= 4632 block_cnt= 1 chunk_cnt= 1 mod=PxPoolBkt [MEMORY] hold= 4832 used= 4632 count= 1 avg_used= 4632 block_cnt= 1 chunk_cnt= 1 mod=ExTabletCntMap [MEMORY] hold= 4832 used= 4632 count= 1 avg_used= 4632 block_cnt= 1 chunk_cnt= 1 mod=ResRuleIdMap [MEMORY] hold= 4832 used= 4632 count= 1 avg_used= 4632 block_cnt= 1 chunk_cnt= 1 mod=ResGrpIdMap [MEMORY] hold= 4288 used= 4096 count= 1 avg_used= 4096 block_cnt= 1 chunk_cnt= 1 mod=MacroFile [MEMORY] hold= 4160 used= 3520 count= 4 avg_used= 880 block_cnt= 3 chunk_cnt= 3 mod=PoolArenaArray [MEMORY] hold= 4096 used= 2328 count= 1 avg_used= 2328 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoMap [MEMORY] hold= 3328 used= 1728 count= 4 avg_used= 432 block_cnt= 4 chunk_cnt= 4 mod=NegParamIdex [MEMORY] hold= 3328 used= 1792 count= 8 avg_used= 224 block_cnt= 1 chunk_cnt= 1 mod=LogIOTask [MEMORY] hold= 3312 used= 3112 count= 1 avg_used= 3112 block_cnt= 1 chunk_cnt= 1 mod=PendingFreeMap [MEMORY] hold= 3312 used= 3112 count= 1 avg_used= 3112 block_cnt= 1 chunk_cnt= 1 mod=TFileFLCtx [MEMORY] hold= 3200 used= 3008 count= 1 avg_used= 3008 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlDfc [MEMORY] hold= 2528 used= 2328 count= 1 avg_used= 2328 block_cnt= 1 chunk_cnt= 1 mod=DASMemHashBuck [MEMORY] hold= 2528 used= 2328 count= 1 avg_used= 2328 block_cnt= 1 chunk_cnt= 1 mod=VecIdxLSMgr [MEMORY] hold= 2528 used= 2328 count= 1 avg_used= 2328 block_cnt= 1 chunk_cnt= 1 mod=MedCheckMap [MEMORY] hold= 2528 used= 2328 count= 1 avg_used= 2328 block_cnt= 1 chunk_cnt= 1 mod=IRMMemHashBuck [MEMORY] hold= 2240 used= 2048 count= 1 avg_used= 2048 block_cnt= 1 chunk_cnt= 1 mod=CreateContext [MEMORY] hold= 2048 used= 1272 count= 1 avg_used= 1272 block_cnt= 1 chunk_cnt= 1 mod=LSCheckMap [MEMORY] hold= 2048 used= 1856 count= 1 avg_used= 1856 block_cnt= 1 chunk_cnt= 1 mod=LogIOWS [MEMORY] hold= 2016 used= 1824 count= 1 avg_used= 1824 block_cnt= 1 chunk_cnt= 1 mod=PxResMgr [MEMORY] hold= 1920 used= 1728 count= 1 avg_used= 1728 block_cnt= 1 chunk_cnt= 1 mod=timer_service [MEMORY] hold= 1808 used= 1424 count= 2 avg_used= 712 block_cnt= 2 chunk_cnt= 2 mod=DetectManager [MEMORY] hold= 1744 used= 1544 count= 1 avg_used= 1544 block_cnt= 1 chunk_cnt= 1 mod=TXBFRetryId [MEMORY] hold= 1744 used= 1544 count= 1 avg_used= 1544 block_cnt= 1 chunk_cnt= 1 mod=TabStatMgrLock [MEMORY] hold= 1472 used= 1272 count= 1 avg_used= 1272 block_cnt= 1 chunk_cnt= 1 mod=HashBucApiGroup [MEMORY] hold= 1472 used= 1272 count= 1 avg_used= 1272 block_cnt= 1 chunk_cnt= 1 mod=DUP_LS_SET [MEMORY] hold= 1472 used= 1272 count= 1 avg_used= 1272 block_cnt= 1 chunk_cnt= 1 mod=GCMemtableMap [MEMORY] hold= 1280 used= 1088 count= 1 avg_used= 1088 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 1248 used= 1056 count= 1 avg_used= 1056 block_cnt= 1 chunk_cnt= 1 mod=LOG_HASH_MAP [MEMORY] hold= 1232 used= 1032 count= 1 avg_used= 1032 block_cnt= 1 chunk_cnt= 1 mod=BFLoadTask [MEMORY] hold= 1200 used= 1000 count= 1 avg_used= 1000 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletMapLk [MEMORY] hold= 1120 used= 120 count= 5 avg_used= 24 block_cnt= 2 chunk_cnt= 2 mod=FreezeTask [MEMORY] hold= 1024 used= 832 count= 1 avg_used= 832 block_cnt= 1 chunk_cnt= 1 mod=tenant_rm_mgr [MEMORY] hold= 976 used= 776 count= 1 avg_used= 776 block_cnt= 1 chunk_cnt= 1 mod=DepTabletSet [MEMORY] hold= 896 used= 696 count= 1 avg_used= 696 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfo [MEMORY] hold= 752 used= 552 count= 1 avg_used= 552 block_cnt= 1 chunk_cnt= 1 mod=LSFreeze [MEMORY] hold= 704 used= 208 count= 1 avg_used= 208 block_cnt= 1 chunk_cnt= 1 mod=[T]ObMacroBlock [MEMORY] hold= 672 used= 472 count= 1 avg_used= 472 block_cnt= 1 chunk_cnt= 1 mod=Election [MEMORY] hold= 640 used= 224 count= 2 avg_used= 112 block_cnt= 1 chunk_cnt= 1 mod=Coordinator [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=MSTXCTX [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=VectorIndexVsag [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=HAScheduler [MEMORY] hold= 544 used= 144 count= 2 avg_used= 72 block_cnt= 2 chunk_cnt= 2 mod=TntSrvObjPool [MEMORY] hold= 512 used= 120 count= 2 avg_used= 60 block_cnt= 1 chunk_cnt= 1 mod=UserResourceMgr [MEMORY] hold= 448 used= 64 count= 2 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfFSCbNode [MEMORY] hold= 416 used= 2 count= 2 avg_used= 1 block_cnt= 2 chunk_cnt= 2 mod=ObLogEXTHandler [MEMORY] hold= 400 used= 200 count= 1 avg_used= 200 block_cnt= 1 chunk_cnt= 1 mod=ExTabletMapLk [MEMORY] hold= 384 used= 102 count= 1 avg_used= 102 block_cnt= 1 chunk_cnt= 1 mod=TxLSLogBuf [MEMORY] hold= 304 used= 104 count= 1 avg_used= 104 block_cnt= 1 chunk_cnt= 1 mod=RedoLogBuckLock [MEMORY] hold= 256 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=[T]TSILastOper [MEMORY] hold= 256 used= 48 count= 1 avg_used= 48 block_cnt= 1 chunk_cnt= 1 mod=[T]TSITabletTab [MEMORY] hold= 256 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 240 used= 41 count= 1 avg_used= 41 block_cnt= 1 chunk_cnt= 1 mod=KeepAliveBuf [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=RebuildCbNode [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfRCCbNode [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=logservice [MEMORY] hold= 389804064 used= 379547427 count= 32900 avg_used= 11536 mod=SUMMARY [2025-08-14 09:07:30.749198] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=28245] [sql nio session](*s={this:0x7f98f72d0030, session_id:3221589415, trace_id:YB42AC4D0007-00063C4965FA43DC-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:245}, err:0, last_decode_time:1755161751714366, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1662, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:07:31.741710] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=48] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 230686720 used= 228884480 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 block_cnt= 110 chunk_cnt= 110 mod=Memstore [MEMORY] hold= 228884480 used= 228779760 count= 110 avg_used= 2079816 mod=SUMMARY [2025-08-14 09:07:31.741727] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1 ctx_id= EXECUTE_CTX_ID hold= 1163264 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 7 washed_blocks= 15 washed_size= 13516800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:07:31.741722] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [410][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=991021] [sql nio session](*s={this:0x7f98f72d5030, session_id:3221590200, trace_id:YB42AC4D0007-00063C4965FA43E0-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:248}, err:0, last_decode_time:1755161776777167, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1593, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:07:31.741741] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 483328 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1613824 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 block_cnt= 1 chunk_cnt= 1 mod=PartTranCtxMgr [MEMORY] hold= 466944 used= 460992 count= 1 avg_used= 460992 mod=SUMMARY [2025-08-14 09:07:31.741761] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 51617792 used= 47452416 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 84 washed_blocks= 334 washed_size= 126640128 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 40241472 used= 36454879 count= 2829 avg_used= 12886 block_cnt= 2829 chunk_cnt= 78 mod=SqlPhyPlan [MEMORY] hold= 3768320 used= 3427728 count= 321 avg_used= 10678 block_cnt= 84 chunk_cnt= 49 mod=SqlPlanCache [MEMORY] hold= 1524288 used= 1400877 count= 158 avg_used= 8866 block_cnt= 158 chunk_cnt= 11 mod=SqlPhyPlObj [MEMORY] hold= 1379840 used= 1261568 count= 616 avg_used= 2048 block_cnt= 223 chunk_cnt= 58 mod=CreateContext [MEMORY] hold= 536448 used= 523776 count= 66 avg_used= 7936 block_cnt= 66 chunk_cnt= 24 mod=OB_SQLSTAT_OBJ [MEMORY] hold= 2048 used= 1664 count= 2 avg_used= 832 block_cnt= 1 chunk_cnt= 1 mod=PlanCache [MEMORY] hold= 47452416 used= 43070492 count= 3992 avg_used= 10789 mod=SUMMARY [2025-08-14 09:07:31.741753] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:983) [414][sql_nio7][T0][Y0-0000000000000000-0-0] [lt=25256] sock ref clean, do destroy(*s={this:0x7f98f7335030, session_id:3221590537, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:265}, err:5, last_decode_time:1755162392718915, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1366, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:07:31.741774] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= WORK_AREA hold= 1458176 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 13 washed_blocks= 66 washed_size= 25804800 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:07:31.741885] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 14204928 used= 10970224 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 17 washed_blocks= 277 washed_size= 31932416 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10953744 used= 6065548 count= 18570 avg_used= 326 block_cnt= 372 chunk_cnt= 12 mod=PlJit [MEMORY] hold= 16480 used= 13376 count= 3 avg_used= 4458 block_cnt= 2 chunk_cnt= 2 mod=PlCodeGen [MEMORY] hold= 10970224 used= 6078924 count= 18573 avg_used= 327 mod=SUMMARY [2025-08-14 09:07:31.741906] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 111149056 used= 108896256 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 block_cnt= 211 chunk_cnt= 53 mod=CoStack [MEMORY] hold= 108896256 used= 108693696 count= 211 avg_used= 515136 mod=SUMMARY [2025-08-14 09:07:31.741925] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 56623104 used= 56623104 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 56623104 used= 0 count= 27 avg_used= 0 block_cnt= 0 chunk_cnt= 0 mod=KvstorCacheMb [MEMORY] hold= 56623104 used= 0 count= 27 avg_used= 0 mod=SUMMARY [2025-08-14 09:07:31.741942] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 35651584 used= 34365376 limit= 429496720 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 29491200 used= 26236400 count= 400 avg_used= 65591 block_cnt= 400 chunk_cnt= 15 mod=L_TabletPool [MEMORY] hold= 4784128 used= 4765440 count= 73 avg_used= 65280 block_cnt= 73 chunk_cnt= 3 mod=N_TabletPool [MEMORY] hold= 90048 used= 84768 count= 2 avg_used= 42384 block_cnt= 2 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 34365376 used= 31086608 count= 475 avg_used= 65445 mod=SUMMARY [2025-08-14 09:07:31.741962] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 8003584 used= 4947392 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 10 washed_blocks= 23 washed_size= 12967936 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4947392 used= 4793344 count= 604 avg_used= 7936 block_cnt= 66 chunk_cnt= 8 mod=MemtableCallbac [MEMORY] hold= 4947392 used= 4793344 count= 604 avg_used= 7936 mod=SUMMARY [2025-08-14 09:07:31.741991] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=20] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 147456 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=LobPersist [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:07:31.742007] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 65536 used= 31024 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 2031616 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 31024 used= 27058 count= 20 avg_used= 1352 block_cnt= 6 chunk_cnt= 1 mod=SqlPsCache [MEMORY] hold= 31024 used= 27058 count= 20 avg_used= 1352 mod=SUMMARY [2025-08-14 09:07:31.742025] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= RPC_CTX_ID hold= 14827520 used= 6111232 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1949696 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 5922816 used= 3917696 count= 241 avg_used= 16256 block_cnt= 241 chunk_cnt= 5 mod=OB_WRS_CLUSTER_ [MEMORY] hold= 98304 used= 65024 count= 4 avg_used= 16256 block_cnt= 4 chunk_cnt= 2 mod=OB_LOG_REGISTER [MEMORY] hold= 24576 used= 11904 count= 3 avg_used= 3968 block_cnt= 2 chunk_cnt= 1 mod=[L]OB_WRS_CLUST [MEMORY] hold= 24576 used= 16256 count= 1 avg_used= 16256 block_cnt= 1 chunk_cnt= 1 mod=OB_TENANT_MGR [MEMORY] hold= 24576 used= 16256 count= 1 avg_used= 16256 block_cnt= 1 chunk_cnt= 1 mod=OB_LOG_GET_STAT [MEMORY] hold= 8192 used= 3968 count= 1 avg_used= 3968 block_cnt= 1 chunk_cnt= 1 mod=[L]OB_LOG_REGIS [MEMORY] hold= 8192 used= 3968 count= 1 avg_used= 3968 block_cnt= 1 chunk_cnt= 1 mod=[L]OB_TENANT_MG [MEMORY] hold= 6111232 used= 4035072 count= 252 avg_used= 16012 mod=SUMMARY [2025-08-14 09:07:31.742042] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 3481600 used= 2391232 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 4 washed_blocks= 10 washed_size= 4907008 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2121728 used= 2105152 count= 3 avg_used= 701717 block_cnt= 3 chunk_cnt= 3 mod=MEMTABLE_ARENA [MEMORY] hold= 261376 used= 253952 count= 32 avg_used= 7936 block_cnt= 21 chunk_cnt= 2 mod=TX_DATA_SLICE [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=TX_DATA_ARENA [MEMORY] hold= 2391232 used= 2367040 count= 36 avg_used= 65751 mod=SUMMARY [2025-08-14 09:07:31.742059] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 3964928 used= 3080192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 2 washed_blocks= 3 washed_size= 229376 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3080192 used= 2983936 count= 376 avg_used= 7936 block_cnt= 24 chunk_cnt= 2 mod=MdsTable [MEMORY] hold= 3080192 used= 2983936 count= 376 avg_used= 7936 mod=SUMMARY [2025-08-14 09:07:31.742076] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= SCHEMA_SERVICE hold= 212992 used= 8192 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 1 washed_blocks= 1 washed_size= 1884160 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=NonPartTenCac [MEMORY] hold= 8192 used= 7936 count= 1 avg_used= 7936 mod=SUMMARY [2025-08-14 09:07:31.742091] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 31457280 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:07:31.742103] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= MERGE_NORMAL_CTX_ID hold= 884736 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 6 washed_blocks= 9 washed_size= 11698176 [MEMORY] request_cached_chunk_cnt= 0 [2025-08-14 09:07:30.749327] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [412][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=6960149] [sql nio session](*s={this:0x7f98f733a030, session_id:3221590255, trace_id:YB42AC4D0007-00063C49651A3232-0-0, sql_handling_stage:11, sql_initiative_shutdown:false, reader:{fd:257}, err:0, last_decode_time:1755161952736711, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1716, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:07:31.743687] WDIAG alarm_if_necessary (ob_timer_service.cpp:134) [807][T1002_TimerWK3][T1002][YB42AC4D0007-00063C495F8A198A-0-0] [lt=9787][errcode=-4389] timer task too much delay in thread pool(token={this:0x7f991b7a7918, timer:0x7f99329e19d0, task:0x7f98f55f1250, task_type:"N9oceanbase5share24ObVectorIndexHistoryTaskE", scheduled_time:1755162402751940, pushed_time:1755162402752014, delay:10000000}, delay_in_thread_pool=48007139, threashold=500000) [2025-08-14 09:07:30.749365] INFO [RPC.OBMYSQL] print_session_info (ob_sql_nio.cpp:1171) [411][sql_nio4][T0][Y0-0000000000000000-0-0] [lt=6954911] [sql nio session](*s={this:0x7f98f72c3030, session_id:3221590336, trace_id:YB42AC4D0007-00063C49634A0941-0-0, sql_handling_stage:3, sql_initiative_shutdown:false, reader:{fd:255}, err:5, last_decode_time:1755162354747599, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1422, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-08-14 09:07:31.745446] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [1027][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=1970597] current monitor number(seq_=-1) [2025-08-14 09:07:31.746087] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [385][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:31.746123] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [568][T1_LSArchiveMgr][T1][YB42AC4D0007-00063C495B799C16-0-0] [lt=439] gc stale ls task succ [2025-08-14 09:07:31.746720] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [472][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571c1730, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571c1a10, time_wheel:0x7f9959ff7990, thread_pool:0x7f9959ff7170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:31.664, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.746784] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:573) [730][T1001_LSArchive][T1001][YB42AC4D0007-00063C495C899C16-0-0] [lt=158] gc stale ls task succ [2025-08-14 09:07:31.750602] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919b14b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891968810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:31.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:31.750646] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=41][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891982a30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970810, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:31.741, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:31.750669] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919827b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891970690, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:31.749, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:31.751618] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [474][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571f8410, caller:election_proposer.cpp:register_renew_lease_task_:260, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_88, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f99571f86f0, time_wheel:0x7f9959ffb990, thread_pool:0x7f9959ffb170, is_running:1, time_interval:500.00ms, expected_run_time:09:07:31.744, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.756032] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [391][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:31.756069] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [395][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:31.756214] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [399][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:31.756278] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [383][RpcIO][T0][Y0-0000000000000000-0-0] [lt=57] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:31.756298] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [389][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:31.756308] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [397][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:31.757420] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [394][BatchIO][T0][Y0-0000000000000000-0-0] [lt=40] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:31.757455] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [392][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:31.757484] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [384][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:31.757499] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [386][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:31.757663] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [856][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-08-14 09:07:31.757869] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [401][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=14] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:31.760671] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180f1b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f890, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:31.756, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:31.765079] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [770][T1002_CoordTF][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9957163880, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f9957163b60, time_wheel:0x7f99535a5990, thread_pool:0x7f99535a5170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:31.655, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.746858] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [658][T1001_CoordTF][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99571fd450, caller:ob_failure_detector.cpp:mtl_start:72, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f99571fd730, time_wheel:0x7f9953599990, thread_pool:0x7f9953599170, is_running:1, time_interval:100.00ms, expected_run_time:09:07:31.693, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.770438] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fd8d0, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fdbb0, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:31.755, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.770466] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [772][T1002_ElectTime][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f99319fc950, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f99319fcc30, time_wheel:0x7f99535c1990, thread_pool:0x7f99535c1170, is_running:1, time_interval:250.00ms, expected_run_time:09:07:31.759, task_priority:1, with_handle_protected:1}) [2025-08-14 09:07:31.771719] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f72742b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98919b4910, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:31.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:31.771751] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98919840b0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f9891971590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:31.765, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:31.771764] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180ef30, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f710, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:31.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:31.771778] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f989180ecb0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f739f590, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:31.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:31.771790] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f98f7286330, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f7288b10, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:31.767, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:31.771803] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:184) [374][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] last commit task not scheduled yet, give up committing new task this time(ret=0, ret="OB_SUCCESS", *this={this:0x7f9891810ab0, caller:election_event_recorder.cpp:report_event_:87, function_type:ZN9oceanbase4palf8election13EventRecorder13report_event_ENS1_17ElectionEventTypeERKNS_6common8ObStringEE5$_169, timer_running_flag:1, total_running_count:418, func_shared_ptr_.ptr:0x7f98f73a0790, time_wheel:0x7f9959fdf990, thread_pool:0x7f9959fdf170, is_running:1, time_interval:15.00s, expected_run_time:09:07:31.761, task_priority:1, with_handle_protected:0}) [2025-08-14 09:07:31.772823] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [396][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-08-14 09:07:31.777052] INFO [LIB] operator() (ob_malloc_allocator.cpp:556) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 500, limit: 9223372036854775807 hold: 684654592 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 333369344 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 39403520 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 85983232 limit= 9223372036854775807 [MEMORY] ctx_id= LIBEASY hold_bytes= 3145728 limit= 9223372036854775807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 4194304 limit= 9223372036854775807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 3063808 limit= 9223372036854775807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 37584896 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 78925824 limit= 9223372036854775807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 98983936 limit= 9223372036854775807 [2025-08-14 09:07:31.777344] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:247) [303][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=28] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 333369344 used= 301911920 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 499 washed_blocks= 2229 washed_size= 922714112 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 52641792 used= 50996736 count= 3317 avg_used= 15374 block_cnt= 1499 chunk_cnt= 42 mod=TZInfoArray [MEMORY] hold= 35672064 used= 35651584 count= 1 avg_used= 35651584 block_cnt= 1 chunk_cnt= 1 mod=qdtable [MEMORY] hold= 33292288 used= 33275904 count= 16 avg_used= 2079744 block_cnt= 16 chunk_cnt= 16 mod=ObTxDesc [MEMORY] hold= 25067136 used= 24984064 count= 160 avg_used= 156150 block_cnt= 106 chunk_cnt= 50 mod=TenantCtxAlloca [MEMORY] hold= 12873728 used= 12856400 count= 1 avg_used= 12856400 block_cnt= 1 chunk_cnt= 1 mod=MemDumpContext [MEMORY] hold= 7397376 used= 7340256 count= 7 avg_used= 1048608 block_cnt= 7 chunk_cnt= 7 mod=ConcurHashMap [MEMORY] hold= 7275040 used= 6334546 count= 1107 avg_used= 5722 block_cnt= 571 chunk_cnt= 6 mod=CharsetInit [MEMORY] hold= 5373952 used= 5236736 count= 32 avg_used= 163648 block_cnt= 32 chunk_cnt= 15 mod=Iterator